Mom is 80, very difficult, refuses to go to the doctor. She stays up all night. What over the counter sleep aid or relaxing aid will help?

Is mom living with you?
Is mom living with your sister?
Are you all living together or is mom living alone?
Maybe none of these matter but she should not be living alone. And you and or your sister should not be neglecting your own families to care for your mom.
If your mom has been diagnosed with dementia she is no longer capable of making a rational decision regarding her care. Would you allow a child to say.."I'm not going to the doctor."?
Over the counter sleep aids make your brain pretty foggy when you wake up. Added to a brain that is already foggy with dementia I would think it would make the fogginess even worse. (try taking a PM medication or even Benedryl and see how you feel in the morning....and long time use of sleep aids have been linked to dementia)
And I would not be me if I did not mention SAFETY. I based keeping my Husband home on safety. If it was not safe for him for me to care for him at home I would have had to place him in Memory Care. If it was not safe for ME to care for him, again I would have had to place him. If your mom is being physically abusive to you or your sister you might have no choice but to place her in a Memory Care facility. If she hurts you or your sister who will care for either of you? If mom gets hurt not only will either of you feel guilty but any injury can make it more difficult for you to care for her and then the choice to place her may well be out of your hands.

One last comment. If you have not talked to an Elder Care Attorney to obtain Guardianship you should do so. (Probably a bit late for her to sign papers making you or your sister POA for health and financial decisions.) but you could be made her Guardian. The Attorney will also help with other papers that may/will be necessary

Do you and your sister live with your mother (or her with you)? When do you think the time will come when your sanity and health will be more important than trying to help your verbally abusive mother?

Can you see any other options for your mother's care? What about a facility?

We found that a 3 oz glass of dry red wine would not only calm mom down, it would rosy up her cheeks and change her attitude.