There is a rumor in my family that my stepbrother will be getting an attorney to declare my Mom incompetent. What legally can I do to prevent this?

Is your stepfather still competent? Seems like the easiest all around would be for him to appoint the son Executor, if that is what he wants. Or to appoint someone else if he wishes.

If your mother has dementia, even if she is still competent in the legal sense, wouldn't the responsibilities of administering the estate after her husband dies be highly stressful for her?

Sounds to me like both Mom and StepDad need to discuss their wills and final wishes with an attorney.

Her competence is really not up to him, it's up to a judge. You hint that his father is still alive? His father can write a Will and make him Executor, this is totally up to the father and son. There's not much you can do about the stepfather's choice. Your duty is to protect your mother, if she will let you. Take her to an attorney who can offer her advice, and be prepared to pay the retainer at that visit.