Is someone with power of attorney able to decide which family members are able to live with the person that's being taken care of?

So your uncle who owns the house in which you live invited you and your mom to live with him in order to be his caretakers? If that same uncle does not have dementia, and his brother can't prove that you are doing anything nefarious or illegal, I don't think he can evict you. Even if he could evict you, he needs to go through the legal eviction process, which gives you 30 days to get out. Your friendly uncle can create a new PoA any time he wants and doesn't need to inform his brother nor get his permission. Hopefully he has all his mind and this whole scenario isn't a power maneuver over money or inheritance. Sounds like that is what his brother is worried about. Does the PoA brother live in the house with you all? Even if he does, if he's not the homeowner, and your friendly uncle is of sound mind, his mean brother can't kick you out.

Are you all living in one house?

What happened that he doesn't want you around?
Dealing with that would be my 1st step.

If he lives in the same house and it is his property, yes he can have you evicted.

Have you and mom talked about what you would both do if he tries to evict you? I would have a plan in place.

Your uncle doesn’t have POA over anyone. He has POA FOR your uncle. He doesn’t have control over your uncle, he has the authority to make decisions on his behalf and he has to follow what your grandfather wants or would want and he has to act in his best interest.. And depending on what type of POA he holds and other circumstances, he may be able to have you evicted from the home.

A power of attorney for health care helps to make the decisions the patient wanted when able to express his or her own wishes, and helps to monitor that the care is rendered in a way that was the expressed wish of the person. POA for financial carries out the wishes so far as known by the person financially, and pays for their care when they are unable, making the soundest choices and keeping meticulous records. If the person involved is unable to make any decisions then the POA can hire help at home to help the person. And can make decisions about who will live in the household I believe. But if a person does not have dementia they are free of course to live with whomever they like, and are free in fact to revoke any POA previously given. So I think this is a matter completely of your own circumstances, and we don't have enough information to give you any guidance. If you have questions about someone "moving in" on a person with dementia over whom you have POA you may need to see an elder care attorney about guardianship, which isn't necessarily difficult if you are already doing so much; you then would be able to make the decisions legally. It is always best to check out the legal facts in your own part of the country.