I am lead caregiver and Guardian for my Mother. Her estranged husband and my 2 sisters (all of which live within 10 miles of Mother and I) ask to take Mother out to dinner every 5th week with no contact in between. Mother is just beginning the final stage of Alzheimer's and I feel a familiar caregiver must always accompany Mother on these outings. The 3 of them disagree and are accusing me of keeping Mother from them.
To give another side of the tale, I used to get annoyed when my sister came from out of state to visit Mom. When she wanted me to come along, I felt that this was MY free time. I felt sis should have alone time with my Mom.
Tell them they are more than welcome to bring dinner to the house. Olive Garden has carry-out menu, as does other restaurants. That would cause less confusion for Mom, or maybe not with having more people in the house. Maybe it is time when just one person at a time comes over and brings supper for Mom and you.