My husband with dementia does not want to see his family. Yet, they keep pressuring me to visit him. How can I continue to protect him?

For the last 2 years, my husband with FTD has clearly communicated that he does not want to speak with his family. Of course, they think that I am the one keeping him away. Despite our POA and Nomination of Guardianship docs in our Revocable Living Trust signed years ago when he was well, I had to fight a cruel court battle to keep my beloved husband of almost 30 years. After years of ignoring him, and now that he cannot protect me, especially with an inheritance coming, the greed has reared its ugly head. His family continues to disrespect my rights as his wife and insist on seeing him on their terms. They have tried to secretly get his signature before. I cannot justify putting him in that stressful situation when he does not want to see them. What's worse is, that others are telling me that his family should be allowed to see him. They say that once they see how ill he is, they will leave you alone. I know better. The last time he saw them, he was so agitated that he walked out of the ALF and walked 5 miles home. I'm so grateful that he wasn't hit by a car, suffered heat stroke, or otherwise got injured. Since then, I cannot be out of his sight or he will walk out to find me. And how can I trust someone to take him to them knowing full well that after a couple of minutes, he will walk out and start for home. How can I, in good conscience, stress my husband and put him in danger of leaving? And how can I get people to stop pressuring me to do that? How can I get people to respect his rights, if not mine? I am completely exhausted and depleted. My last reserves must last to keep him safe. Can I get into any legal problems since I am his court appointed legal guardian responsible for not only his physical health, but also his mental wellbeing. Anybody have similar experiences or thoughts? Much appreciated.