What do you do when your loved one's legal determinations are questioned?

I have been with my loved one (non-relative) for 14 years. He has maintained his own house, job etc until 2 years ago. 5 years ago, we opened a joint checking account (joint, not a convenience account with a POA) with equal access and equal rights to it. While he has always had a card and checkbook to use the account I have paid all his bills and mine, took care of the shopping etc. While his deposits have been steady mine have varied depending on the job or whether or not I decided to deposit at all (cash tips) He also made me power of attorney (separately) 5 years ago, made me his medical power of attorney last year when he started to get sicker (old age) and made me his beneficiary when his family began trying to force me out. He has had a very limited relationship with them since before we met. Last year he had to move in with me because he was no longer able to live on his own. He was healthy and happy for several months until a fall put him in the hospital. Since then he was in and out of the hospital and a nursing home for rehab but his health as continued to deteriorate. In january of this year he became very ill with pneumonia and suddenly his family appears and filed a claim with adult protective services stating that his health was my fault and that I was stealing his money. They took me to court and the court granted them conservatorship. Now I'm facing possible criminal charges. What I don't understand is that he was not allowed to speak at the hearing. He was diagnosed with dementia but he was lucid and clear headed when he signed all of this legal documents putting me in charge of his affairs. Now he's in a home, sad, angry and demanding to be heard and I don't know how to help him without going to jail. When he became sick he was under the care of home health (I was not solely in charge of his medical care) and as for stealing his money we opened the joint account years ago. Neither one of us understands how this happened. He has no desire for his family to be involved in his affairs and now they have full control. I'm not even allowed to see him because his family had barred me from the nursing home where he is. What do I do?