How do you separate an elderly parent from his live in caregiver that he has gotten too dependent on?

Father still lives in his home with a non-relative caregiver (nurse). The caregiver has complete control over his life and he has no contact with any of his children - will not answer the phone or anything. She has been with him for 4+ years and the controlling is just getting worse and worse. She threatens him with leaving him and has him convinced that his children just want him in a nursing home and his money. We don't want that but think an assisted living center would be the best place for him. We have tried Social Services but they say it's his choice and there is nothing we can do. Lately, he has been in the hospital for a couple of long stays and we were not notified because he doesn't want us to know - thinks we will make a scene with his caregiver at the hospital or put him in nursing home. Any suggestions? Anyone else ever have this problem? He is so in love with her (my mother died 2 years ago) that all he wants is his caregiver. He is missing out seeing his children, grandchildren and great-grandchildren. Just breaks my heart but what can I do?