Am I being taken advantage of by friend's family?

Greetings. I've been care-giving for a friend/roommate going on 4 years, he's going on 81, and for now he has the typical issues: forgetfulness, short term memory problems, incontinence, and mobility challenges. Luckily no major illness, yet, therefore it's manageable, it's not easy by any stretch, but I've adapted to what I currently have on my plate centering around the usual care-giving tasks: cooking, cleaning, housework, shopping, doctor appointments, companionship, and errands.

He has a retired ex-wife and two sons (50ish) that no longer work because of their own medical/emotional issues and now I'm finding myself getting pulled in to help them and I want to be a nice guy here but it's already wearing me out because I have to work full time on top of looking after their Dad, which they are not only physically incapable of doing, but don't seem to want to, I can count on one hand the number of times they've even called to check on him, and we stopped asking for the rare "day off" for me by having one of them sit with their Dad even for just an occasional Saturday.

Yet here they are with their hands out for my time because they know their Dad can't do it. The one son is a hoarder and his mother "volunteered" me for the task of assisting with cleaning up his unit while she's out of town, the other son has more neuroses than I can count, most of which center around paranoia and agoraphobia and therefore can't be called upon for anything, but has no issue with gallivanting across the country with his wife for various reasons.

I'm growing more angry and resentful by the day and facing burnout: I'm being treated for anxiety and see a therapist weekly. I suffer from severe anxiety and I'm on Buspirone and Lorazepam due to painful panic attacks/hyperventilation; I'm not depressed yet, but it's gaining ground and I can't afford to end up unable to work because unlike these people I have nobody to take care of me.

The whole situation has my friend very upset, his BP is up, and I'm one phone call away from telling his selfish children to grow up even though I'd gain nothing but more aggravation. None of them work, they all have incomes via retirement or public assistance. My friend's ex-wife comes from a prominent family and while she's up there in age too, her resources are plentiful, yet I don't see her hiring a housekeeper to help clean up her son's hoarding.

I just want to lay down and cry most days because at 48 I feel like I should be in my prime enjoying life while I still can. I have early stage kidney disease and I might not always have the stamina to do what I currently enjoy doing. I also feel trapped, I know I can walk away at any time, there isn't a ball/chain attached, but I couldn't live with myself, I am the only person my friend has.

Besides just having the chance to vent, has anyone else encountered this with the families of the people they're caring for? How do you graciously say no? Something that I suspect is difficult for many in the care-giving community.