When do you throw in the towel?

I have been friends with Richard for nearly 20 years. In 2018 he gave up his house and moved in with his youngest son and family. That turned out poorly. In August of that year he asked me to start coming over to help him at their home. By November they had asked him to leave. We moved him to his current studio apartment, where I have continued to help him. When he first moved here he was still somewhat independent, got around with the aid of his rollator, could still walk over to check his mail. Slowly he is getting weaker, and now he is having memory problems. This past week has been very stressful for me. I do not trust the adult son's wife, but someone besides ME needs to step up and start managing his bills and such. Up to now, he would have me login to his bank, use bill pay, or give me his ATM card to go take care of things like groceries. I was fine with that, but Monday he gave the card to the son "for safekeeping" and then forgot he had done so. He wanted me to pay his phone bill, but could not find the card. I went through the whole place including trash, only to have him text the son and ask where the card was. Son just left after dropping the card off.....I feel like he is losing trust in me, and may begin accusing me of mishandling his funds. The son has already shown he can't be trusted, when their Mom died she left money for all 3 sons, two of whom are mentally challenged. The youngest was appointed trustee, and somehow managed to get ALL of the funds transferred to himself, using the money to put the downpayment on his current house.