Elderly father's attraction to caregiver, should I be concerned?

Sounds a bit fishy to me.

When I was caring for my FIL, I specifically DID NOT oversee his showers, b/c he did have some sexual proclivities of which I was only too aware. I didn't want "that" to be my takeaway from caring for him, nor as his DIL.

I made sure he showered and he was capable of showering himself. He didn't like bathing, ever, but when the female aids came, suddenly he was all for it.

We did have a couple of his female aids help him shower and they told us he was not always appropriate in his behavior. After that, only the male aids would help with showers.

I would be disgusted too. Try saying this, "Dad, you're married! It almost sounds like you're attracted to her. Mom wouldn't be happy at all if that's the case. Try saying more nice compliments to Mom. She deserves it much more."

When I learned to give bed baths in a home nursing segment of home ec class, we were told to give the washcloth to the patient and let them clean their private areas. It was also added that if they weren't capable, we should do it for them.

If dad is capable, instruct the caregiver to make sure he cleans his own private areas. She probably doesn't much like doing it anyway.

A younger, attractive woman has her hands all over him when he's in the shower, it's easy enough to understand how he could mistake those cues as sexual interest, and if she's got a personality he clicks with too it's small step to something more than that. I'd have a frank chat with the caregiver that he's getting the wrong message because I'm pretty sure that's not her intent, you should certainly insist that its best if he's expected to shower with minimal assistance.

Why is the caregiver giving your dad a shower if he can shower himself?
You did not give any indication in your profile what level of care either of your parents need. Any ADL's that they can do themselves should not be done by a caregiver. (Unless they are not doing them well.)
If your dad has been diagnosed with dementia sometimes oversexualization can happen. It is not your dad it is the disease. The caregiver needs to enforce boundaries. If you find that your dad is getting "handsy" and the caregiver is not correcting the situation then I would have a serious talk with the caregiver. If it continues I would look for a replacement.
The last thing you need is a "sexual harassment" lawsuit to deal with.

The OP is probably not at home those 3 days

Speak to this caregiver about what your dad told you. Could be he's fabricating it a bit.

If your dad showers himself on other days, why does the caregiver shower him? Did you hire the caregiver and assign shower duty? Personally, I would shut it down, but that's me.