My brother gave my dad his keys back. If he has an accident and hurts someone, can the family be sued?

Anyone can be sued for any reason. The jury or the judge would decide that.

Frankly, I wouldn't concern myself about a lawsuit. How would your dad or your brother feel if your dad killed an innocent person or caused them grave harm? I get it about feeling the freedom being able to drive wherever and whenever; however, if there's a reasonable chance he could negatively impact others people or his life, then it must change.

I hope you impress on your brother and your dad to take the keys and come up with various and convenient ways for your dad to travel without his driving. Immediately.

Another thought, but this is beyond my legal knowledge and would require a determination by someone familiar with auto negligence:

Years ago there existed a legal concept known as "contributory negligence."   I don't recall the transition, but I recall that over time the issues changed and concept changed.   

If your brother was considered to have influenced or facilitated your father's driving, it might be considered contributory negligence.  However, this is just an observation.  You'd have to consult an attorney with experience in this field to get an up-to-date opinion on potential legal issues.

Whose name is on the title to the vehicle?   That individual would be the target.
Do you know if there are any "additional insureds" on the vehicle insurance?   They could also be targets.  

Assuming that your father isn't subject to any conservatorship, I think the only other individuals might be anyone with authority to act on your father's behalf, such as a proxy under a DPOA or POA.

If there is an accident, injury and suit, plaintiff's attorneys are going to target those who have potential and actual liability, and most certainly are going to look to the insurance coverage to see how much is available for any judgment or settlement.   That's how "additional insureds" could be included as parties defendant.

I would not wait for a doctor to do the reporting to the DMV (which doc: his ophthalmologist? Neurologist? GP?) You go online and do it anonymously, stressing his macular degeneration and any other issues such as getting lost due to memory problems, etc.

Then the DMV sends him a letter telling him to come in for a test (most probs an eye test). NO ONE should take him to this appointment, including your brother, neighbors, friends, church acquaintances, etc. If he did get there he will most likely fail that test and his license will be kaput. If he continues to drive with an expired or invalid license then remove the vehicle completely if you are his PoA. My uncle killed his own wife and injured people in the other car when he went through a red light because his children didn't want to confront him with this issue.

If your brother is POA then he may be liable, but generally they will go after your father, and everything he has. The real problem is that he may kill someone. You need to inform the doctor, and ask that he inform DMV; he has ways of doing that; let him know that his not doing that leaves him open to suit, understanding he has an incompetent driver on the road. Doctors are supposed to ask for the license and turn it in and that happened to my brother's ex. Doctor asked to "see" the license and then that was that.

His eye doctor should have reported him to DMV. Yes, report him and do not feel guilty. You may save someone's life. Just like drunk drivers, they seem to survive a crash but not the person they hit.

Our laws need to change. Drs need to be held responsible to report. Everyone is so worried about taking the license away. Its a privilege not a right. A car is a weapon in the wrong hands.

Dad would be sued and possibly your brother for doing such an astoundingly stupid thing in enabling Dad.

"The family?" Probably not.

Macular degeneration robs a person of their central vision, i.e. what's right in front of them. That alone is reason enough to stop driving. The whole "I only drive to a few places I know well" excuse is irrelevant if that's being used. So did my mom with MD -- she only drove to her church three blocks away, but she did it during the week and there were two schools between her and the church. We put a stop to that immediately.

What in the world is your brother thinking??

YOU contact the DMV to report an unsafe driver, and YOU contact Dad's insurance company to report him as well. Maybe point out to your brother that if Dad's sued for driving and killing someone, there won't be any inheritance money for him later on.

Your father can be sued and lose everything he has.

Make sure you tell brother that HE's the one who will be caring for demented dad with no assets.

Here is how to report an impaired driver in Wisconsin:

https://wisconsindot.gov/Pages/dmv/license-drvs/mdcl-cncrns/reportingunsafedriver.aspx

Consider calling your father's lawyer and asking for advice.