Can a home care agency refuse to provide 24 hour care while the main caregiver (a family member) is hospitalized?

Yes, Medicaid determines how many hours a client receives. The agency gets paid accordingly. Maybe the doctor could call Medicaid and explain the situation.

You will have to private pay for the additional hours. Medicaid is approved for 40 hours and any other hours you will have to pay. You petition Medicaid for more hours but no guarantee that they will approve.

If the agency is only getting paid for 40 hours a week any time other than that would have to be private pay.
You might want to ask them if it would be possible to use all the hours at one time so that for 3 full days that would take up the hours for about 3 weeks. BUT you would have NO caregiver for 3 weeks. That is a long time if you will still be recovering.
Is mom on Hospice?
If so you have a GREAT benefit with Hospice. Medicare, Medicaid and most other Insurance will cover about 1 week of Respite a year for a Hospice patient.
If mom is not on Hospice you might want to check to see if she would be eligible.
Other option other than private pay for the caregivers you would need while you are recovering would be to place mom in Respite and that also would be private pay. Most facilities will do a respite stay.

Check the contract you signed to see if this kind of situation is addressed.   

Have you asked an agency rep why they've declined to provide staff for more hours?

Can you contact mom's Medicaid caseworker for advice?

Your mom may need to be placed temporarily in a NH in order to get the care she needs while you are incapacitated.