I think this is a best possible outcome to be honest.
He keeps his job.
You get a new doctor you feel comfortable with.
No messy dating hassles - you can focus on yourself, your 10 month old and other 3 kids for now. Imagine how freakin messy your (and your kids) lives would have become if your doctor had taken the bite at the cherry.