I haven't read all of the other posts but some of them I have I totally agree with. I do not blame you OP you have been prayed on by a predator. That's what this man is a predator using his position to benefit him. Dr and patient should not be casually meeting up in the first place that's your first red flag.
I won't go into it but I know a lot about emotional intelligence and behavioural psychology and this is a classic sign of abuseing power of authority. He has been grooming you like a paedophile would groom a child… It is actually making me feel physically ill you are defending him which is what most victims do. You are a victim. He does not have your best interests at heart. You need to report this man.
Put it this way if you were your daughter would you be happy about this situation. I always use my own children as a gauge and if I would be furious at this with my own children then I wouldn't put myself in the situation either. People like this pick people who are easily influenced.
Ultimately you are a grown woman and you can think for yourself and I or anyone else can't stop you but this is wrong. As a single mother my self I see it in my absolute power to avoid people like this because my children are precious. I would not have a human being like this around my children. And if you say it hasn't got that far what do you want or expect? If you date he will eventually have to be in your children's life. I have a fierce fierce mumma bear instinct and I am getting NO GO Zone here. In fact akan like this would never try me because he knows I'd tear him to shreds if anything happened.