'He also always puts his hand on my knee while talking to me, and hugs me when I see him'
So he has already crossed the line in a major way. He has willingly and knowingly breached his own medical ethics.
I just don't know how you will be able to trust him with his other patients once you are dating him or married to him down the track. Especially when his job involves ladies intimate parts.
It's also surprising that both of you are willing to flaunt the rules and risk his career. Hope it's worth it. Would you like him as much if he had to go and stack shelves?