Ok, dirty great big red flag went up when you said he wanted you to stop taking the meds because he could make you better.
Honestly, he sounds like he has manipulated you into a seemingly permanent state of dependency, as soon as you started showing signs of improvement he got nervous and told you to stop them so you stay needy of him. You can not just stop taking the meds, and certainly not after such a short amount of time.
You doctor seems to have rather large problems of his own, and you absolutely can no longer trust him to have your best intentions at heart. Get out of this situation now, as soon as you can. Tell your psychiatrist that the effexor (I think that was what you said had helped a bit) seemed to be helping but that you can absolutely no longer see your GP ever again, and let her take over the care of your meds and cut ties with GP ASAP.
I can not believe that some of the people we trust to look after the vulnerable and sick people in our community can do this, it makes my blood boil. I just hope you can see how very very wrong this situation is and can build enough support quickly to cut ties with him as soon as possible. Good luck OP
ETA: reply to OP's question. Ok, apart from in the long term you healing from your current anxiety and depression related issues, if you see this as an emotional affair that you consciously bought in to (as opposed to your mental illnesses allowing your judgement to be clouded and for you to get into this situation), then later on you are going to have to come to terms with your feelings of guilt around the 'affair' you feel you had as well. And that is just completely not fair on you. No one blames a child for being sexually abused, yet many victims continue to struggle with their own feelings of guilt even though everyone would correctly reassure them in a heartbeat it was NOT THEIR FAULT. So if you start out already assuming you were guilty (rather than just taken advantage of) then I can only imagine how much tougher it would make your recovery process.