It sounds like you are still currently a patient. it's just the nature of being a gyno means you only see him sporadically. I'm putting myself in your place and if I got involved with my dr deep down I would be thinking if he was happy to cross clear moral and ethical guidelines to date me, what other morals would he break? I kind of liken it to dating a person you've been cheating with and wondering if they cheated with you, will they cheat on you?
Why do you have to tell him? People change Specialists all the time for a million reasons. You are the consumer, you don't need to explain why you want to see another gyno.