He's your Dr. There is no 'outside the Dr-patient setting', or outside the Dr-patient relationship. OP, run, don't walk, away from this one. Any s!exual or romantic relationship would end in disaster. If you care for him, the best thing you can do for him is forget about it and find a new Dr.