I do disagree that his gender matters here- he didn't do anything to you, he didn't yank the curtain aside to ogle you, he went into the toilet for a few minutes, then left. Men are just as entitled to enter this room- it's a 'parents' room, after all, and although we don't know his reason for being in there, we can most likely assume it was to do with his job. If he'd come in and stared at you while bf, fair enough, but he didn't approach you.
Just as an aside, many women feel breasts are sexual as opposed to nourishing too. I was kind of horrified to realise this. But it's true. If you go through the comments on news articles to do with bf in public, there is a disturbingly large number of women banging on about being covered up for that very reason.
Funnily enough, the last time I used a parents room, I was alone in there with a man. I didn't even think of it until now. He was in the bf "booth" next to mine, fixing something on the light. I was more concerned with the giant chair in the tiny booth that meant my knees were sticking out and that I had to sit in the booth directly opposite the toilet as the electrician was in the other. Stiiinky! I only went in because I wanted a more comfy chair for my bad back.