A small girl has not yet learned about social expectations and mores.
Yes I would be distressed. Why? Because in my past there have been instances of...how do I put this...males who have not respected my personal space. I'm not the only woman who's had to deal with this, I believe the statistic is around the 30-40%.
DF has taken DD into the mens toilets. He assured me she showed no signs of distress. I have taken DS into the women's toilets numerous times (and did it today) and again, no signs of distress or trauma.