Yeah, I get the same when I tell people I'm feeling so much better now (in my second trimester) that I'm feeling well enough to exercise. My mum told me exercise was not good for me and blamed my sister in law's miscarriages and still born on her exercising! I was livid and told her off. (It still makes me angry, so willing to blame the mum than to understand it was a genetic problem)
Most people don't seem to understand that exercising is really good for you and the baby and that sitting on your butt for 9 months is not (unless you've been told to rest of course). I think some people just like to judge and assume some kind of vanity on the mums part.
I'm constantly telling my partner I'm not doing anything dangerous but I guess its sweet that he cares.
Good luck with the hormones, I try to avoid dealing with people and I keep apologising to my partner when I'm not biting his head off!