People just say things for the sake of saying them and most of the time they aren't even remotely aware of the impact. I wouldn't dwell on it. If I had a dollar for every time someone said to me " oh you have just the two then" " why do you have such a big car when you only have two children " " gee that's a big gap between your two children". Different example but same sort of thing - people love passing comment and don't realise the effect on others. You just have to suck it up.