A huge Congratulations! Sounds like you have a lovely little family with your kids and your partner. Who cares what other people think. Would you rather stay in an awful relationship just so people don't talk about you??? People will always find something to talk about.
Live your life for you,,,, not any one else
Sounds to me like you have a great future together as a family.