Don't worry about it. People will always find something that makes them feel inadequate - I'm certain they're just a little envious that you can do something that they find difficult. I am one of those who find it difficult, but if I was a friend of yours and knew how well your house was kept, I'd be thinking "good for you".
In saying that, I don't think people with a messy home are necessarily more lazy. It may be just that they don't have as much time, or as much control over others in the house as they would like. There are different situations that can dictate how messy or tidy your home can be.