Thanks for all the replies, I'm glad to see so many people who feel the same way I do. I think I'll start mentioning from now that we're not going to have any smoking in the new place so it doesn't come as a surprise. DH and I used to be smokers and we smoked when we first moved into this house which is why our friends I guess find it natural to smoke around us, but we've since quit and I'm quite repulsed by it now and can't believe I ever did it, so I'd really like to stop it happening at the new place.
To answer a PP, in my case if someone's come around during the day when DD is awake then when they go out for a smoke I stay inside with DD. If they're around at night and DD is asleep we're usually all outside together on the patio, in which case they walk a little further into the backyard and I hang back.