I guess it depends on the situation and the length of the stay. If it's for a one week holiday either side of your due date if be inclined to say no. But if they're coming just to see you and only for a night or two, that would depend on how close you are with them and how you're feeling.
I'm facing this dilemma a bit at the moment. Dh's family are really understanding and aren't coming to visit (they live two states away) until two weeks after my edd, and they're renting a holiday apartment for their stay. I know that they'll hire a car, be super considerate of not hanging around too long on any one visit and call before coming over.
My parents on the other hand are planning to come (from ten hours away) and stay immediately after the birth for "a few nights". They have said they'll just head this way as soon as they get the call I'm in labour. They're really uncomfortable about coming to the city and I know they won't do anything other than just hang out with us. I'm feeling like I don't really want the added stress of having people around while I'm learning to be a mummy and recovering from labour, but know that my mum in particular would think I was being incredibly rude if I said I didn't want her to come/stay with us, no matter how I phrase it. :/