OP how old is your baby?
I understand what you are saying, especially if your baby is quite young.
The first few weeks/months is like a bubble, I felt I wanted to only be with my
micro family. My mum came to visit for 2 weeks when bub was 5 weeks old and it was way too early. I wanted her to go away and leave me alone with my baby.
this time is just so precious.
So that's even with my own family...
Don't stress, things will be easier when bub will be a little older and less dependant.