What a difficult situation.
The most important thing is the safety and security of your child. If your DH won't agree not to take your child to visit, then you will have to insist on going with him and not letting your baby out of your sight. If they are horrible to you while you are there then your DH needs to stand up for you and tell them to treat you with respect, otherwise they will never get to visit with him or the baby.
I hope things work out.