Good on you for refusing to allow him back home! It's great that you're standing up for yourself and letting him know you won't take that kind of behaviour.
It sounds like you know what you want to do with the pregnancy. If your DH decides to walk away that's his problem. It certainly sounds like the marriage is in need of serious help regardless of what you decide to do.
I know everyone has said he's terrible, but there must be some good qualities there too - I hope he can get some help to deal with his issues and become that better person. It's wise to make him do that alone before allowing him back, so I think you're handling it really well despite how stressful it must be for you!