You poor thing!!! it's not nice to be treated like that by anyone. If my DH told me to duck off, he would be on his first and final warning.
He's not working, not looking after his son, doesn't want to spend time with either of you and speaks to you with such disrespect. That's not a relationship.
I would seriously consider moving to your parents place.