I think you should leave now too. You already don't feel good around him I'm sure you tense up as soon as he comes home. I know I would. That's no way to live.
You do a lot as a mother and you never clock off. You are doing enough. More than enough.