I hope he hasn't been trying to tell you that you can't do this on your own and that you need him etc. Because you are strong and you CAN do this on your own. Don't let him or anyone else tell you any different. You weren't going to have him at the birth or move back etc but now you are and I'm just hoping it's because he's giving you positive reasons to do so, not giving you the negative reasons why it would be bad if he wasn't there for the birth or living with you.
Has he broken up with the mistress?