Do you actually want him around? I mean, without thinking about the complexity of the situation? You said if you weren't pregnant you'd leave and move on.
This whole thing should have been on your terms, but once again it's all about what he wants. What do YOU think would be best for you, keeping in mind I'm referring to you specifically since you can't control his actions or feelings.