What you are feeling is normal. You have a child with him and spent 5 years with him.
You can do this. You were on your own anyway, he was never around physically or emotionally. Now you don't have to walk on egg shells trying to please him, like everything is your fault.