I can see that this really strange contact from your ex-bf is really playing on your emotions. I really have no advice or no understanding of this - but after DD was born I kept getting contacted by an ex. Things had ended very poorly for us and I didn't understand why she wanted to see me, but I went along as I felt I needed to explain the end of our relationship (I ended it). I definitely got some closure from this meeting. To me, if you don't want to get back together with your ex, there is nothing wrong with meeting with them. DP is really good friends with an ex.
I hope that you can work out these feelings. I can't imagine it feels good walking around carrying this weight with you.