How can he not see how difficult this has been for you? I am just so amazed at his behaviour, it's awful.
I could tell you what I would do in your situation (or what I would like to think I'd do) but you're the only person with the whole picture so it's entirely up to you. Just make sure you get yourself into counselling ASAP, either together or by yourself. This sort of experience can cause pain later on if not dealt with now.