I see both sides here. On one hand, it was an inappropriate text 4 years ago. I think you either need to decide the trust is gone and walk away, or work at trying to trust him.
On the other, I would question why he needs to get trashed every business trip? Not only does it make him look bad in front of work colleagues, it tells me he lacks self control. Also given he knows how anxious and upset you are, he should be trying to reduce that anxiety not increase it. Like answering the phone at 9pm. My Dh doesn't go away much but when he does attend conferences there is no getting blind. He wants to get on the soonest flight home to get back to us.
Has he made contact this morning? I'm also wondering if there is other questionable behaviour going on to make you feel he's still untrustworthy?