I cannot think of any occassion when I have trusted my intuition and regretted it.
I can think of numerous occassions when I went against my intuition and explained away a situation or talked myself out of something and then regretted it.
Mother's intuition is strong.
What you are describing are warning signs we've all been told to look out for. It might not be obvious to everyone, but predators are usually manipulative and groom parents and children for abuse. They need trust from parents and kids and they need access in order to abuse.
Keep being vigilant! Definately set boundaries with him too so that he knows you're watching him. If you aren't facilitating the interaction, you set boundaries and its obvious that you're not playing the game, your kids are far less likely to be the target because there is no opportunity to groom and escellate the behaviour. Even if it is innocent, you're uncomfortable and the behaviour is unusual and you need to set boundaries.
I would rather be a b1tch with safe kids than to be nice and polite for the sake of not upsetting a predator.