I have been in your exact position, right down to the ***** who got pregnant ASAP and DS has a half-brother only 7 months younger. I have raised DS on my own from day one, my ex and I split right around the time we finally started our first cycle of IVF, to which I fell pregnant.
Basically, I felt/feel the same as you. I made it clear that I would be calling the shots with regard to visitation and sleepovers. DS has only ever had a few sleepovers to date. I am incredibly protective of DS and want him sleeping in his own house/own bed every night. Stability is so important.
DS is 3 now and I have only just started allowing him to stay at daddy's for the night occasionally, because I feel in my heart he wants to and is ready, not to mention he can actually express his thoughts in words and says he wants to - he loves daddy and everything else aside his dad is very affectionate and gentle with DS and does everything with DS when he has him. It helps that I still get along great with his family and they all adore DS and are often there when he is with his dad. None of them ever mention the s.kank to me, ever! Which also helps
Basically, having been there myself, it's been very important that I am given the respect to make the call regarding sleepovers with DS. His dad cut back to just the one day per fortnight when his other son approached toddlerhood, so I did not allow night sleepovers at all because he wasn't spending enough time with DS! I didn't want DS waking during the night and being frightened (he always comes into bed with me but I doubt he would do that at his dads!). His dad assured me he would go in and sleep with DS if he woke, but how would he know? I didnt feel comfortable with it at all. DS is a quiet child and would just lie there, he would never cry out if he was frightened, so there's no way I felt comfortable with it until now.
Anyway, lately he sleeps through and doesn't usually come into my bed until morning, so that, combined with the fact he expresses his desire to stay at daddy's means that I now feel quite fine about him sleeping there for a night here and there.
I feel that if you are feeling the way you are right now, it's not the best time to start sleepovers, and as for alone visits in the day, start eith short, 2-3 hour ines. be firm and assure their dad you are thinking only about the boys. One good thing is that the boys have each other when they go there. Message me if you want to chat about anything else - I've been exactly where you are (I'm there now!).