I agree that the child is likely quite confused and needs support. My point is that the OP shouldn't be feel bad about bringing up her concerns with everything to her DH and she's not being selfish. Maybe the relationship between the OP and her DSS isn't mature/developed enough yet for the two of them to be alone all weekend - that's not fair to the DSS or the OP.
My judgement may be clouded here as when my younger DSS was 4 the thought of looking after him all weekend would have sent me into an almost panic attack. We didn't know each well enough, I had never disciplined him, he wasn't well behaved and I simply wouldn't have known how to manage - it would have been horrible for both of us. I'm picturing this being how the OP feels now and I have sympathy. Fast forward a few years and now I'm more than happy to look after DSS.