I completely understand. I had to leave DD every night for 102 days while she was in the NICU, and its caused some control and anxiety type issues for me now. I'm fine with my mum coming and watching my DD here, even if I go out for a couple of hours. But the thought of her in the car being driven to my mum's house to be watched for the same length of time makes me VERY anxious. I know it's probably not rational, but having gone through the trauma of a very long NICU stay, it's going to take a while.
Baby steps! Maybe try leaving him at your mum's when you're only 5 minutes away so you can "hold out" as long as you can and then be back really quickly, even if you only ended up being gone 15 minutes?