I worked in child care for many years and have seen this a bit. My advice would be possibly changing the days to maybe a Monday/Thursday if you can, that way its not such a big time gap from one week to the next.
When you drop him off, how long do you stay for? I find with clingy children it is honestly, and as tough as it is, to almost do a drop and run. At no point let him see or sense you anxiety about it (if you are doing this already, then awesome!) always talk positive with him, if he doesn't eat say, "oh well, maybe you can try and see if you can eat more for mummy next time." and same goes with the toilet.
In the morning before care, let him know whats happening (even the night before), stay positive and smile. When you get to the centre encourage him to help you find where his bag goes, his lunch, drink bottle, etc. Even have him show you what his favourite thing to do is. See the carer (just to make sure they know you are about to leave) and tell him you are going and you'll be back later, don't linger, give a kiss goodbye and go.
When you pick up in the afternoon, maintain that positive vibe and praise him for having a good day and talk to him about the awesome art he did and what not.
If you continue to do this on a regularly basis and don't steer from it, it should really help. As a mum I know how heartbreaking it can be seeing your child in distress, but it's very important to stay in that positive routine.
Also in regards to the swimming lessons, are you able to leave the room while he swims? I know at my school when one of the little boys was getting quite anxious about it mum would leave for the lesson and it took him a few lessons but he ended up settling down? could be something to try?
The fact that you feel comfortable with the centre and carers is really important, and hopefully you little boy can start to take in some of that positive energy too.
Don't feel alone in this though, it happens to many kids.