Kids benefit alot from sport! I don't understand why you think he wouldn't. Especially team sports too.
Your post does come across as kinda selfish, sorry. However, I personally think that at 4yo an organised sport like that isn't necessary, so I get where you are coming from about being a bit of a waste of your time. Does your DS start school next year? If not, I would tell ex that you don't feel he would benefit from an organised sport until the following year. I did that with my ex, so my youngest has got until next year sport free (excluding swimming).
I also feel your pain re. ex's decision but you're the one doing it; my eldest plays soccer, my middle child will probably play netball like she did this year, and my youngest my ex insists he plays rugby league. Ex is a shift worker. So that's 3 kids, 3 midweek training sessions, 3 Saturday morning games at 3 different fields!! I have explained to ex that he will have to figure out his work schedule to be able to help if he wants DS2 to play a different sport to the others!