I agree with others that it may just be that he isn't as interested in some of the activities you have chosen.
Also even though he may have been swapping houses between you and your ex for some time now, he may be going through some issues with the exchanges and learning to adjust to this as he gets older and more aware of how the world works.
Some suggestions that may help (if you don't do so already)
Let him take as much time as he needs when he arrives at your place to settle in, go to his room and feel comfortable with the change of houses before beginning any major conversations or discussions on what the day holds.
Just give him some quiet time to settle and let him know when he is ready he can come and have a chat and hug etc.
(A little like when you arrive in a new destination on holiday, you just want a minute to put your bags down and gather yourself before you jet off to a day trip etc)
Perhaps ask him to help you write out on small note paper some ideas and activities you can do together for when he comes over. Just some ideas..
Movie day at home
Kicking the footy
Planting herb garden
Making a 3d model
Going for a walk
Go out for hot chocolates
Shopping for clothes
Playing wii games
Visit to library
Watch the planes come into land
Visit pet store
Wash the cars together
Collect bugs in a microscope/bug catcher
And of course some of his ideas on what he likes to do and has fun with.
Then put all the notes into a jar. Each weekend before he goes home, he gets to take one note out and that's something you will do the following weekend he comes over.
Theory being that he has a bit more ownership of what's coming up since he helped to plan and chose the activity as well as gives him plenty of time to prepare himself (and hopefully get excited) for what's coming up.
Hope things start to settle for you so you can all enjoy your time together a bit more!