If he's 3, I would start trying to casually ask him how he feels when at Dad's place, what do they do together etc
If the grandmother and Aunty are awful to him, that's not a nice environment. I just hope the Dad gives him the love and security while he's in his care.
I agree with PP though, I wouldn't be happy with unsupervised visits until my son was comfortable. Especially no overnight visits at the moment either.
It's tough OP as your son deserves a relationship with his Dad. But if its not providing him with a safe or happy experience, then it should be changes or stopped.