We see me family quite often- they always pop in and help out. Df's family is dysfunctional- fil can't even remember dd's name and though I know he has an ABI it still stings. They see our kids a few times a year and expect them to come running up to them with hugs and kisses when the kids barely remember them. Not to mention we know the cousins on the paternal side get a lot more from their grandparents than our kids do, we don't really feel motivated to see them all that often.
Op, i agree with others. Cut back the hours, tell fil he is welcome to come over to your house and spend time with your immediate family as a priority.