It didn't sound harsh, I could see where you were coming from. DH is at the point where he has just drawn a line in the sand and said he's not thinking about it anymore on Fathers Day. He knows now not to out any expectations on BIL.
As you can tell I'm not quite at that point. We have done so much to help him out. He wouldn't be living in the house he is if it wasn't for us.
As for saying something, I won't be doing that. I'm not a confrontational person. In fact the closer I am with someone (with the exception of DH) the less likely I am to say something if something has bothered or upset me.