I would be absolutely livid if I came home to a screaming child and my partner told me he just 'didn't bother' feeding him and had left him to cry alone. I wouldn't be able to trust him with our baby anymore if he ever did that.
I don't buy into the whole "well, it doesn't come naturally to dad's so we should cut them some slack" argument either. Partners should be involved with their child so they know what's what and how the routine goes and what to do, so that they can cope with the baby alone. It's called being a parent. Fair enough if it doesn't come as natural to them or takes some time to figure out but if they put in the effort they will learn.
And in the OPs case...his argument was that it takes longer for the baby to settle when you pick him up...seriously? Longer than it would take if you just left him there, hungry, to cry by himself? Yeah, right.