I'm outraged on your behalf that anyone else outside of your marriage thinks their opinion on how many kids you should have is even worth paying attention to. also, your dh should tell his family to go fly their kites elsewhere. Jesus h Christ, since when did how many bubs you pop out have any bearing on anyone outside of the two of you?
we're only onto our first (due next month) but I agree with the other posters, it's entirely up to the two of you how many you have. it needs to suit your budget, your lifestyle and ultimately what you think you can personally cope with.
we're personally leaning towards two. maybe consider a third once we have 2 and see how we feel. I am not so sure I could cope with 4 but hats off to all that can and do. it's a personal choice. tell BIL to shut his gob.