Definitely agree with what others have said. Feed more frequently during the day, 4-5 is a long time to go between feeds for a 5 day old. Wake him if you have to (I'd normally say never wake a sleeping baby but if he is not getting enough milk he needs to be woken up for frequent feeds). Stop giving top ups and feed more often instead. When you top up with a bottle feed you are encouraging your baby to go longer without boob which means you won't make enough milk, and you also introduce the risk of nipple confusion, which means bub won't know how to suck effectively on the breast because it's different to the bottle.
Don't use the amount you can express as an indicator of how much your baby is getting. You will never get as much milk with a pump as what your baby can get from feeding. He is probably getting more than you think.
As long as he has wet nappies (around 5 a day) and is pooping fairly regularly there is no need to stress. Just put him on the boob as often as you can to encourage milk to come in.