I don't think it's selfish You have given them your colostrum, which has helped them recover from birth, gotten their digestive tract running smoothly and given the first immunisation. The first days where you are getting your milk in are so so SO hard... so congratulations! Celebrate what you have accomplished
However, in line with PP and going from my own experience... how will you feel about this decision later? Can you get extra help from family for the next few weeks? Or will you be happy knowing you gave it the best shot you could at the time?
If you don't think you can continue with how things are going, then go for it! You have to look after yourself first and foremost to be able to look after anybody else, so do what works for YOU.