Thank you so much to everyone who has replied. It's so good to hear your thoughts on it. Of the people close to me, they seem to be divided between shrugging it off as 'nothing' and not even thinking about it, or else being massively opposed to the whole thing (often for reasons that DON'T bother me)!
Rocky - the FS said potentially 4 left over, but as others have said there could be zero. It's really unknown until you do it. All depends on how many eggs grow, and how good those eggs are.
Buttoneska - that's exactly how my DH sees it too. He also pointed out that the contraceptive pill stops a fertilised egg from implanting (if one is released), which is not too dissimilar.
Thank you to everyone else too. All very helpful thoughts. Reducing stress/ thinking about it later is probably a good idea, but I didn't want to leave myself in a situation where I didn't know what to do. You're right that it might not even be an issue though.
I've booked an appointment with a psychologist through my obstetrician (they routinely offer them as part of IVF). I'm going to that tomorrow, so will hopefully have a chance to talk things through. At the moment it's all feeling a bit bleak, but I'm sure that we will come to terms with it, and move forward more positively.