My midwife told me I'd be more likely to be hit by a car then suffer the effects of listeria.
Its all about using good judgement. If you've eaten it already, there's not much that can be done. The risks are extremely low but there are risks. It wasn't too long ago that there were sandwiches on a domestic flight that were contaminated with listeria. Two pregnant women on that flight went on to have their babies prematurely. So it happens.
A lot of the foods that may carry risk can still be enjoyed piping hot or if prepared at home. I'm addicted to camembert cheese so I have been baking it in the oven till it's steaming and dipping bread in it.
I'm currently pregnant with #3 and have definitely relaxed a lot but if I remember, then I usually avoid.