I was told the same. You can wear it all through the pregnancy, but I was also told that my doctor found most women felt better in pregnancy as the uterus pulled every thing up as it grew. Mine felt worse with DD2, but I didn't have a pessary in at that time as I wasn't seeing my doctor back then.
OP... My doctor also said that you keep it in during sex and that you don't have to take it out when TTC. Did your doctor tell you to have it out while TTC or do you prefer to have it out?