I think you'd be fine! I didn't know I was pg until 6 weeks (not until after late AF) so I was still playing a full 70 minute game of hockey every week plus training until then. I also did a 25km bush hike one weekend with friends before I knew!!!
I've continued going to the gym right through pregnancy, trust me when I say your body will let you know when you need to slow down! At the moment at 36 weeks I do about 30 minutes walking on the treadmill at 4.5km/h or do about 45 minutes in the pool (slow swim or walking laps). It is seriously all I can do now!