My DH has low count and motility too. I also have poor quality eggs to add to the mix and we have conceived naturally twice, resulting in one ectopic and a beautiful DD.
There is nothin to say you can't conceive, it's just that your odds are reduced - it's entirely up to you how long you are willing to try naturally. With age on your side I probably wouldn't rush to IVF, but it is still good to keep in contact with your Dr and make sure everything else is on track.
Good luck, sending lots of baby dust your way