Your DH obviously doesn't really know much about the test then? It's not only for DS but also a few of the other syndromes like Trisomy. It also doesn't give a definite yes or no, just a ratio of risk. I personally don't see the harm in getting the test, for peace of mind and also more for the other syndromes.
I'd just do the test, and I'd tell him that you are carrying the baby in YOUR body and it's not up to him to 'let' you do things to your own body.