As others have said, I think it depends on context.
I wouldn't mind if DH said it, as it would probably be in a funny way (and vice versa).
If I picture someone not so close saying it, then I hear it said in an uppity schoolmarm-ish way - so I probably would be offended to be on the receiving end of that.
If I asked a friend/ acquaintance a question and they replied with that (and no further explanation) then I would be offended big-style. I'd be pretty embarrassed, and think "Oh no, they think I was prying!" I think it would be blunt verging on rude to say that with no other explanation.
If, in the same situation, they said "Oh, sorry, that's a bit personal" or "That's not something I want to to talk about" (said gently) then I would be fine with it.
"That's none of your business!" immediately sounds like a telling off (and an exclamation) to me.
But DH could say it all he liked, and I would just laugh at him