As others have said, I think it depends on context.

I wouldn't mind if DH said it, as it would probably be in a funny way (and vice versa).

If I picture someone not so close saying it, then I hear it said in an uppity schoolmarm-ish way - so I probably would be offended to be on the receiving end of that.

If I asked a friend/ acquaintance a question and they replied with that (and no further explanation) then I would be offended big-style. I'd be pretty embarrassed, and think "Oh no, they think I was prying!" I think it would be blunt verging on rude to say that with no other explanation.

If, in the same situation, they said "Oh, sorry, that's a bit personal" or "That's not something I want to to talk about" (said gently) then I would be fine with it.

"That's none of your business!" immediately sounds like a telling off (and an exclamation) to me.

But DH could say it all he liked, and I would just laugh at him