It would annoy me a bit due to the confusion it’s causing.

Once my mum called me on the phone and left me a teary message to say “dad’s in the hospital”. Naturally I thought she meant my dad, not hers, and I was pretty annoyed when I found out it was actually my pop (and it wasn’t a dire medical situation to begin with).