"My baby" in conversation would definitely bother me.
'My [endearment]"... no. It's an endearment, so yes it refers to the person saying it.
ETA: So, none of you who dislike people calling your kids "my xyz" call your partners "my love" or anything like that right? 'cause they are someone else's kid.