I'm not sure if DS was saying it at 4 but he definately says it at 7! Than again, he has 2 older cousins, and a few older kids through friends so I wasn't overly surprised.
It's definately not nice and a bit confronting to hear, but I don't think it's anything to call a physchiatrist over. Sometimes DS will get rather frustrated and upset at his cousins and say "I wish they were dead", or sometimes they might say "I'm going to kill you" when playing with each other.
Sometimes I ignore it, other times I might give DS a hug and explain that I understand that he's angry but that's a really hurtful thing to say. If there mucking around than I might say "Well, that's a little drastic- why not just tickle him" (and then proceed to grab or chase them and tickle them). Kids think in the spur of the moment, they haven't quite got the grasp of mind your tongue.
I wouldn't make too much of a big deal about it. There's plenty of time, and plenty of words/phrases that you'll want to ban altogether.