Depends on what you mean by 'baby'
Obviously I agree 3/4/5 mnths is WAY too young for chocolate/ice-cream/hot chips.
I stuck to food guidelines till DD was 1 (except I gave her peanut butter, cheese, eggs etc all earlier then recommended) her first 'naughty' food was one bite of chocolate at Easter when she was 10 mnths. She also had 1 maccers nugget at 15 mnths old.
I'm pretty relaxed now. I feed her what I'm eatting (this makes me eat healthier too)
Today she had weetbix, a boiled egg, a ham & salad whole grain rap with seeded mustard, a strawberry banana smoothie, rice with sweet n spicy chicken & veg. She's 19 mnths.