We took DS1 to Europe when he was 2 years 3 months. We had a great time. He's four now and still talks about it often.
Unless we decide to expand our family first we'll be going to the US next year when DS1 will be 5 and DS2 will be about 19 months.
I think it depends why you're traveling and if you intend to keep traveling. For us the trip is for all of us, not just the kids future memories and we want to head overseas at least every 2-3 years. We will put some trips off (such as to more difficult destinations or museum hopping trips) until they're older, but if we fell in love with somewhere we'd be happy to return anyway