Nah, I don't buy it. My parents started taking me at 4 years old and I vaguely (very vaguely) remember it. We went every other year after that and the most vivid, lasting, hooking me on Disney memories came when I was 6 and 8 years old. The trips after were great too, but the magic happened between those years, not before. This is personal experience. Yes, I've been lucky enough to go ALOT in my life, so has my brothers who started going before me, and they will say the same thing.
Besides, what's the point of bringing a little one that can't go on 95% of the rides and to go through the hassle of dealing with diapers, feedings, strollers, strollers, strollers, puke, feedings, crying, napping, etc etc?
Hey, more power to you if you have the ridiculous patience that that will require. But hey, my kids aren't going till their 5 or older. Until then, there will be a happy grandma ready to stand the kid duty for a week.