My mom started taking me to Disney World when I was 3 years old (actually 2 years and 10 months from what I gather). From what I remember, and from what my mom told me, I had an absolute blast even at that young age. I definitely remember constantly wanting to go back (and perhaps spoiling me somewhat, I was obliged that wish quite often). Hell, I even adored Epcot as a child (including the educational aspects). Maybe I was a weird kid though. I was exposed to museums very young and grew an appreciation for "edutainment" so to speak. But I wasn't conditioned into liking these things, I just enjoyed it.
I don't know how mom felt about having to deal with me, she seems to have been fine with it. Sure i'd get tired and would go back to the hotel to take naps and such, but overall I think we both had awesome experiences even at my youngest.