We took our youngest for the first time at age 14 months for a total of 9 days, and she has now been a total of 4 times before turning 4. I've also heard of others who took their young ones to WDW and didn't have a great experience (although I think planning and being prepared may have something to do with that). So, as others have said, there is no "right age" to start taking kids to the World. I personally wouldn't want to take a child that isn't walking, but every kid/family is different. We had a blast, and so did she, on every one of our trips.
You worry about them not remembering the trip, but our DD remembered lots of details about the rides during our last trip (she was 3 at the time, and had just been a year before at age 2).
We have found that 4 days in a row of hitting the parks hard is about as much as we can handle without a break. So we always work in a mid-week resort day to just play at the pool and rest. Have a plan for each day and work in naps. Every kid is different, but a decent stroller is a big plus if yours doesn't like to walk a lot (and you don't want to end up carrying them around, which I can tell you, in the August sun is not fun...)