I got a lot of this before my girls' first trip. There were so many people who thought I was out of my mind...trying to tell me the perfect age and that I was clearly missing the mark and wasting my $. There were of course more positive people who didn't discourage the trip, but wanted to help me avoid any letdowns by reminding me "you know they won't remember it, so just remember that this trip is really more for you."
The last time I checked....
Happiness and Enjoyment know no age!
Memory is a tricky thing. Many people have this delusional view that an 8 or 10 year old is going to remember all of it and make it worth your while. My parents waited until my brother was 7 for us to go...it was rooted partly in not wanting to have to use a stroller, but memory was the other issue. Well, I was 11 and while I do remember parts of the trip...there is also a lot I forget. i.e. you forget things over time no matter what your age. So for that reason, the whole "they'll never remember it" logic is utterly pointless to me.
Yes, some people think about the $ and the big green jealousy monster is to blame. While some are ignorant to all of the discounts and pricing seasons to reduce costs, other people have never been (and have opinions but no first hand perspective) or went with parents, relatives, friends, etc. who planned poorly - so the mere mention of WDW makes them think "waste." Some people were deluded into thinking it's just for kids, so you'd have to be a fool to go before your kids could do everything. Ultimately, there are tons of petty reasons why people get like this, but ... it's your vacation, so who cares what they think!
Seriously, if I want to backpack with my kids through the trails of Yellowstone, expose them to all of the history in DC as infants, or go on a cycling trip in Europe with toddlers...that's my choice! And if we plan accordingly to make it pleasant and safe for all who are traveling - it's no biggie! Soooo- Enjoy your trip!!!
P.S. My younger one had just turned 2 on her first trip to WDW and she had a blast! She remembers very little, but she loves hearing stories about the trip, looking at the pictures, and the rest of us have a handful of wonderful memories. $ WELL SPENT!