Our son was born May of 2006. On our April 2007 trip and our April 2008 trip, we left him home with the grandparents. Last April was his first trip ever to WDW. However, we didn't go in 2009 due to my wife being pregnant with our daughter, who was born in June of that year. So if we would have gone in 2009, we would have brought our son (who would have been almost 3 at that time). This year, we gave serious thought to actually bringing our daughter with us for our upcoming April trip, and she will be 2 in June. But we ended up deciding to leave her home again like we did last year, if the inlaws are still willing to keep her. We drive from Illinois, and our daughter is still at that age where she can't talk and can only express herself by crying, and she has a tendency to get something in her craw and cry all the way to church, all the way home, etc. And we don't relish the thought of listening to that all the way to WDW. She may get upset that her toy fell on the floor, or maybe that she can't get her shoe off, or whatever.
Last year, when we took our son, he was old enough that he is out of a crib, and he was able to sleep on a bed just fine, either in a seperate bed by himself, or in bed with us, and most importantly, he was old enough to walk on his own (although we pushed him in a stroller alot in the parks), and he was tall enough for all of the 36 inch height requirements. So basically, we felt like he was the perfect age.
Our daughter, on the other hand is 34 inches tall, so she doesn't meet most of the height requirements and we don't want to have to skip rides or do the rider swap thing. We don't want to have to worry about setting up a playpen for her in the room and hoping that she can sleep with the sounds of the rest of us in the room, the TV, etc. And now that I have the itinerary all planned based on being able to be out until midnight, get up for AM EMH, etc, I don't want to have to redo the plan to accomodate for a toddler. I'm just personally of the mindset that if someone else wants to take their infant/toddler to WDW, more power to them, but it's just not for me. I prefer for the child to at least be old enough to comprehend what's going on and to be able to take care of himself to some degree. Obviously, there were things we had to do differently and slow down on with our son last year, as opposed to the trips with just the two of us, but they were changes that I was able to live with, vs the kind of changes that would be required when travelling with an under two year old.