For crowd levels (in order):
Late Jan-Early Feb
Late Oct
Early Sep
Early Dec
I have never been in Dec, and would love to go because everyone says the decorations and special events are really great.
I absolutely loved the Food and Wine festival!
I have found that going at the times when the crowds are really the absolute lowest means that the park hours will be the shortest. So as a general rule, just before and after the times listed above are ideal for me (avoiding Christmas and Thanksgiving). The parades are better, and the parks are open late!
And I know you said forget work and school, but our favorite times to go when we can actually go are Passover, when it doesn't coincide with Easter, and the week before Labor Day, when the kids in the NY Metro area haven't started school yet, but most of the rest of the country has.
Once again forgetting about that, I've been most every time of year except Christmas, and the only time I would hesitate to go is Mid-May to Mid-August (especially mid-June to the end of July), Easter Week, Christmas Week, and Thanksgiving.
Every other time of the year is fabulous. Heck, I've been in June before and it was great. In fact anytime you're in WDW is great! It just depends on what you do with it!