I have been a couple of times that week, it is when my kids finish spring semester in college. I find it to be a great time to go.
Yes, public schools are still in, the spring breaks are over, so crowds are relatively low. Wait times build up during the afternoon at major attractions, and Toy Story Mania will always have a long wait, but overall wait times are minimal. I have been able to walk up to Table service and get a table with little wait too.
Weather, obviously, varies day to day, and year to year. Highs are usually 85ish. You get afternoon showers rolling through, like most of the time. Comparitively, it is one of the drier months statistically. I have been there in June, July, August, and Sept., and I find it to be cooler than all of those overall (and smaller crowds other than Sept.). As far as comfort goes, I guess it depends on what you are used to. I live in SC, so 85 and humid is a cool mid-summer day to me.
I would go back again that week this year, but my AP's expire before then, so I had to move it up a couple of weeks.