We thought about doing it last summer and are glad we didn't.
My parents are big NASCAR fans, and when we spotted someone in a "Daytona 500" T-Shirt, we stopped and asked them how the race was.
They told us the race was great! But not great enough to make up for the terrible time they had getting back to WDW/AKL.
They had set aside one day to travel to Daytona and see the race. Everything was fine until the race was over. They said they, and everyone else, waited at least an hour, bottlenecked, to get out of the stands and to the parking lot. When they got to their car, it took another half-hour for them to get on the main highway. And then another three hours to get onto I-4. From there, traffic was heavy and it took another hour to get into Orlando, and then back to WDW. They said that they didn't get back to WDW/AKL until 4:30 the next morning, and it ruined the next day for them, as they spent a good portion of the day sleeping. They said they would never do it again.
It's up to you though!