It's ~60 miles from Port Canaveral to WDW (depending on where in WDW you're going). Yes, it can be as little as an hour, but it can also be much more. If you're taking the Beachline there are very few exits for long stretches of that road and if there's an accident then it can be a lot longer.