I know a few people who have done this. Whether it is worth it would be something only you could determine for yourself. For many it's not worth the risks, for a few it is. I will say the few that I know that have done this did not do so in the Sea Racer boats or other boats you can rent. The problem with these boats is you will have to tie it to the dock and even if no one spots you docking it it will eventually be spotted while your enjoying your visit to the island.
There are some risks you should consider:
First, it is most definitely not allowed and as has been stated you could likely find your self leaving the property via an Orange County vehicle.
Most of the docks and walkways are in disrepair, you could potentially fall through or otherwise hurt yourself.
There is an overabundance of wildlife. Alligators, Snakes, spiders and insects.
The payoff if you do go is a pretty interesting experience, and everyone I know has brought back some really neat mementos.
My suggestion would be to first try out the walkway around Bay Lake to the east of Ft. Wilderness, this walkway while not specifically off limits (at least the last time I was there) is in a similar state of disrepair and has a similar amount of wildlife. This at least would give you an idea of what to expect.
If you do go, and I don't recommend it especially if you don't have a boat other than the Sea Racer or other rentable boats, good look and be sure to take some photos for us.
