I like walking through the caves and checking out the fort. It's not an attraction I visit every time I go, but it's worth 30 minutes or so of your time. And if you have kids, it's even more worthwhile. You can let them go psycho nuts on the Island while you either leisurely stoll the area or just have a seat and people watch. If Aunt Polly's is open, you can even have a snack. One of my biggest reasons for liking it is that it feels so removed from everything else. You could almost believe you're really alone (at times) deep in the middle of Missouri during the times of Tom and Huck.
ed: I would have to say the main reason I don't go on it every time is that I go to Disneyland almost weekly (and to Tom Sawyer's Island once every few months), and up until recently, the two were quite similar, so I didn't feel the need. Now, however, with the pirates overlay at Disneyland, I'll probably be visiting the WDW version every time I go (and Disneyland's less than before).