You say that, but you're actually wrong -- most people do care about it even if they don't consciously realize it because it's an integral part of the design of a large part of the park.
When the waterfront is gone from LS/FL, people are going to dislike it. They may not even be able to place a finger on why, but they're going to think the area feels wrong now, and it's because that whole section of the park was designed to have a waterfront.
That doesn't mean Tom Sawyer Island itself needs to stay (I don't think it does) but eliminating the waterfront entirely is going to really screw up that section of the park and people are absolutely going to notice it, even if it's solely a subconscious feeling, and it's going to negatively affect their experience.
It's similar to if they filled in the World Showcase Lagoon and put a bunch of buildings there. It would really mess up the World Showcase, because the whole loop is designed with a waterfront in mind.