There are two main problems with IOA.
1. (This is the biggest). It just doesn't really offer anything that appeals to families. Walt's old adage that his park should be a place where parents and children can both have fun together isn't followed at IOA. Its a beautiful park, but far too many of the attractions are of the thrill variety.
2. (This kind of ties into #1). There just isn't enough to do in the park, especially if you don't like coasters. I can get through it in just a few hours because we skip most of the thrilling things. It focuses way too much on one demographic without really giving a decent amount of things to everyone. If you don't want a coaster or something with a drop or spin in it (Say you get motion sickness) then you've cut out The Hulk, Spiderman, Both Toon Lagoon rides, Jurassic Park River Adventure, Dueling Dragons, and possibly the Cat in the Hat. What else is there to do?
IOA has a decent amount of wonderful rides with thrills, what they need to do next is focus their next several rides on being E-Ticket caliber rides, but that are of the variety that everybody can ride together. I'm hoping that HP will be able to do that. That is also the reason that USF carries more people. While not being a beautifully themed park, there's just more to do for the whole family. The only rides that really keep a significant amount of people out of them are The Mummy, and possibly Jimmy Neutron. (We'll have to wait and see how bad The Simpsons Ride is because I used to put BTTF in this list as well.) However you're still left with Shrek, Twister, ET, Disaster, Terminator, Jaws, Men In Black, etc...