Yes, but people without the passes, and who are physically waiting in line, also are unable to do another house during that wait time. GAP-holders aren't entitled to be able to experience more attractions than the average guest.
For those of us with someone dealing with a cognitive disability, the inability to do something "exciting" between rides can defeat the entire purpose of having the pass. They can't sit on a bench or travel all the way to the hotel and back any better than they can tolerate standing in that line. My Autistic son loves the park. Next week I will fly us across the country, spend four nights in a hotel on property, and have tickets for five days to the parks. Thousands of dollars later, we are lucky if we are able to spend 10-15 hours total in the parks over the course of those 5 days before he hits his capacity for being in the chaos. The disability passes are not a "perk" to those of us who legitimately need them. They are the only way we ever get to experience the more popular rides, and the ability to squeeze -literally anything- in between them might be the only way we ever get to do more than one.
I know this isn't everyone's experience, but it doesn't seem like you are considering these types of circumstances in your responses.