Line stallers don't hurt anything. Unless they are the next to board as soon as they move you catch up with the end of the line. They haven't delayed you a bit since the same number of people who were ahead of you are still ahead of you.
As for line jumpers, especially those who hop over the rails from the standby line to the fastpass line, they should be expelled from the parks for the day.
Generally, #1 doesn't phase me, but there ARE times when it matters, and THEN I find it a little frustrating. In short, it CAN matter on Space Mtn. On busy days, the CM's send some FP holders to Omega, and some to Alpha. They release standby riders in chunks, and determine those chunks- in part - by how well the standby folks are paying attention.
Alas, I don't think we've ever swapped lines since FP went electronic, but back in the paper FP days, if the standby was short, we sometimes opted to keep the paper FP as a souvenir.
We never hopped into the FP line without a FP, but sometimes - if we had a FP AND the standby lines wasn't moving- we'd switch tactics -and use our FP instead of keeping it. You possibly saw us do this. If someone without a FP does swaps, I agree they should be expelled from the parks. I don't see a problem though if someone WITH a FP swaps lines.
Also, the Kali queue is VERY confusing. More or less, whenever we ride it, we see someone accidentally end up on the wrong side of the queue due to the poorly marked queue set up. The queue is kind of a roped trail, but the roping has multiple long gaps. People think they have reached the merge, or they aren't paying attention and wind up on the wrong side.