Based on my experience with asking guests if they have an Annual Passport (during Merch transactions, inquiring about possible discounts coming to them), I agree with you. If they're not already a veteran AP, the majority of guests show confusion with the phrase, no matter how I fine-tune it ("do you have the Annual Pass?" or "are you guys Annual Passholders?" or "Do you have any discounts, like the Annual Pass?", etc).
As an aside, I'm peeved at how many AP holders think that the merchandise discount comes automatically coded into their magic bands or ticket media. Until they modify our merch POS tech, we have to see the discount card and a photo ID before we can give the discount.