Taking away the fact that the rides aren't actually part of the MCU but I guess a spin-off universe that is similar but different.....
The Collector has never been revealed to be a true "antagonist" in the way, say, Thanos or Loki are. His main appearances so far include Lady Sif bringing him the Reality Stone at the end of Thor: Dark World, his scene in Guardians where, again, the protaganists brought him the Power Stone before his slave blew the place up, and his "appearance" in Infinity War, which was actually just a fake reality created by Thanos. The only Avengers who would have had an interaction with the Collector by this point are Thor and the Guardians, and Thor trusted the Collector enough to willingly give him an Infinity Stone, while the Guardians....well, we know where they're at. The first time the Collector ever truly sits in the antagonist role is in this ride; otherwise, he occupies the space of meddlesome but not outright malicious up to this point.
Everything else kinda slides into place from that point. The Avengers wouldn't notice that the Guardians are being held prisoner because the Terrans are the first visitors (and if we're being fair, the Avengers probably aren't doing a thorough inspection of the entire place so the Collector could have just hid the Guardians up to that point). The Avengers wouldn't know where the Guardians are in the same way Thor didn't know Hulk was on the Grandmaster's planet until he saw him. Seems pretty simple really.