I've been to both...personally I would stay at Port Orleans. POFQ is smaller, there for easier to navigate, than PORS but both have beautiful buildings and landscapes and are my favorite resorts at Disney.
I'm not a huge fan of Caribbean...but that's just one opinion!!!