Great points but you are neglecting the psychology of the survey, Now the word is out that you have to LIE to the survey to participate which will skew the results even further. If you want to collect good DATA which even facilitates longitudinal surveys and comparisons between demographics like social media users and non social media users.
Part of why I think the person commissioned the survey added this behavior was so they could answer truthfully this is ALL the survey data because as we know the other way the same result could have been achieved would have been something like the SQL select statement below to begin the analysis of the data.
SELECT * FROM RESORT_FEE_SURVEY_RESULTS WHERE SOCIAL_MEDIA_USER EQ FALSE;
This would have selected only the respondent community of interest and would likely have given BETTER results because the respondents had no incentive to lie on the survey in order to participate. Collecting the data like this would also allow other questions to be analyzed like which community is most accepting of resort fees.
It's simply not how study results are collected or calculated. Correct design or incorrect design, it's standard practice to drop people after a certain demographic number has been reached. It's likely because most surveys don't have a community of people who would be apt to LIE in order to get into the survey and answer things. Part of the whole idea behind a survey is that you are looking for honest opinions.
But part of the risk of qualitative surveys. You can never bank on 100% honesty in results. You have to design your study as such so that it takes false responses into account.
Not being an expert in the qualitative side, my guess would be that if you didn't limit your results in the targeted demographic segments, there would be actually
more of an impulse to use
all of the data.
Let's say you design your study to look into equal population segments of 1000 social media users and 1000 non-social media users. You do not limit the collection of the data, you get your 1000 non-social media users, but you get 10,000 social media users. Because you designed your experiment (survey) to look at equal numbers, you cannot just assume that you can linearly scale the results (take the metrics and divide by 10 and assume that it is equivalent). If you grab a random sample of 1000 records out of the 10,000, you are more likely to want to repeat that over and over, trying to drive results out of an average of 100 random samplings. This might make sense, but your other group, your non-social media responses have not had the same methodology applied, and you've introduced something into the process that cannot be replicated by the smaller dataset, and you can no longer draw similar conclusions because you've changed your experiment methodology and your samples are no longer collected in the same method. It's much more simple to just collect 1000 answers from each population and measure your results.
But this is all speculation. If you'd like I can talk to our department here that specializes in qualitative research and get their opinion.
Or we could just assume that Disney = Evil and their management are a bunch of bumbling buffoons who will gamble their short-term career on purposefully skewing results to reach incorrect conclusions and hope that it will not come back to bite them in the buttocks before they move along. Of course, there are likely some of those folks in the Disney organization, there are folks like that in every corporation. But there are also likely people with integrity which will ensure that things are done in an ethical manner which balance those types out.
