STATEMENT ABOUT INTERNET POLLS
by NCPP Polling Review Board
While different members of NCPP have different opinions about the potential validity and value of online surveys, there is a consensus that many web-based surveys are completely unreliable. Indeed, to describe them as "polls" is to misuse that term.
Journalists who are considering reporting the results of internet-based surveys should ask the following ten questions:
1. Is the internet-based survey designed to be representative, and if so, of what population?
If not, it is not worthy of being reported.
2. What evidence is there that the sample is representative of the population it claims to represent?
Unless the internet-based survey can provide clear evidence that the sample is representative by demographic and/or other relevant information it is not worthy of being reported.
3. How was the sample drawn?
Many internet-based surveys are just "call-in" polls or are asked only of people who happen to visit a particular web site. These surveys usually do not represent or make any pretense to represent any other population, and are not worthy of being reported.
4. What steps does the organization take to prevent people from voting more than once?
Any poll which allows people to vote twice, or more often, is not worthy of being reported.
5. How were the data weighted?
Survey data may contain biases from a variety of causes. The magnitude of these biases and random errors are usually unknown to the researcher. Even so, weighting may minimize these biases and errors when there is a strong relationship between the weighting variable and data in the survey. If there is not a strong relationship weighting may make the survey results worse. Demographic weighting of internet-based surveys is essential but is not sufficient. Some firms, in addition to demographic weighting, are weighting on other variables in an attempt to reduce the biases of online data.
6. What is the evidence that the methodology works and produces accurate data?
Unless the organization can provide the results of their other internet-based surveys which are consistent with other data, whether from the Census or other surveys, the survey results are not worthy of being reported.
7. What is the organization’s experience and track record using internet-based polls?
Unless the organization can demonstrate a track record of obtaining reliable data with other online surveys, their online surveys should be treated with great caution.
8. What is the organization’s experience and track record as a survey researcher using traditional survey methods?
If the organization does not have a track record in designing and conducting surveys using the telephone or in-person surveys, it is unlikely that they have the expertise to design and conduct online surveys.
9. Does the organization follow the codes of conduct of AAPOR, CASRO, and NCPP (whether or not they are members)?
If they follow none of these, they are probably not a qualified survey research organization. The more of these Codes they follow, the more likely their data are to be reliable and be trusted.
10. Is the organization willing to disclose these questions and the methods used (as required by the codes of conduct referred to in #9 above)?
If the organization is unwilling to disclose, or unable to provide, the relevant information the survey is probably not worthy of being reported.
In addition to these ten questions, journalists should review NCPP’s "Twenty Questions A Journalist Should Ask About Poll Results."
For more information about this and other polling issues, contact the NCPP Polling Review Board Members.