- Why am I or my friends never included in political polls conducted for the media?
- Is a larger sample always better than a smaller sample?
- How can a sample of only 800 or 1200 truly reflect the opinions of 200 million Americans within a few percentage points?
- Why do polls often disagree with how I and my friends feel about things?
- Why is job approval for elected officials asked so often in polls?
- How accurate are polls?
- Can wording of questions bias poll results?
- Why should I respond to a poll?
1. Why am I or my friends never included in political polls conducted for the media?
The reason is fairly simple. There are about 200 million adult or voting age Americans. But the average poll has a sample size of 1,000 adults. This means that only one person in 200,000 will be included in any one national or state poll. To put it another way, it would take 200,000 polls with samples of 1,000 for pollsters to get around to all Americans - and this assumes no one is called twice.
Of course, national and local media organizations conduct several polls in one year. The number of national or local media political polls you see in a single year is about 250 - but can vary depending on where you live. But even two hundred fifty polls in a single year means your chance of being interviewed at least once is still small.
2. Is a larger sample always better than a smaller sample?
Larger samples are generally more precise, but sometimes not. The important rule in sampling is not how many poll respondents are selected but, instead, how they are selected. A reliable sample selects poll respondents randomly or in a manner which insures that everyone in the area being surveyed has a known chance of being selected.
3. How can a sample of only 800 or 1200 truly reflect the opinions of 200 million Americans within a few percentage points?
Sampling methods and measures of sample reliability or precision are derived from a mathematical science called statistics. Statistics is a subject taught in colleges and some high schools. Text books on the subject are available in most libraries.
At the root of statistical reliability is probability; i.e., the odds of obtaining a particular outcome by chance alone. As an example, the chances of having a coin come up heads in a single toss is 50%. Heads is one of only two possible outcomes. The chance of getting two heads in two coin tosses is less because two heads are now only one of four possible outcomes; i.e., a head/head, head/tail, tail/head and tail/tail. As the number of coin tosses increases, it becomes increasingly more likely to get outcomes which are either very close to half heads or exactly half because, as with two coins, there are more ways to get such outcomes.
Sample survey reliability works the same way - but on a much larger scale. As in coin tosses, the most likely sample outcome is the true percentage of whatever it is we are measuring across the total population. Next most likely are outcomes very close to this true percentage. A statement of potential margin of error or sample precision reflects this and often appears in poll stories. Using a sample of 1000 as an example, the statement could read: the chances are 95% of coming within +/- 3% of a hypothetical survey conducted among all members of the population. This means that 95% of all samples which could possibly be drawn will yield an outcome within 3% of the true percentage among the population.
Keep in mind that estimates of potential sample error always assume random samples. But even in true random samples, precision can be compromised by other factors such as the wording of questions or the order in which questions were asked.
There is no single ideal sample size. Samples of any size have some degree of precision. The question is always whether there is sufficient precision to draw conclusions as determined by statistical formulae.
4. Why do polls often disagree with how I and my friends feel about things?
It is unlikely that you have a circle of friends as diverse as a randomly selected sample described before, a sample of the entire nation or of the state where you live. This would mean you have friends from all neighborhoods, of all ages, very wealthy friends and friends with no wealth, who are from all walks of life and with educational levels ranging from grade school to post-graduate.
No poll has ever shown all people feeling exactly the same way on one issue. So the next time you see a poll showing only 30% in agreement with your point of view, remember, although you may not be in the majority, it still means that 60,000,000 Americans feel the way you do.
5. Why is job approval for elected officials asked so often in polls?
Job approval is a question asked in polls for over 50 years, asked about presidents from Harry Truman to Bill Clinton. It is a measure of potential electoral success for an incumbent president (or state and local elected officials) and is useful for trend purposes. Incumbents who fail to win approval from a majority of the public for the job they are doing are very often in trouble on election day.
Experience shows presidential job approval has a lot do with how well people think things are going in the country today. High job approval does not mean the office-holder is necessarily held in high, personal esteem. Job approval is just one question which when analyzed in combination with response to other questions gives a good overall picture of how an
incumbent is perceived.
6. How accurate are polls?
The NCPP analyzed final presidential election polls conducted by the national media dating back over 50 years. When compared with actual election outcomes, average poll error for presidential elections between 1956 and 1996 has been declining. Average poll error on each candidate during this period was 1.9 percentage points.
Important to this analysis of accuracy is that most of these polls were conducted within days or even hours before election day. Polls conducted 1-2 weeks before election day or even longer by local newspapers and TV stations cannot usually be expected to closely match election outcomes. Earlier polls are intended to monitor the success of campaigns and to identify the issues or events which will influence voter preferences on
7. Can wording of questions bias poll results?
How questions in a poll are worded is as important as sampling procedure in obtaining valid results.
Most professional polling organizations and their media clients review the wording of questions as carefully as editors would examine a manuscript before publication. This process usually calls for a review of several drafts prior to fielding a poll. Questions are checked for balance, that is, are they worded in a neutral fashion without taking sides on an issue? Does the question represent both sides of an issue fairly? Answer choices read to poll respondents must also be balanced; e.g., approve or disapprove, favor or oppose.
The order of questions must be in logical order. That is, general questions are asked before specific questions. For example, overall job approval of an incumbent must be asked before specific questions are asked which may remind respondents about the incumbent’s successes or failures. The same goes for questions asking respondents which side they take on an issue which may influence a later question about opinion of a candidate who takes the opposite side.
Questions are written using clear, unambiguous, concise language to insure all respondents regardless of educational level understand them. And since most polls are conducted by phone, a writing style suitable for the ear is often adopted as opposed to a style more suitable for reading.
8. Why should I respond to a poll?
Each year, tens of thousands of Americans respond to poll interviews on subjects of national interest. They do so because they want their opinions heard. A poll or survey is an unbiased way for people to make their views known to each other, to their government, to businesses, to
educators and many other institutions. This is one way for average Americans to add their voices to the debate over important issues of our day. It is our sense that people want to be included in polls.
Polling is only a part of a $3 billion U.S. survey research industry that affects your life in many ways. Medicines in your cabinet became available only after they had been tested for effectiveness and safety, tested among samples of patients. Manufacturers use sampling methods to guard against producing defective products. Businesses conduct market research by sampling consumer opinions to help develop products people want and avoid costly mistakes. Even the Bureau of the Census only interviews everyone in the nation once every ten years. The Census Bureau along with other government agencies also conduct sample surveys to obtain social and economic data we see in the news.
One thing legitimate survey researchers will never do is try to sell you something. That is called telemarketing. According to direct marketing industry estimates, there are now 2 million telemarketers who call the general public to sell products and services. The number of survey
research interviewers who conduct telephone interviews is probably only
a small fraction of 2 million.