As part of New Jersey’s public research university, Eagleton Center for Public Interest Polling considers education a core value. As such, we make all our non-proprietary polling information available to researchers and other interested citizens. Through a joint effort with the Scholarly Communications Center at Rutgers, Eagleton has established an on-line searchable archive of its New Jersey media polls.
How to Assess a Poll’s Validity
There are no cut-and-dried answers to what makes a poll “valid.”
All public reports of survey findings should include reference of the following:
- Sponsorship of the survey
- Dates of the interview
- Method of obtaining the interviews (in-person, telephone or mail)
- Population that was sampled
- Size of the sample
- Size and description of the sub-sample if the survey report relies primarily on less than the total sample
- Complete wording of questions upon the release is based
- The percentages upon which conclusions are based
Key things to look for are:
- Size and corresponding margin of error – be cautious of results when the margin or error exceeds 5 or 6 points
- Whether the sample was scientifically chosen or whether respondents were self-selected (which is typical of “900” call-in polls and many web-site polls)
- The wording and order of the questions (look for questions that seem to “push” the respondent to an answer by describing some options more attractively than others).
More information is available in the short publication “Twenty Questions A Journalist Should Ask About Poll Results” by Sheldon Gawker and G. Evans Witt. It is available on-line from the National Council on Public Polls.
But I’ve never been polled! How do they know what I think?
What makes polling work is “random sampling.” Most public polls are based on interviews with between 800 and 1,200 people selected through some type of random sampling. Polling is a scientific effort to understand what people think. Thus pollsters make calls to a mix of households throughout the state or region of interest – in the cities, the suburbs, rural areas, the shore. People who study random sampling have demonstrated that talking with around 1,000 people can get enough variation to be representative of the whole population within what is called the “margin of sampling error.”
For most Rutgers-Eagleton Polls, the margin of error is around +/-3.2%, based on about 950 respondents. Assume we find in a poll that 48% support something and 40% oppose it. That means that we are 95% cure that opinion on that issue is 48% in support, 40% opposed in New Jersey, PLUS OR MINUS 3.2%. In other words, we are very confident that between 44.8% (48 – 3.2) and 51.2% (48 + 3.2) of the population are supporters. While it is possible that we are off by more than this, 95 out of 100 times the result will be within the margin of error. What we report is in the middle number, the most likely estimate of public opinion within this margin of error.
It is true as more people are polled, the margin of error goes down, so we become more confident in the range of our results. If we polled 3,600 people instead of 900 or so, our margin of error would be +/- 1.6%. This means we would expect to find a result similar to what we found with the smaller sample, but we would be be able to be more precise in the findings. It does not mean the findings would change, and in fact they would be unlikely to change significantly even if we called more people, as long as the process is a random number.