Guest AuthorSecond in a series.
Wanna sound smart?
Ask your pollster, “So what’s the margin of error?”
Wanna sound really smart.
Don’t ask it.
While there are entire courses dedicated to what are known as bell curve or parametric statistics and you can learn about smart sounding things like random distribution, confidence levels, and standard deviations (blah-blah-blah), the fact is that the way we commonly use “margin of error” is considered by most Ph.Ds to be wrong.
But as a consumer of polling, all that you really need to know is quite simple.
The bottom line is this: the larger the sample size, the smaller the margin of error. All things being equal, a poll that surveyed 1,000 people would have a smaller margin of error than one that surveyed 500 people. The range of numbers within a margin of error simply signifies where the likely “real answer” would be if you surveyed the entire population. The +/- means the deviation can go both ways…up or down.
If you are an occasional reader of polls or someone who simply enjoys reading the horse race numbers — who’s ahead or who’s behind — read no further, as that’s really all you need to know.
For those of you who want to delve a little deeper, here are a few items that will help you get the most from your data:
- O.E. only applies when you have fairly large populations. (If, for example, you want to poll a population of only 200 people, margin of error does not apply.) The vast majority of political polls have populations that are large enough to qualify.
- When we say, “at the 95 percent confidence level,” that means that if the poll were conducted 100 times (in the same exact manner) the answer would likely fall in the margin of error range 95 times and outside of that range five times. (My old mentor, Dr. Jim Kitchens, used to say, “By definition, this means that one in 20 polls are wrong.”)
- When you are looking at crosstabs — or subgroups — within a poll, the margin of error does not That is because you are looking at smaller sample sizes, so you must view differences among these groups with more caution. (There are statistics and methods to help you know if the differences between two subgroups are significant or not but that is super-geeky stuff.)
- Be cautious when looking at questions with a lot of categories and especially “open-ended” questions. The real margin of error formula kind of freaks out with these kinds of questions.
- If the respondents who participated in the poll were not randomly drawn, then the underlying assumptions that allow you to apply the margin of error formula do not apply. (So add an extra touch of caution if you are using Internet polling and are rewarding people to participate.)
- If the respondents are not representative (meaning, they don’t “look like”) the overall population, then the formula does not apply or is misleading. So, for example, if you expect 54 percent of voters on election day to be female, yet only 35 percent of your respondents are female, then the results of your poll are likely to be skewed. (Unless of course, men and women think exactly alike on the issues you are testing.)
This same concept applies to geography, age, race and other so-called demographic variables. In short (too late for that, huh?) the more your sample “looks like” the population you are trying to measure, the more reliable your results will be. If your sample is skewed, you can have 25,000 respondents — or 100,000 respondents for that matter — and your results will still be skewed if your sample is not properly balanced.
This last point is vital and why most professional pollsters recognize that slapping on a “margin of error” statement is both necessary (the media and many clients demand it) yet potentially misleading. In reality, the potential margin of error is truly open-ended and a statistically valid and reliable poll is dependent on much more than just sample size. Are the questions non-biased? Was the sample properly drawn? Is the sample balanced? What methods were used to ensure that (in this case) likely voters were surveyed?
But don’t let the above scare you, as there is something really neat about the margin of error and bell-curve statistics.
If you use standardized techniques and sound polling principles, most polls are very accurate and the results usually fall well within the margin of error range.
Consider that this month alone there have been a dozen or so publicly reported polls in the governor’s race and, let’s face it, they pretty much are telling us the same story. (This is especially true when we adjust them so they all have the same turnout model.)
When you put it in perspective, collectively those polls only spoke to around 10,000 or so people out of a universe of nearly 12 million voters — a miniscule sample. Yet, almost magically, most of them are very close to each other. After nearly three decades of doing this, I still find that to be very cool.
And for the skeptics who don’t believe in random sampling or how accurate it can be, I suggest you try this exercise. The next time you go to the doctor and he or she wants to test your blood by sampling a few drops, tell the doctor you don’t believe in random sampling and insist they take it all.
Steve Vancore is the President of VancoreJones Communications and Clearview Research and has been conducting polling and voter research in Florida since the mid-1980’s. He can be reached at [email protected]. Column courtesy of Context Florida.
