Steve Vancore
It was trumpeted (pun intended) as an “Exclusive Poll” for Bay News 9, and it showed Trump so far in the lead we just had to scratch our heads.
Anytime you have an outlier poll with unusual findings it is vital that you turn a skeptical eye to the results – and so we did.
SurveyUSA conducted the poll, and they used what our lawyers will advise us to call, “an interesting methodology.” Although the Bay News 9 release says the poll was “conducted via telephone” the detailed statement clarifies that “69 percent of registered voters were interviewed on their home telephone” and the rest were “shown a questionnaire on their smartphone, tablet or other electronic device.”
It also appears that they used interactive voice recording (IVR) technology – also known as “robocalling” – and not live operators.
There are several problems with that.
IVR, while it has some problems, is usually best for simple polls with a few questions and a few choices. Longer polls with multiple and long-worded answers are notoriously problematic when a voice is telling the respondent, “Press 1 if you blah blah blah blah, Press 2 if blah blah, Press 3 if blah blah, and Press 4 if blah.” You can easily see why this alone could lead to inaccurate results.
It was a fairly long poll with a good number of questions and multiple responses so that causes some concerns.
But, the problems grow from there.
They then administered the poll to those who were not answering a landline via a visual medium. That too can work, but you will often get different results. Blending the two raises serious concerns about the internal reliability of the responses, especially if the differences between the groups were not disclosed and the demographics were not properly balanced – and there is no indication that they were.
It is unclear how they determined the respondent was a voter because the poll was taken of “3,000 state of Florida adults” of which, “2,712 were registered to vote” and “2,400 were determined to be likely to vote.” It appears that the respondent volunteered that information, as there is no indication it came from the voter file. Voters notoriously exaggerate their propensity to vote and so if respondents were asked about their likelihood of voting, that is another real problem. Angry citizens, for example, are more likely to say on a poll they will be voting, even if they have never done so.
(Hmmm … maybe this is why Trump did so well?)
If you are not using the voter file to make this determination, then you leave yourself open to self-selection problems, where someone claims to be something they are not and that will get you a grain of salt.
And, while they did not disclose the demographic breakdown, we see from some of the raw numbers that 38 percent of their likely voters were Republicans, 34 percent were Democrats and 27 percent were “Other.”
That’s not going to happen. To be clear, there is almost no chance Republican voters, in a presidential election, will outnumber Democrats by 4 points. For the past two cycles, Democrats nudged out Republicans by a point each time. Plus, we can’t see a scenario where “NPA/Others” will top 22 percent or 23 percent, and that is likely generous, so pegging them at 27 percent is again not going to happen. As a result, this poll skews in a way that under samples Democrats by a fairly large margin.
And, if that isn’t enough, imagine an election where less than 1 percent of the Florida electorate is “Cuban.”
We were tempted to delve further, but the deeper we dug, the more we were convinced that while this poll and its results are indeed entertaining, it should not be taken seriously.
Odd blended sampling. Unrealistic and undisclosed demographics. Self-selected participants. Self-determined party affiliation and voting propensity.
Do we have enough salt left in our salt shakers?
This morning, fivethirtyeight.com did an analysis of a SurveyUSA poll from earlier this week. After the geniuses in Nate Silver’s shop (and we believe they are actual geniuses) analyzed SurveyUSA’s methodology in that poll – a poll that found them off in a Kentucky race by a very wide margin – they attributed the errors to an “electorate modeling problem.”
We agree and we believe they have the same problem in the Bay News 9 poll.
There are more than enough problems with this sample to warrant serious concerns. This poll, showing Trump lapping the field, should not be taken seriously.
Our verdict: A full shaker of salt.
Key for the Salt Shaker test:
- No salt needed: Solid pollster, solid methodology, and the sample appears to be nicely balanced.
- A grain of salt: The poll has one or two non-critical problems and should be taken with a grain of salt.
- A few grains: There are several concerns with how the poll was conducted, but not enough to throw it out entirely.
- A half shaker: There are enough problems with the methodology to warrant serious concerns, and the poll should not be taken seriously.
- A full shaker: The poll has so many problems it should not only be completely disregarded but pollsters receiving multiple “full shakers” will no longer have their polls covered by Florida Politics/SaintPetersBlog.
Steven J. Vancore is the president of Clearview Research. He has nearly 30 years’ experience conducting polls and focus groups throughout the state. He serves as an adjunct instructor in the Masters of Applied American Policy and Politics program at FSU and can be reached at [email protected].
