Steve Vancore
St Pete Polls recently released findings from a poll showing strong support for two statewide ballot initiatives being proposed in Florida. The first is John Morgan’s second take on medical marijuana and the other is a $10 per hour minimum wage being offered by a group known as “League of Voters, Extraordinaire.”
Morgan is, of course, well known, but the second group has been around since 2011 and has only raised $1,100 for its efforts thus far.
St. Pete Polls recently polled both and found they both scored enough to pass with comfortable margins (68 percent and 65 percent respectively).
Let’s break out the saltshaker.
It’s necessary to begin this analysis with a few words about how this poll was conducted. From its methodology statement: “The poll was conducted through a Web-based email polling system.”
As long as we are doing these analyses, this type of polling will always start out with a grain of salt for three very specific reasons:
1. All Web-based polls automatically deselect a respectably large portion of the electorate. Elderly voters, poor voters, and that segment of the electorate that only recently purchased a flip-phone are all shut out of this methodology. It is very hard – although not impossible – to correct for all of these.
2. Email polling asks respondents to select themselves. This will create an inherent selection bias that generally means we pick up a disproportionate number of extreme opinions on both sides of any equation.
3. It is harder to validate that your targets (for example, a middle-aged white man, or 20-something Hispanic female) are actually who they say are. This tends to be less of a problem when you pull names from the voter file and shoot out thousands of requests, but the problem is still real enough to at least note it.
For those polls that use panels where respondents are incentivized (i.e. paid) to partake, the above problems only get worse. Finally, a larger sample does not fix the problem. A skewed sample is still skewed, irrespective of size.
Since this poll was conducted using this method, it begins with a grain of salt.
In evaluating these kinds of polls, the inherent biases in a poll may be minimal if the concepts being measured do not differ for those taking the poll and those who are more likely to be excluded.
For these two issues (and the crosstabs bear this out) it is very clear that under-representing the elderly for the medical marijuana ballot question skews the numbers to the “yes” side of the equation. (Those over 70 years of age are 20 points less supportive than millennials.) Likewise, it is pretty obvious that lower-income folks (income was not measured on the StPetePolls.org poll) would be more supportive of increasing the minimum wage. So, when the issue is at least somewhat dependent on a balanced sample along some of those variables, this type of polling can be problematic.
So add a second grain of salt.
Then we checked the demographics.
The St Pete Polls poll was balanced pretty well by party – it was a smidge too NPA – and was surprisingly very close on age. Further, it looks good on geography because respondents were spread nicely across the state. On the other hand, it was significantly off by gender and (no shock) came in 10 points too male. Further, it was too white (again, no surprise) by nearly 14 points.
Too male and too white means we have to add another grain of salt, even while commending them for getting party, age, and geography all correct.
But here is where St Pete Polls goes – uncharacteristically – off the rails (sorry, guys).
Here is the question as it was asked:
“Last year the constitutional amendment to legalize medical marijuana was narrowly defeated. If the new medical marijuana initiative makes it on to the ballot this year [sic] will you vote for it?”
That’s all fine and dandy if you want to know the answer to that question, but that question will not be on the ballot. The 2016 ballot will not have information about the 2014 ballot and how it fared. The actual ballot language is quite different and begins, “Allows medical use of marijuana for individuals with debilitating medical conditions as determined by a licensed Florida physician.”
If you want to know how people will vote on a specific ballot initiative you need to ask a question nearly exactly as it will appear on the ballot – that is a fundamental tenet of polling – and it was not done here. So, in that regard, we really don’t have a good poll testing the current medical marijuana amendment – heck, it could score higher or lower. That’s something we don’t know because it wasn’t tested and this error moves this poll into the half-shaker category.
As an aside, we take a slightly different view of the minimum-wage amendment, because that language is short and to the point. However, since it shows almost no sign of actually making it to the ballot and it doesn’t affect our view of this poll, we will leave further analysis of it to someone else.
Finally, we are certain there will be some pushback on this analysis from the good folks at StPetePolls.org. They have been remarkably accurate in past predictions and have done some excellent work in Florida. But, fair is fair and we must apply our predetermined standards equally.
With all those disclaimers, we give this product a half shaker. There are enough problems with the methodology to warrant serious concerns, and the poll should not be taken seriously.
Key for the Salt Shaker test:
- No salt needed: Solid pollster, solid methodology, and the sample appears to be nicely balanced.
- A grain of salt: The poll has one or two non-critical problems and should be taken with a grain of salt.
- A few grains: There are several concerns with how the poll was conducted, but not enough to throw it out entirely.
- A half shaker: There are enough problems with the methodology to warrant serious concerns, and the poll should not be taken seriously.
- A full shaker: The poll has so many problems it should not only be completely disregarded but pollsters receiving multiple “full shakers” will no longer have their polls covered by Florida Politics/SaintPetersBlog.
Steven J. Vancore is the President of VancoreJones Communications and Clearview Research. He can be reached at [email protected].
