Steve Vancore
The recently released Gravis Marketing poll of the Florida Presidential Preference Primaries and the U.S. Senate races is truly confounding.
On one hand, it appears thorough and reasonably well-conceived. On the other, it has some very serious problems that all but make the poll useless and even misleading.
The poll had large numbers of voters and was conducted using so-called “robo-calling” technology (also called “IVR technology”), which is not necessarily a bad thing, especially for simple test ballot questions.
This poll’s major problems stem from its sampling breakdown — and that calls the validity of the results into question.
The first thing that jumps off the page are the results themselves.
How on earth does Alan Grayson go from being largely unknown and (in other polls) sitting in the high teens or low 20’s, to a shocking 63 percent of the vote? (Mason-Dixon recently had Grayson at 14 percent and Public Policy Polling had him at 22 percent.)
Gravis sets the stage in the prologue by claiming the following:
- “The numbers reflect and confirm several recently released survey’s [sic] from other public opinion research groups that show the upward trend of Florida Congressman Alan Grayson.”
- “Grayson was the beneficiary of an unlikely ally in recent weeks when the conservative Club for Growth unexpectedly launched a statewide media buy…”
The first bullet and the idea that Grayson is on an “upward trend,” confirmed in “several recently released survey’s” [sic], is an unsubstantiated claim.
First, there have not been that many publicly released polls on this seat and second, Real Clear Politics (for example) shows an actual decline in Grayson’s numbers (from 22 percent to 14 percent). Even the partisan polling outfit, Vox Populi recently showed Murphy up 10 points in the initial test ballot.
But, even if that were true, moving from 22 percent to 63 percent is hardly an “upward trend” and is more like an in inexplicable explosion.
The second bullet sounds more like a justification for the outlier numbers than any actual analysis.
The Club for Growth ad is part (the third part) of an overall $1 million campaign, so we must assume they spent less than that on the so-called “Export-Import ad” that Gravis attributes to Grayson’s 40-point jump.
Assuming they spent $500,000 on the ad (a generous assumption), there is simply no way it would triple Grayson’s numbers.
Further, Gravis completely discounts the slew of bad press (from Grayson’s “shitting robots” comments to stories related to his alleged ethical lapses) and not only ignores the steady stream of news stories about well-known Democrats endorsing Patrick Murphy, but actually claims Murphy has been under attack from Democrats.
In short, voters don’t move THAT much in this short of a period of time, irrespective of how much money the Club for Growth spent – and they really didn’t spend all that much to move statewide numbers 40 points.
And what about the balance of the poll?
Gravis claims the sample was balanced to look like the electorate, but we have no idea where they got their numbers from or how they decided on that balance, because according to our recent pull from the voter file, some of their numbers are very far off.
Without getting too technical or too nit-picky, there were a few really big problems with their sample:
For the Democratic Presidential Preference Primary –
- It is WAY too young. With 42 percent of Gravis’ sample being under the age of 50 that is nearly double the percentage of under-50 voters from the most recent Presidential Preference Primary election and is more than 22 percentage points higher than the most recent statewide primary.
For the Republican Presidential Preference Primary –
- It is WAY too ethnic. Only 73 percent of Gravis’ sample is white, roughly 16 percentage points lower than white voters from the most recent Presidential Preference Primary election as well as the most recent statewide primary.
Both samples-
- Are WAY too educated. (No, that’s not a subjective analysis.) With 87 percent of the Gravis respondents claiming some college education and 54 percent with a college degree or higher, this sample is far more educated than a typical electorate. And, while we don’t have reliable figures for a specific electorate, we are confident that a typical electorate will be in the mid to high 20’s on the college degree measure.
There are other minor issues with the demographics of this poll (for example, it is slightly more female than the Democratic presidential primary-voting electorate and it under-samples black voters by a decent margin as well. Whereas, under-50, in the Republican presidential primary-voting electorate is only slightly over sampled), but when your sample is so overwhelmingly off-balance in the three areas we focused on, it is kind of a showstopper.
We opened by saying the poll was confounding and it was. While the Grayson-Murphy numbers are simply not believable, the top-of-the-ticket test ballots (Bush, Rubio, Clinton etc.) all seem reasonably consistent with other polls taken around the same time.
But, that does not forgive the very serious lapses in balancing this sample; so we score this poll somewhere between a half and full shaker of salt.
Again, here is the key for the Salt Shaker test.
- No salt needed: Solid pollster, solid methodology, and the sample appears to be nicely balanced.
- A grain of salt: The poll has one or two non-critical problems and should be taken with a grain of salt.
- A few grains: There are several concerns with how the poll was conducted, but not enough to throw it out entirely.
- A half shaker: There are enough problems with the methodology to warrant serious concerns, and the poll should not be taken seriously.
- A full shaker: The poll has so many problems it should not only be completely disregarded but pollsters receiving multiple “full shakers” will no longer have their polls covered by Florida Politics/SaintPetersBlog.
