Salt Shaker Test: Take Mason-Dixon poll with Jeb crushing Florida with a few grains of salt

The most recent poll from Mason-Dixon shows Jeb Bush all but lapping the current GOP field holding a strong 12-point lead over the 2nd place finisher, Marco Rubio, who scored a solid 16% in a field of 16 challengers.

You read that right, 16 challengers named on a phone poll.

Mason-Dixon did well to rotate those names, to try and avoid inherent ballot and polling dynamics related to ordering. But we must recognize that when the final ballot is printed and if the list remains even half that long, things like primacy-recency and other ordering effects will matter to some extent.

So we have no problem with that. They played the best cards they could with what the Republican field is dealing us right now.

However, we have a real issue with the fact that this poll interviewed, “500 registered Republican voters and 500 registered Democratic voters.”

Why?

In the most recent Presidential Preference Primary election in 2012, turnout was (according to the Secretary of State) 41%. While we have no way of knowing what turnout will actually be next March, we can be reasonably certain it will be somewhere between 20% and 45% due to voter behavior in past primary (presidential and otherwise) elections.

So, while we don’t know exactly what turnout WILL be; we are certain that the electorate will not be comprised of “registered” voters. It will be a much smaller subset of those voters.

Mason-Dixon should have pulled a list of voters based on some measure of prior voting history or even tried to gauge some level of engagement once they had respondents on the phone. It is critical to note that in order for a poll to be taken very seriously, it must be comprised of respondents who will look like the electorate. In the present circumstance, likely Republican and likely Democratic voters is an absolute must.

From its release, it appears M-D did not attempt to do that, and for that we simply have to take this poll with a few grains of salt.

This methodology (of interviewing casual voters) could explain, in part, why Bush did so much better than expected, as he is likely better known among those voters. It might also explain why some of the other candidates (Carson, Jindal, Santorum, Perry, Christie, Graham and Pataki) ALL scored 1% or less, as they are not going to be as well known among less frequent voters.

Additionally and interesting to note, the poll was balanced by county (good) to ensure the state was well represented. But, again, it was balanced by “registration” figures and not turnout figures. Further, the pollster mentions that some attempt was made to talk to cell phone users, but it was unclear if those callers were verified as voters, let alone likely primary voters. They stated that the, “cell phones were selected from a list of working cell phone numbers.” We are not quite sure what that means and if it makes the sample better or worse.

On the bright side, Mason-Dixon might have been tempted to treat this as one big 1,000-sample poll and applied the smaller margin of error of 3.1%, but they didn’t. They correctly noted this was actually two separate polls and stuck with the larger 4.5%. Good for them.

Even though the very smart folks at Mason-Dixon were dealt a tough hand in having to poll more than a dozen candidates on the GOP side of the ledger, we think they dropped the ball on choosing a too-broad sample of voters.

With that, we suggest you take this poll with a few grains of salt.

Again, here is the key for the Salt Shaker test.

• No salt needed: Solid pollster, solid methodology, and the sample appears to be nicely balanced.
• A grain of salt: The poll has one or two non-critical problems and should be taken with a grain of salt.
• A few grains: There are several concerns with how the poll was conducted, but not enough to throw it out entirely.
• A half shaker: There are enough problems with the methodology to warrant serious concerns, and the poll should not be taken seriously.
• A full shaker: The poll has so many problems it should not only be completely disregarded but pollsters receiving multiple “full shakers” will no longer have their polls covered by Florida Politics/SaintPetersBlog.

Steven J. Vancore is the President of VancoreJones Communications and Clearview Research. With a Master’s degree in Marketing Communications from Florida State University, he has nearly 30 years experience conducting polls and focus groups throughout the state.  He serves as an adjunct instructor in the Master’s of Applied American Policy and Politics program at FSU. He can be reached at [email protected].