Steve Vancore: Applying the salt shaker test to the Florida Southern College poll

Can a poll have flawed methodologies and still be relatively “accurate?”

Of course it can, much in the same way a dart can be close to its intended target even if the player throwing the dart does not use good form.

It might be that way with the newly released Florida Southern College poll, which shows, like many others, Trump leading and Rubio in second place in the Republican Party Presidential Preference Primary (PPP).

But, the poll itself and the way it was conducted needs to be taken with a whole lot of salt.

Here’s why …

1. Wrong voters.

The poll is a survey of “registered voters” and “registered voters” are NOT the droids we are looking for if we want to know where the PPP stands. The droids we want in that instance are “registered voters who have a history of actually showing up in elections.” And, while we simply cannot know what the actual turnout will look like (Will Bernie Sanders still be a factor? How different will the Republican Party field be and how will that impact turnout?) we can be certain that a fairly large percentage of run-of-the-mill, show-up-only-in-the-general-election types will NOT be voting in a closed Presidential Preference Primary. This cannot be determined by querying respondents chosen via random sampling (as citizens notoriously exaggerate their propensity to vote) but only by pulling that data from a scrubbed voter file.

2. Too small a sample.

With only 608 total voters interviewed, we have serious concerns about that sample size. Not necessarily about the general election results, but the PPP numbers. If we do a little math, we realize that only 195 registered Democrats (for example) were surveyed and that would yield a margin of error that is so large as to render the findings nearly meaningless — especially when we consider, as noted above that they are not exclusively made up of likely voters.

3. Excessive weighting.

The writers of the methodology went to some pains to explain the margin of error (good) but failed to mention that when you rely on reweighting a small sample by a large degree you magnify your margin of error. Tweaking a data set with weighting is normally fine, especially if you have a robust sample size and it is fairly well-balanced, but, in this case, the sample appears to be neither large enough or close enough to allow for weighting without creating serious statistical problems. To be clear, weighting a flawed sample only compounds — it does not fix — the problem.

4. The poll is out of balance.

This poll shows that Republican Party voters will outnumber DEM voters by more than 12 percentage points. In the past two presidential elections, Democrats outnumbered Republicans by a point. And, even in the most heavily skewed (off-year) elections, Republican Party voters top Dems by only 6-8 points. This large of a skew cannot be fixed by reweighting the poll for reasons noted above. Further, it appears they attempted to balance the poll by voter registration and that is somewhat problematic in that polls measuring test ballots should be balanced by likely voters, not all But, even with that, this poll has too many men, too many white voters and as noted above too many Republicans.

Taken as a whole, this poll has some serious methodology problems and even yields some inexplicable results. For example, we see a heavy skew with Republicans outnumbering Democrats by a wide and unrealistic margin, yet Hillary handily defeats the Republican Party front-runner Trump and runs within three points of Cruz, Rubio and Bush. If Republican Party voters outnumber Democrats by 12 points, there is no way — NONE — that she is beating Cruz by 3 points or losing to Rubio by only 3 points. Further, we also see some of the weakest primary candidates (Bush and Sanders) doing extremely well in their respective General Election matchups, with Sanders beating everyone. This might make some sense if the poll broke out likely and unlikely voters, but it does not.

The methodology problems and many of the results leave us scratching our heads — and reaching for a full shaker of salt.

***

Steven J. Vancore is the president of Clearview Research. He has 30 years of experience conducting polls and focus groups throughout the state and currently serves as an adjunct in the master’s of Applied American Policy and Politics program at FSU. He can be reached at [email protected].