The political polling community has taken its share of hits over the last few years and there has been no shortage of brickbats thrown at pollsters both in the US and UK. All sorts of explanations have been suggested.
The response within MR has been especially amusing, focusing primarily on measurement error (asking the wrong questions in the wrong way). The polling community, on the other hand, has mostly focused on sampling and it’s no surprise that much of what AAPOR found (somewhat mirroring the UK investigations) falls into that bucket.
Anyone who has a serious interest in the issues should read the AAPOR report which is posted here. I’ll just quickly summarize the main findings.
- At the national level, the polls did well. On average, they predicted a 3 point Clinton win and she won the popular vote by 2 points. No controversy there.
- But the state polls were a problem, especially in the upper mid-west, thus not suggesting the narrow electoral college win for Trump. There are a bunch of reasons here, including an unusual number of late deciders (always a problem), but the main problem goes to sampling. There were not enough of those “low information” no college degree voters we read so much about in many of the samples. The bias was toward college educated voters. The report argues that the failure to correct the bias through weighting by education was a critical mistake. This is reminiscent of the explanation for the polling miss on Bexit.
- There also were significant turnout issues. In Michigan where I live, Trump got over 100,000 fewer votes than Romney in 2012 and Romney lost the state to Obama by close to a half million votes. A lot of Clinton voters stayed home.
- There is no evidence of “the shy Trump voter” as a serious factor.
This ought to be a reminder for all of us in MR that sampling is every bit as important as measurement, at least if our goal is to produce accurate estimates. Yet to say that sampling (in the sense of scientific sampling) is no longer a point of emphasis in MR is to understate the case. The whole idea of representative sampling has gone out of vogue. Fortunately for us and unlike the case of political polling, the “correct” answer generally is not known. There is no way for our clients to know if we are right, wrong, or by how much.
An odd circumstance indeed.
Reg Baker is Executive Director of the Market Research Institute International.