Back in 2009 at am MRIA conference in Toronto a speaker (whose name you would recognize but I’d rather not embarrass him) led off with his presentation by saying, “We have allowed this industry to be taken over by venture capitalists and technology geeks.”
I was reminded of this while watching some of the videos from SAMPLECON 2016. I had become intrigued by what went on there after meeting up with several old friends from the sample biz at the Quirk’s event in Brooklyn. All worried that online sample suppliers had lost their way. So I had a look and found it hard to disagree.
In the run-up to the event GRIT circulated a short survey on online sampling that collected a few hundred responses roughly split 50/50 between buyers and suppliers. Some results were presented at the conference but as far as I can tell have not yet been released beyond that. I also gather from the videos that a substantial proportion of the conference attendees started the survey but did not finish it, which is, well, interesting. So we probably should not make too much of a small number of interviews with a convenience sample and high breakoff rate, but let’s suspend disbelief and take it at face value.
One of the goals of the survey was to learn more about what buyers want in samples. There are two things: (1) quality respondents and (2) representative samples. The first is a problem that the established panel companies have pretty much solved and most recent research has shown was not an especially big biasing issue to start with. That message seems not to be getting out, or perhaps undermined by the misinformed. The second obviously is critical and there the industry seems to have no answer. One presenter asked the question, “What is a representative sample in 2016 anyway?” I, perhaps naively, assume it’s the same as it was in 2006, 1996, 1986, and so on. The amount of progress the industry has made on this issue over the last two decades is beyond discouraging, especially as it becomes clearer and clearer that the current practice of probability sampling is not as reliable as it once was.
So what are sample companies doing to meet the expectations of buyers? Mostly it seems they are working on process (technology) with the goal of delivering more and more potential respondents at an ever lower price per complete and preaching the gospel of engagement, which, too, is a lot about technology (interfaces) with a hefty dose of blaming buyers for surveys that are too long and not device agnostic. If we would just design short surveys that run on mobiles, the argument goes, then greater respresentivity will naturally follow. Device agnostic surveys may make respresentivity easier, but that’s not the same as actually doing it.
Way back in the 1990s when all of this online stuff started the pioneer at the time (what eventually became Harris Interactive) recognized that if online is to be accepted by buyers they needed to develop a model that could be used to correct the obvious bias, just as probability samplers had long tried to correct bias due to nonresponse using (mostly) demographic weighting. It is a sad commentary on the current state of online sampling that few suppliers are paying much attention to this sort of thing any more, relying on relatively simple quota sampling and screeners that ask whether you bought or are considering buying this or that product. It’s especially odd when you consider that buyers are asking for it.
To be fair, what buyers did not say they want but suppliers are focused on doing is delivering sample at lower and lower prices. This would not be the first time suppliers complain about clients wanting higher quality while insisting on lower prices. Nonetheless, I think we sell ourselves short if we can’t figure out how to use technology to generate more representative samples. One of the SAMPLECON presenters said something like our needing to build technology that expresses the mind of a trained researcher. Venture capitalists and technology geeks, please take note.