We have met the enemy.

I have been spending time this week wrapping up my presentation for tomorrow’s AMSRS webinar on The Future of Surveys. It’s caused me to step back and take a broad look at what has happened with surveys over the last 75 years or so, and I am once again impressed by what a great source of insight a survey can be, when done properly.

There are two parts to that. The first, and perhaps most important, is the massive amount of scholarship that has gone into leveraging science and theory to establish a set of principles in sampling and measurement that when followed get us to a reliable and useful approximation of the truth. The second is a process for collecting, processing, analyzing, and sharing data that is remarkably efficient in its ability to reduce if not eliminate errors.

The sad reality of contemporary market research is that much of that science is in danger of being lost. Whether that’s due to a lack of creativity in adapting those principles to a world of public indifference and constant pressure to do more for less, or to too many practitioners never having learned those principles in the first place, is open to debate. At MRII, we obviously believe the latter is the place to start.

On the other hand, the efficiency with which we collect, process, analyze, and share data has improved immensely, largely through the creative application of technology. We may have work to do in survey design, but we are great at execution.

A major theme of contemporary MR is that it is time to move on from surveys, mostly to one form or another of big data, however defined. Maybe I’m missing the subtleties of the paradigm shift, but it seems to me that the issues of science and process are still there to be grappled with. Representation and measurement remain important issues. Size still does not trump coverage. Whether a data item really measures what it claims to measure is still important.

And on the process side, the rigor with which “found data” is collected and processed often falls short of what we are accustomed to with surveys. There are major challenges in data governance, curation, and provenance. It is not at all clear to me that these are widely understood and, more importantly, practiced.

The opportunities presented by a world awash in data are undeniable, at least in theory. One might say the same about the survey toolkit. But the use of that toolkit in contemporary MR can be pretty shabby. What makes us think we will do any better with big data?



Leave a Reply

Your email address will not be published. Required fields are marked *