Research automation: boon or bane?

Hollerith sorting machine, 1890.
Hollerith sorting machine, 1890.

There is a long history of technology being used to great benefit in survey research. The use of punch cards to tabulate the US 1890 Census is often cited as the beginning. More familiar to some of us is the introduction of CATI in the early 1970s, the first successful attempt to represent a survey questionnaire as a computer program and the foundation for much of survey automation to this day.

And yes, I was around in those early CATI days although still in graduate school when the first automated interviews were taken. But by the late 1970s I was starting a career in research and in the roughly three decades that followed a central focus of my job was automation, not just of interviews (on the phone, in person, on the web, etc.) but the whole end-to-end survey process. I sold automation to my employers as a way to reduce costs and cycle time while improving quality. The savings in cost and time were measurable and substantial; the quality improvements more elusive. I tend to agree with Ray Poynter when he says, “Nobody can claim to be cheaper, faster and better.”

Which brings us to the explosion of research automation offerings we are seeing in MR today. The essence of their pitch is that they free researchers from the dreary work (data cleaning seems to get a lot of attention) to allow them to focus on the important, thinking parts of doing research. Who among us does not long for that? And I take them at their word when they say they are cheaper and faster when doing it.

But is the research better? Do these systems free us to think more about the design of the research and the story in the data or do they make such thinking optional? Is there some analogue to worries about spurious correlations in machine learning?

In principle at least, an automated system does not free us from the responsibility to make an informed judgment about the representativeness of the sample, the relevance of the questionnaire given the problem being studied, and the degree to which the questions measure what they are supposed to measure. Tempting as it may be, we skip those steps at our peril.

One of my favorite books form the 1990s is Shoshana Zuboff’s In the Age of the Smart Machine. She makes the point that technology has the capacity to liberate, enable, or enslave and much depends on how we use it. There are automating systems that replace humans with technologies and, in the process, deskill their users. And there are informating technologies that collect, organize, and deliver information about those processes so that their users can make better and more informed decisions. I always worried that CATI was treated too much as an automating system that reduced the importance and value of well-trained interviewers, and by extension the quality of the data they collected. CATI did not make interviewers better. I worry that survey automation systems may be taking researchers down that same road which would be a huge mistake for our clients and for our profession.

Faster, cheaper, and, hopefully, not worse.

Facebooktwitterredditpinterestlinkedinmail

One thought on “Research automation: boon or bane?

  1. (Full disclosure, i work in automation)

    Thank you for raising this topic, Reg. I agree w Annie in that their is a time and place, as an example I am not seeing a world where segmentations or ethnographic research can ever be automated but repeatable functions that live in silos and lack scale certainly can and should.

    Automation as I see it can scale human expertise and if rigor is applied up front you can be rigorous w sampling set up, survey construct and analytical modeling on the back end. As an example, we run digital copy testing, sample against a typing tool and flag variables in each project so that in time we can teach the automation system the key drivers of various business objectives so on a day to day basis the expertise of the researcher who designed it can be scaled across the organization, freeing them to focus on scaling their business.

Leave a Reply

Your email address will not be published. Required fields are marked *