At IIEX NA, Leonard Murphy moderated a discussion with Melanie Courtright of Research Now and Frank Kelly of Lightspeed on improving the mobile experience for research participants. “Ten years in, mobile experience is still an issue,” Lenny said.
I’ve paraphrased the commentary below.
Melanie: Mary Meeker’s Internet Trends Report showed that smartphones are leveling off, indicating the industry is maturing, but we still haven’t figured out how to do great smartphone research. At Research Now, we classify surveys as mobile friendly, mobile optimized, mobile possible, and mobile impossible: our averages in each category haven’t changed. In fact, we haven’t changed the survey since I grew up writing surveys for phone… and for paper.
Frank: Five years ago, we saw a big increase in members joining the panel from mobile, but they quickly left the panel when they found out that they couldn’t really take the surveys on their phones. I did research on survey modularization – treating a symptom, length. Then there was the app phase, which is brilliant for on-the -go research but it doesn’t do much for shortening surveys either. We need shorter and better designed surveys: 80% of Vice traffic is mobile, so 80% of traffic from under 35 year olds. Mel and I have shared results on representativity on non-mobile-friendly surveys. There are differences when you do in it a mobile friendly way or not. I’ve written hundreds of surveys but only recently tried my hand at mobile-friendly surveys: in 6 or 8 months, I’ve found that it really is difficult. You do have to live with certain constraints. I started going down the path, but I’m writing mobile-friendly surveys, doing research on research, so I can afford to get it wrong. I learned some things and am getting better. One thing I want to show you: I started to write my surveys in these templates [holds up page with three silhouettes of smartphones]. It really works. If it doesn’t look good in a phone template, then it won’t get good results. So this is something that I now do for every mobile survey.
Lenny: We experienced this with the GRIT survey. Every negative comment, and there were a lot, was about the survey not being mobile friendly. I’m preaching to the choir but I’m still not doing it.
Mel: You are trading one problem for another problem. For customers of companies like Lightspeed and Research Now, representativeness is still king. You may be keeping some trended data but you are getting further and further away from truth. Lightspeed and Research Now will set competitive concerns aside. We are sharing data on mobile engagement and behavior. We are looking at the impact on representativeness when the surveys aren’t mobile friendly.
Rolf : Can you bring your panelists together for a true mobile panel…?
Mel: The problem isn’t having enough mobile respondents. We have mobile respondents for days. We take steps to prevent them from taking lousy surveys. We don’t have enough mobile content for well-designed mobile surveys; 40% of all research is on mobile devices.
Frank: We are stuck with the dilemma. Do we relax our standards for surveys we send to mobile respondents?
Mel: We want to show that the pain of changing the survey to be mobile friendly is less than the pain of not changing the survey. Make this a problem of protecting the data: people don’t change unless they experience pain. We tried to use carrots. We’d give good discounts for a good survey but then we’d get delivered a mobile-impossible survey, and they would still want the discount.
Lenny: GreenBook can support this however we can. How do we move this from perennial topic at every event? If we haven’t gotten it right for mobile in 10 years, when will we? Will we get it right before the next wave, virtual-reality research? We don’t have a lot of time. The world has moved on with us.
Jeffrey Henning is president of Researchscape International.