Mechanical Turk to the Rescue of Psychology Research?

One of the problems faced with psychology research — really, with all medical research — is finding enough appropriate subjects to study. Subjects have to be obtained in a way that is representative of the population as a whole for research findings to be generalizable.

Which is a real problem, because as I noted back in August 2010, there are literally thousands of psychology studies based upon nothing more than a bunch of college students from a single campus at a university in the U.S. While young adults who are attending college may indeed help us understand some aspects of human behavior, you can’t just assume that the behaviors you observed in those studies apply to 60-year-old women and men too.

Enter Amazon.com’s Mechanical Turk service to the rescue. Can technology help solve the quandary of finding enough people that are more representative of the population, and do so in a cost-effective manner?

To begin, it may help to understand what Mechanical Turk is. It’s an online marketplace that allows sellers to buy human labor at very inexpensive costs for some sort of online task. “The site boasts a large, diverse workforce consisting of over 100,000 users from over 100 countries who complete tens of thousands of tasks daily,” note the researchers. These workers get paid on a piecemeal basis for each task, sometimes as low as 1 cent. The tasks vary and can be doing anything from browsing a website looking for errors, to filling out surveys.

Ah yes, filling out surveys. A lot of psychology research is based upon having people fill out surveys. Wouldn’t it be nice if we could get people — people who weren’t college students — to do this at something less than $10 per survey… Say, 10 cents?

Buhrmester et al. (2011) set to find out if such Mechanical Turk people would fill out surveys in a way that was consistent with people you might ask to do something similar face-to-face.

First, the researchers surveyed the demographics of 3,006 Mechanical Turk participants, and compared them with a large Internet sample not obtained from Mechanical Turk. They found the Mechanical Turk (MTurk) participants were more diverse than both the Internet sample and the standard college student population sample.

Then they checked on different price points (2, 10 and 50 cents) for filling out a set of surveys that took varying amounts of time (5, 10 and 30 minutes). What they found should not be surprising to anyone who’s used MTurk:

These analyses suggest that participants can be recruited rapidly and inexpensively. Participation rates are sensitive to compensation amounts and time commitments, but our findings demonstrate that it is possible to collect decent-sized samples via MTurk for mere dollars. Even when offering just 2 cents for a 30-minute task, we accumulated 25 participants (in about 5 hours of posting time). Moreover, by increasing the compensation just slightly (e.g., to 50 cents), we were able to obtain the same number of participants in less than 2 hours of posting time.

What good is cheap labor, however, if it’s not good quality and doesn’t meet the psychometric standards that psychology researchers need?

The researchers found that even at the lowest compensation level of 2 cents, payment levels do not appear to affect data quality. Researchers can pay pennies for surveys to be completed, and reliably get good quality data returned.

The researchers computed mean alpha reliability scores across all compensation levels for the six personality questionnaires they administered online. They found that the MTurk alpha scores were within two hundredths of a point of the traditional-sample alphas. They also tested test-retest reliability scores, and found the mean r = 0.88 with a 60 percent retest completion rate. These are all very good numbers and compare favorably with traditional methods.

The upshot of this research study suggests that Mechanical Turk is a reliable alternative to using college populations to obtain survey data — and may even be a better alternative than traditional Internet surveys. For just pennies, people will complete even lengthy questionnaires in their efforts to help others with their projects and research. While more research needs to be done to confirm these findings, MTurk appears to be a marketplace more psychology researchers should be considering as they prepare their next study.

Reference

Buhrmester, M., Kwang, T., & Gosling, S.D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5.