Psychology Secrets: Most Psychology Studies Are College Student Biased

Psychology, like most professions, holds many little secrets. They’re well known and usually accepted amongst the profession itself, but known to few “outsiders” or even journalists — whose job it is to not only report research findings, but put them into some sort of context.

One of those secrets is that most psychology research done in the U.S. is consistently done primarily on college students — specifically, undergraduate students taking a psychology course. It’s been this way for the better part of 50 years.

But are undergraduate college students studying at a U.S. university representative of the population in America? In the world? Can we honestly generalize from such un-representative samples and make broad claims about all human behavior (a trait of exaggeration fairly commonplace made by researchers in these kinds of studies).

These questions were raised by a group of Canadian researchers writing in Behavioral and Brain Sciences journal last month, as noted by Anand Giridharadas in an article yesterday in The New York Times:

Psychologists claim to speak of human nature, the study argues, but they have mostly been telling us about a group of WEIRD outliers, as the study calls them — Westernized, educated people from industrialized, rich democracies.

According to the study, 68 percent of research subjects in a sample of hundreds of studies in leading psychology journals came from the United States, and 96 percent from Western industrialized nations. Of the American subjects, 67 percent were undergraduates studying psychology — making a randomly selected American undergraduate 4,000 times likelier to be a subject than a random non-Westerner.

Western psychologists routinely generalize about “human” traits from data on this slender subpopulation, and psychologists elsewhere cite these papers as evidence.

The study finds that American undergraduates may be particularly unsuitable — as a class — for studies about human behavior, because they are so often outliers in their behavior. Both because they are American (yes, it’s true, American behavior is not equal to all human behavior on Earth!), and because they are college students in America.

I don’t know about you, but I do know that my interaction with others, the world around me, and even to random stimuli is very different now in my 40s than it was when I was a young adult (or teenager, since most freshmen are only 18 or 19). We change, we learn, we grow. Generalizing human behavior from people of such a young and relatively inexperienced age appears short-sighted at best.

Scientists in most fields typically look for what’s called a randomized sample — that is, a sample that reflects the population at large. We hold large corporations accountable to this gold standard — the randomized sample — and the FDA demands it in all drug trials. We’d be aghast if the FDA approved a drug, for instance, upon a biased sample made up of people not representative of those who might end up being prescribed the drug.

But apparently psychology has been getting away with something far less than this gold standard for decades. Why is that?

• Convenience/laziness — College students are convenient to these kinds of psychology researchers, who usually are employed by universities. It takes a lot more work to go out into the community and garner a randomized sample — work that takes a lot more time and effort.
• Cost — Randomized samples cost more than convenience samples (e.g., college students at hand). That’s because you need to advertise for the research subjects in the local community, and advertising costs money.
• Tradition — “This is the way it’s always been done and it’s been acceptable to the profession and journals.” This is a common logical fallacy (Appeal to Tradition) and is a weak argument to continue a flawed process.
• “Good enough” data — Researchers believe that the data they gather from undergraduates is “good enough” data to lead to generalizations about human behavior more globally. This would be fine if specific research existed to back up this belief. Otherwise the opposite is just as likely to be true — that this data is fatally flawed and biased, and generalizes only to other American college students.

I’m certain there are other reasons researchers in psychology continuously rationalize their reliance on American college students as subjects in their studies.

There’s little to be done about this state of affairs, unfortunately. Journals will continue to accept such studies (indeed, there are entire journals devoted to these kinds of studies). Authors of such studies will continue to fail to note this limitation when writing about their findings (few authors mention it, except in passing). We’ve simply become accustomed to a lower quality of research than we’d otherwise demand from a profession.

Perhaps it’s because the findings of such research rarely result in anything much useful — what I call “actionable” behavior. These studies seem to offer snippets of insights into disjointed pieces of American behavior. Then someone publishes a book about them, pulling them all together, and suggesting there’s an overarching theme that can be followed. (If you dig into the research such books are based upon, they are nearly always lacking.)

Don’t get me wrong — it can be very entertaining and often interesting to read such books and studies. But the contribution to our real understanding of human behavior is increasingly being called into question.

Read the full New York Times article: A Weird Way of Thinking Has Prevailed Worldwide

Reference

Henrich, J. Heine, S.J., & Norenzayan, A. (2010). The weirdest people in the world? (free access). Behavioral and Brain Sciences, 33(2-3), 61-83. doi: 10.1017/S0140525X0999152X