There’s a fascinating article in the Nov. 2010 issue of The Atlantic by David H. Freedman that examines the world of medical research and that suggests much of our empirical, research-based knowledge may be flawed.
Anyone who reads World of Psychology regularly already knows about the problems in a lot of industry-funded studies. But this article suggests that the problems with peer-reviewed research go far deeper than simple for-profit bias. Scientists are biased in many, many ways (not just for monetary gain). And this bias inevitably shows up in the work they perform — scientific research.
This is not a new drum to beat for me — I’ve talked about researcher bias in 2007 and how researchers design studies to find specific results (this example involved researchers who found suicidal method websites when searching for — wait for it — “suicide methods” in Google). We’ve noted how virtually every study in journals such as Psychological Science rely almost exclusively on college students collected from a single campus as subjects — a significant limitation rarely mentioned in the studies themselves.
However, here’s the real troubling aspect — these kinds of biased studies appear in all sorts of journals. JAMA, NEJM and the BMJ are not immune from publishing crappy, flawed studies in medicine and psychology. We think of “respectability” of a journal as some sort of sign of a gatekeeping role — that studies appearing in the most prestigious journals must be fundamentally sound.
But that’s simply not true. The emperor is not only naked — his subjects have hidden his clothes in order to further their own careers.