Can a Single Study Have Such Influence Over ADHD Treatment?

Imagine that in a world where thousands of new studies are published every year, and hundreds of studies are conducted on any one condition, that one gleaming, gold-standard study has the ability to completely determine the course of treatment for one condition. For decades.

If you find that hypothetical situation difficult to swallow, you’re not alone. Experts and specialists of a condition such as attention deficit hyperactivity disorder (ADHD) rarely rely on a single study’s results to help guide their treatment decisions. And even when they do, it’s nearly always done within the context of a specific patient’s individualized needs.

So can a single study have such influence over the choice of treatments in ADHD? Let’s find out.

The claimed magical research is the NIMH’s Multimodal Treatment of Attention Deficit Hyperactivity Disorder study published in 1999 (MTA Cooperative Group, 1999). Alan Schwarz, writing for The New York Times, says:

But in retrospect, even some authors of the study — widely considered the most influential study ever on A.D.H.D. — worry that the results oversold the benefits of drugs, discouraging important home- and school-focused therapy and ultimately distorting the debate over the most effective (and cost-effective) treatments.

Considered by whom? Schwarz never says. Clearly it is an important milestone in the understanding of the treatment of ADHD. But all of science is every-changing, and neither researchers nor clinicians look at a study published 14 years ago and says, “Well, we answered that question, let’s close up shop and call it a day.”

So what exactly is the problem with this rigorous, NIMH-funded ((No pharmaceutical funding was involved in the study.)) research?

The study was structured to emphasize the reduction of impulsivity and inattention symptoms, for which medication is designed to deliver quick results, several of the researchers said in recent interviews.

Less emphasis was placed on improving children’s longer-term academic and social skills, which behavioral therapy addresses by teaching children, parents and teachers to create less distracting and more organized learning environments. ((Schwarz, I believe, also doesn’t do the study’s major findings justice, simplifying what the research authors actually said in the paper. They acknowledged, for instance, the importance of considering combined treatment when called for to help deal with non-ADHD domain problems: “combined treatment also fared significantly better than community care for all 5 non-ADHD domains: parent-reported oppositional/aggressive symptoms, parent-reported internalizing problems, teacher-reported social skills, parent-child relations, and reading achievement.”))

The last time I looked, symptoms of impulsivity and inattention are at the core of the definition of ADHD. So it’s not really surprising that the study focused on these symptoms.

Poor social skills, on the other hand, are seen more as a direct result of these kinds of symptoms — an inability to engage in sustained attention to an activity — rather than as a stand-alone problem. People with ADHD have trouble with academic functioning and social skills because of inattention and impulsivity issues. ((Looking at it from the flip side, people with ADHD generally do not have inattention or impulsivity problems because of poor social skills or an inability to cognitively comprehend the educational material they are expected to learn. ))

Nobody would argue that helping children deal with these related issues is vitally important as well. And the evidence is clear — behavioral treatments have been demonstrated to be helpful for children with co-occurring disorders and these kind of related symptoms. But they have been shown to be far less effective in the treatment of the core symptoms of ADHD.

Can One Study Blot Out All Other ADHD Research?

But even if we want to second-guess the design of a 14-year-old study and suggest the researchers should’ve used their crystal ball to not focus so much on the core symptoms of how we commonly define ADHD, you also have to buy into the belief that this single study is all that anyone has read in the ADHD literature. For the past 14 years. ((In fact, not mentioned by Schwarz is that the original 1999 MTA study spawned over a dozen, equally-important followup studies!))

Last time I checked, most experts, clinicians and researchers who study ADHD don’t work that way. Instead, they keep up on the research literature, reading the major studies that come out every single month on attention deficit disorder.

Since the 1999 NIMH study was published, PsycINFO shows that over 2,000 additional peer-reviewed studies have been published on the topic of ADHD treatment. Dozens have been published on the efficacy of behavioral treatments. Not all of them have been positive.

For instance, in a large systematic review and meta analyses of randomized controlled trials of dietary and psychological treatments for ADHD published earlier this year (Sonuga-Barke et al., 2013), the researchers initially found that all dietary and psychological treatments produced statistically significant effects when using raters closest to the therapeutic setting.

However, things changed when blinded assessment was employed: the significant effects disappeared for all but free fatty acid supplementation and artificial food color exclusion (for those with a food sensitivity). In other words, behavioral therapy and cognitive training didn’t meet the cut for being shown as effective treatments for ADHD, leading these researchers to conclude:

Better evidence for efficacy from blinded assessments is required for behavioral interventions, neurofeedback, cognitive training, and restricted elimination diets before they can be supported as treatments for core ADHD symptoms.

Another recent meta-analysis from Rapport et al. (2013) comes to similar conclusions when looking at cognitive training programs designed to help kids with ADHD. The only positive effect they could find was for short-term memory improvement in such programs. Everything else was non-significant:

[…] training attention did not significantly improve attention and training mixed executive functions did not significantly improve the targeted executive functions (both nonsignificant: 95% confidence intervals include 0.0). Far transfer effects of cognitive training on academic functioning, blinded ratings of behavior (both nonsignificant), and cognitive tests (d= 0.14) were nonsignificant or negligible.

Worse, they found the same rater bias effects as the above meta analytic review found:

Unblinded raters (d= 0.48) reported significantly larger benefits relative to blinded raters and objective tests.

In plain language, this means that researchers sometimes introduce bias into their results by using raters to help judge the effectiveness of the treatment intervention. Such raters can be unintentionally (and unconsciously) biased, producing results that, upon further analysis, aren’t as strong as the original research suggested.

Our Take on the Best Treatment for ADHD

There is nothing wrong with calling attention to the emphasis of medication treatment over other types of treatment. Indeed, there is too much quickness — mostly by well-meaning pediatricians and family doctors — in reaching for the prescription pad to treat ADHD. And a reluctance and difficulty in seeking out additional, or alternative, treatments for ADHD, like psychosocial or behavior therapy.

But there is something wrong with a hyperbolic claim that a single study published 14 years ago somehow caused or significantly contributed to this problem. Or that the researchers who were interested in studying the core symptoms of ADHD somehow missed the mark by not significantly expanding the scope (and therefore, the cost) of their study by looking at things that were related to, but were not the core symptoms, of ADHD.

The NIMH study is a solid study that helped increase our understanding of the treatments for ADHD. But it was not the end of the story. One of the followup studies (Molina et al., 2009) to the original MTA study found an important nugget of prognosis for ADHD:

[…. E]arly ADHD symptom trajectory regardless of treatment type is prognostic. This finding implies that children with behavioral and socio demographic advantage, with the best response to any treatment, will have the best long-term prognosis.

In other words, if you’re not poor and have ready access to good treatment and schools, whatever treatment your child responds to best is going to be the best treatment for them. Try different ones until you find one that works best for them.

Our understanding of conditions like attention deficit disorder is expanding and increasing all the time. Science nor knowledge ends with a single study, and it’s a bit silly to suggest it does.

 

Read the NY Times article: A.D.H.D. Experts Re-evaluate Study’s Zeal for Drugs

References

Molina, BSG et al. (2009). The MTA at 8 years: Prospective follow-up of children treated for combined-type ADHD in a multisite study. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 484-500.

The MTA Cooperative Group. (1999). A 14-Month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder (ADHD). Arch Gen Psychiatry, 56, 1073-1086.

Rapport, MD et al. (2013). Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes Clinical Psychology Review, 33, 1237-1252

Sonuga-Barke, EJS et al. (2013). Nonpharmalogical interventions for ADHD: Systematic review and meta analyses of randomized controlled trials of dietary and psychological treatments. The American Journal of Psychiatry, 170, 275-289.