In the last decade, Kaiser Permanente launched a web-based service known as eCare for Moods, meant to help support patients within their health care system with bipolar disorder. Over the years, there’s been some research to support the use of this online tool to help improve patient outcomes.
In the November 2012 issue of Psychiatric Services, another study was published demonstrating the effectiveness of this free online tool for Kaiser Permanente customers. The study, by Hunkeler et al. (2012), found that depressed patients who had access to and used the eCare system had better outcomes — reduced depression and better overall health.
But although the research team clearly believes this is a “win” for eCare for Moods, after examining their data, I’m not so sure. If it’s a win, it’s more of a statistical win than a win for reducing a patient’s depression symptoms.
First, let’s get out of the way the fact that more than half of the study authors (out of 12) are affiliated with Kaiser, the publishers of the eCare system (parts of which Kaiser has also applied for patents, suggesting a potential revenue motive underlying some of this ((Yes, even non-profits can have a revenue, if not profit, motive.)) ). Obviously, when you work for an organization and are researching that organization’s tool for effectiveness — and possible later license or such — there might be some… ahem… incentives for a positive finding.
The study was designed to measure depression symptomology of participants who were divided into two groups. One group would have access to Kaiser’s proprietary eCare for Moods online web-based support and messaging system, in addition to their usual mental health care. The other would have just the usual mental health care provided by Kaiser. This design helps ensure that the primary difference between the two groups is the eCare for Moods program.
The program itself is comprised of a secure, password-protected website that provides personalized self-monitoring tools, secure messaging with the patient’s eCare manager — a trained psychiatric nurse — 8 depression psychoeducational modules, an appointment calendar, and a monitored discussion group (which you could only access after completing all 8 modules). eCare was only offered during the first year of the study. The second year of the study was just ongoing assessment of how the two groups were still doing, to see if the effects of eCare — if any — wore off over time.
How participants were doing was measured in telephone interviews after the initial interview was conducted in-person. The interviews included a number of different health surveys and questions designed to measure progress.
Which brings me to the second nitpick — why didn’t the study authors use a standardized and recognized measure for depression symptoms? Instead, these 12 researchers decided to make up their own, with no rationale for doing so (and no psychometrics provided on their made-up depression scale):
All interviews included an assessment of depressive symptoms experienced over the previous two weeks with questions adapted from the SCID. This provided a psychiatric status rating of the severity of current depression on a 6-point scale derived from the research diagnostic criteria ((Criteria, I should note, that were first proposed back in 1978 — long before the DSM-III-R made the scene. The researchers list no rationale for using this depression scoring method — as opposed to one of the well-researched and regularly-used depression measures. The researchers also fail to acknowledge this as a limitation in the article.))
Worse, the interviewers then had the subjects recreate a week-by-week blow of their mood for the previous 6 months. By memory. Could most people accurately recall how they were feeling three weeks ago — much less 3 months ago?
So the researchers were reliant on accurate memory recall to fill in the blanks every 6 months. It’s not clear why they didn’t simply have participants keep paper-and-pencil journal logs or use some other more frequent data recording method if they wanted weekly data tallies. ((The researchers make no note of this methodology concern in their limitations discussion. Ideally, the study would be designed to incorporate data directly from patients themselves, without retrospective memory coming into play, or the potential bias of the interviewer.))
Weird.
Depression Scores Decrease — for Everyone
The third problem is that the average depression score at the baseline of the study was 3.88 (eCare group) or 3.65 (usual care group). According to the researchers, depression is scored as: “3, moderate impairment; 4, marked impairment but below DSM-IV criteria for [a depression] episode.” In other words, the average of the participants was somewhere below the threshold to actually diagnose anything but the mildest of depression.
But if you think that’s bad, wait — it gets worse.
After a full year of treatment, the eCare group still averages a 3.00 — moderate impairment (technically, still depressed). They’ve dropped nearly a full point on the scale, but that’s it. According to the criteria, they’re still pretty much as depressed as they were when they started (as a group). The usual care group after one year looks pretty darned similar — a 3.10. While they’ve only dropped a half point (since the group wasn’t as severely depressed as the eCare group), their improvement is less impressive. But again, as a group, still depressed.
At the end of 2 years, no statistically significant change occurred in the eCare group — now it’s at a 2.95 average score. Same with the usual care group — now at a 3.11. ((p = 0.047 in a time-by-treatment repeated-measures design))
Now, unfortunately, the clinical trials record doesn’t note what specific primary outcome measure they had planned to use. So we can’t know if they decided to change the primary outcome measure because it was even less impressive than what we’re seeing here. ((This study took a really long time — more so than usual — to make it into publication. This study was begun in 2002 (and was supposed to include bipolar patients too) with data collected primarily in 2003 and 2004. Its results were first presented at a conference in 2009, and it finally appeared in a journal in late 2012. Draw your own conclusions.))
That’s okay — the researchers had another way to make these results look more impressive than they are. They also measured the “presence of depression” (as though depression was like an off/on switch). Any score less than 3 on the PSR means depression isn’t “present” any longer. Score 3 or higher and you’ve still got depression.
If this sounds arbitrary, that’s because it is. It is one of the ways researchers today get around the fact that a treatment isn’t really as effective as they hoped it would be. (Of course, the eCare group came out on top in this measure.)
Summary
The upshot? Here’s a study demonstrating the statistical effectiveness of a treatment intervention supplement — eCare for Moods — that likely had very little real world benefit for the patients who were using it. One of the areas that stand out in the data with a difference between the two groups — learning new coping skills — could be attributed to the psychoeducational modules the program offered.
These modules, of course, could be offered to the general public today and help potentially thousands learn new coping skills.
The eCare system itself has problems with keeping its audience. While 90 percent of patients logged into the system during the first 6 months, that dropped to under 50 percent in the next 6 months. Only 4.6 (on average) of the 8 educational modules were completed by participants — less than half of participants completed the full 8 (a problem common among virtually all online psychoeducational interactive learning modules).
While I appreciate that eCare for Moods can be a helpful supplement to people with mood disorders — such as depression or bipolar disorder — I feel like this research team may have overreached in extolling the benefits of the program. I’m not so sure that weekly depression severity was significantly different from the usual care group, given the data presented in this article alone.
References
Hunkeler, E.M. et al. (2012). A web-delivered care management and patient self-management program for recurrent depression: A randomized trial. Psychiatric Services, 63, 1063-1071.
