Sometimes you wake up feeling refreshed from a good night’s rest. Other days you wake up feeling a little groggy and less put together.
I can only hypothesize that the latter occurred when the author wrote this post in response to our post pointing out the flaws of associating violence with mental illness based upon the CATIE data.
One of the biggest problems with the violence study done on the CATIE data, which I didn’t explictly mention in my prior post (but which is fairly obvious) is a problem associated with any study attempting to piggyback on another study’s resources. Contrary to the author’s assertion in his blog post, it is not commonplace in social science research to do this. In fact, because of how good study design takes place (a planning process that can take years), studies that piggyback on other study’s subjects, as the violence researchers did, have to live with the original study’s limitations.
Biased Sample, Biased Results
One of those limitations is serious — it chose its sample population based upon the psychopharmacology and medical needs of the study’s focus, not of the violence researcher’s focus. This resulted in a large swath of representative people not being included in the study. Enough people, in fact, to seriously alter the study’s findings had they been included.
The author glosses over this point saying, in effect, “Well, that’s the norm in research.” It may be the norm in psychiatric and medical research, but it’s definitely not in social science research. Especially when that research is seeking to establish cause and effect relationships in specific populations. It’s not common to setup exclusion criteria that have little or nothing to do with the subject under study. The original CATIE researchers had good reasons for their particular exclusion criteria; the violence researchers (Swanson et al.), however, had little research justification for using the same exclusion criteria except that they had no choice. Anything that excludes a significant proportion of the study group you’re researching is going to impact the robustness and validity of your results, as was the case here.
The second point the author makes is again related to the fact that the researchers were piggybacking onto the CATIE study. Since they didn’t have control over these things, we’re just supposed to accept this glaring issue?
Is “Grabbing” the Same as “Being Stabbed by a Knife”?
As for the semantics issue, words have specific meanings, and when people change around words to fit their own needs or agendas, and change the meaning of specific theoretical constructs to fit such needs, it should cause us to sit up and take notice.
Because the researchers changed the wording of the MacArthur Community Violence Interview from “other aggressive acts” (which, in the MacArthur’s team’s view, was specifically different enough from violence as to not include the word violence in its description) to “minor violence,” they make broad, meaningless generalizations suddenly valid. The CATIE violence researchers’ justification (to match a government survey about violence) isn’t sufficient, in our view, since it clouds and distorts the actual data and findings.
We see this in action in how people talk about Swanson’s CATIE study results. Instead of the actual 1.6% difference in violence found, it turns into a whopping 19% difference. If they had left the category to its original name, people wouldn’t simply lump the two together as was done (and is still being done) to justify arguments and the stance that violence and mental illness share some sort of close link. Remember, “other aggressive acts” could include something like a heated argument, where two people end up yelling at each other and one grabs the other’s arm. This is definitely not in the same arena as someone being threatened by a weapon or having a weapon used on them. Yet, if all you did was read other people’s reviews of this research, you’d believe that the mentally ill were 10 times more likely to commit any type of violence with no disclaimer about this differentiation being made to law makers or policy makers.
The 1998 Landmark Study Showed No Violence Link
Remember, the MacArthur Community Violence Interview didn’t appear out of thin air. It came from an important, ground-breaking study by Steadman et al. (1998) that found no significant difference in violence between people with psychiatric diagnoses and the community. This was a study, designed from the ground up, to measure violence. Not an afterthought study that glommed onto a pharmacological study of psychoactive medications.
And yes, I brought up the reliability of self-report amongst the worst of the worst of people grappling with schizophrenia. Anybody who has spent any significant amount of time with a wide variety of people who have been diagnosed with this condition absolutely know what I’m talking about. Now, if the researchers had a more solid baseline set of data than the 42% they did have, that would’ve compensated for reliance on self-report from a historically unreliable population to gain self-report from. But put these two components together, and you have a real question about how valid the data they did find even are.
Apples to Apples
Interestingly, the author didn’t refute the point about what you compare the CATIE violence data with — apples to oranges, or apples to apples? So, even if we were to accept all of the above issues and serious problems with this data, the data is useless without context. Some people have chosen the context of 30-year-old data that used completely different measures of violence as a good control comparison (that magically low 2% number). However, more recent and accurate data — research that uses the exact same violence measure — is readily available via Steadman et al. (1998).
What does that data show?
Excluding substance and alcohol abuse (problems that are widely accepted as being linked to a higher incidence of violence), Steadman’s study found 3.3% had committed violence and 13.5% committed other aggressive acts in the community sample. There is no statistically significant difference between 3.6% (Steadman’s study) and 3.3% (Swanson’s CATIE study).
So the research shows exactly what we’ve said all along — there is no strong or significant link between mental illness and violence.
You’re More Likley to Be Assaulted by Someone Without Mental Illness
Yes, that’s true, if your assault is by a stranger (which is what most people’s fears are).
Steadman et al’s (1998) research showed a 22% stranger violent act rate in the general community, and only a 13.8% stranger violent act rate amongst patients. If you’re assaulted by a stranger, it’s one-third more likely that person has no psychiatric disorder. This was not a statistically significant difference, however, but the data is clearly the opposite of what others suggest is true.
I’d like to compare this with the CATIE violence researcher’s data, but I cannot. The CATIE researchers never bothered to ask who the violence was being perpetrated upon. If you’re doing a violence study, wouldn’t that be an important question to ask and track?
Shame on the Treatment Advocacy Center
By the way, the Treatment Advocacy Center (TAC) should be ashamed of themselves for their fear-mongering ways and promoting the “10 times more likely” number as fact. It is truly embarrassing that an organization of their nature would be misconstruing the research to further its own political, forced-treatment agenda.
References
Steadman, H.J.; Mulvey, E.P.; Monahan, J.; Clark Robbins, P; Appelbaum, P.S.; Grisso, T.; Roth, L.H.; Silver, E. (1998). Violence by People Discharged From Acute Psychiatric Inpatient Facilities and by Others in the Same Neighborhoods. Arch Gen Psychiatry, 55:393-401.
5 comments
I would like to make a few points in response to this interesting and worthwhile exchange. I hope this is helpful:
1. Our CATIE study obtained baseline violence data on about 97% of participants, not 42% as stated by Grohol (and repeated by Erickson.)
2. Regarding generalizability of the sample, we clearly state in the article: “The study excluded first episode patients (who might have been less violent) and wholly treatment-refractory patients (who might have been more violent) and, thus, the findings cannot generalize to such patients . . . 7% of screened patients were excluded for this (or any other) eligibility criterion†(Swanson et al., 2006).
However, that’s not the whole story. We were concerned enough about generalizability that we went to the trouble to systematically compare CATIE participants with a quasi-random sample of 1413 patients enrolled in the Schizophrenia Care and Assessment Program (SCAP), an observational noninterventional study of schizophrenia treatment in usual-care settings in the United States — a study without the exclusion criteria that Grohol mentions. As we explain in the article, the two samples were very similar in their demographic and clinical characteristics — variables that might be considered risk factors for violence. In any event, this issue was addressed to the satisfaction of several demanding independent peer reviewers and editors for the Archives of General Psychiatry.
3. Regarding comparisons of violence rates across different studies: It’s important to note that the ECA violence measure (Swanson et al., 1990; Swanson 1994) included what the MacArthur study would have termed “other aggressive acts†as well as “violence.†Specifically, the ECA measure combined items indicating less serious acts (e.g. †. . hit or throw things at your wife/husband/partnerâ€) and more serious acts (e.g. “. . . used a weapon like a stick, knife or gun in a fightâ€). Using this measure, our ECA study found that 13% of persons with schizophrenia residing in the community had been violent within 1 year, compared with 2% of those without mental disorder. The ECA findings are not directly comparable to the CATIE findings, as Grohol notes. However, if one is going to compare them anyway (as Grohol does), the most correct comparison would be between CATIE’s 6-month prevalence rate for schizophrenia patients with any violence (19%), and the ECA’s 1-year rate for community respondents with schizophrenia with any violence (13%). Grohol’s assertion that there is a “difference of [only] 1.6%†between the CATIE schizophrenia and ECA general-community violence rates is specious; this is comparing serious violence in the former with all (minor + serious) violence in the latter. (Forget about apples and oranges; this is more like comparing a grape to a watermelon.)
Grohol also questions the ECA’s “magically low 2% figure†for the general population’s violence rate. Instead, he recommends using the MacArthur study’s “more recent and accurate data — research that uses the exact same violence measure — is readily available via Steadman et al. (1998).†Well, as I’ve noted above, the measures are not, in fact, the “exact same†(although I do think they’re fairly comparable — that is, if one includes “other aggressive acts†in the measure as we did in the CATIE combined index of “any violenceâ€.) But if one insists, as Grohol does, that CATIE and ECA are NOT comparable measure of violence (notwithstanding that he compares them himself, when convenient), then MacArthur and ECA are not comparable either — for the same reason. You can’t have it both ways. (I think Grohol has opened himself to the charge of statistical sophistry here, though perhaps not the more serious crime of intentional academic mendacity!)
But the much bigger problem with using the MacArthur study to estimate the general community violence rate was clearly stated by the study authors themselves:
“Care should also be taken in making patient-community comparisons. We sampled from the census tracts in which the patients resided after discharge. Many of these neighborhoods were disproportionately impoverished and had higher violent crime rates than the city as a whole. We sampled in this manner to control for exposure to environmental opportunities for violence between the patient and the comparison groups. The comparison group was not intended to be an epidemiologically representative sample of the general population of Pittsburgh.†(Steadman et al., 1998).
That’s probably why the MacArthur study found such high rates of “violence†and “other aggressive acts only†in their community sample — 4.6% and 15.1%, or 19.7% combined, in a 10-week period. (Quite a bit higher, actually, than the schizophrenia sample in the ECA, let alone the non-mentally-ill sample.)
In contrast, the ECA violence study used a truly representative sample of community residents in 3 US communities, with a combined sample size of about 10,000. The MacArthur study enrolled 519 community residents from patient-matched census tracts in Pittsburgh. And the subjects were enrolled from acute inpatient units and followed in the community after discharge. About 17% of the MacArthur study subjects had a primary diagnosis of schizophrenia, while 24% had a primary diagnosis of substance use disorder. The point is this: the MacArthur study was never designed as an epidemiological survey of the prevalence of violence in persons with and without mental disorder in the community. That’s just not what it was, thought it’s often cited as such. Rather, it was designed to study violence risk assessment for patients discharged from acute psychiatric facilities. (As long as we’re talking about representativeness. . .)
4. Regarding our use of terminology in the CATIE study, let’s just consider the primary, ordinary-language definition of “violence†in the dictionary: “exertion of physical force so as to injure or abuse†(MerriamWebster.com). That would include stabbing somebody or shoving them against the wall. And of course there’s a difference between these kinds of two acts; that’s why we coded one as “serious†and the other as “minor†violence in CATIE. But also consider that the Bureau of Justice Statistics defines battery as a violent crime. And battery is mainly what we’re talking about — that is, acts that would qualify as battery if reported to the police. So if you want to know about all violence, we think the threshold should be set there.
Here is another way of looking at it: “Choking†is one of the behaviors the MacArthur violence interview asks about. So imagine that Smith and Jones both attempt to choke their respective spouses. Same intent, same reason — the only difference is that Smith is a robust 35-year-old-man, and Jones is a 75-year-old-woman with arthritis; Mrs. Smith receives a visible bruise to the throat and requires medical attention, while Mr. Jones is uninjured and just needs a cough drop. Now, the MacArthur study would reserve the term “violence†for what Mr. Smith did, but would not apply it to what Mrs. Jones did. We, along with the dictionary and the Bureau of Justice Statistics, would use the term violent to describe both acts, when speaking in general terms, while making appropriate distinctions between them when speaking specifically about severity and consequence.
The distinction between serious violence and minor violence is important, but so is the distinction between absolute risk, relative risk, and attributable risk. You can use the same epidemiological data (our ECA study) to support any of the following three statements: (1) The large majority of people with mental illness do not commit violent acts; (2) People with mental illness are more than 3 times as likely to commit violent acts as people without mental illnes; (3) About 3 percent of the violent acts committed in the community are attributable to mental illess as a risk factor net other causes or risk factors (Swanson, 1994). (Which of these statements you choose to emphasize may depend on the rhetorical point you’re trying make.)
5. Speaking of rhetorical points, finally, let’s consider the statement that Grohol quotes from a Treatment Advocacy Center spokesperson: “The CATIE violence study found that patients with schizophrenia were 10 times more likely to engage in violent behavior than the general public (19.1% vs. 2% in the general population).†The TAC statement seems to imply that our CATIE study enrolled a comparison sample of the general public and compared them to schizophrenia patients in an epidemiological study. We did not do so. Taken out of context, then, the TAC statement could indeed be misleading.
– Jeff Swanson
Steadman HJ, Mulvey EP, Monahan J, Robbins PC, Appelbaum PS, Grisso T, Roth LH, Silver E. Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods. Arch Gen Psychiatry. 1998;55:393-401.
Swanson JW. Mental disorder, substance abuse, and community violence: an epidemiological approach. In: Monahan J, Steadman H, eds. Violence and Mental Disorder. Chicago, Ill: University of Chicago Press; 1994:101-136.
Swanson JW, Holzer CE, Ganju VK, Jono RT. Violence and psychiatric disorder in the community: evidence from the Epidemiologic Catchment Area surveys. Hosp Community Psychiatry. 1990;41:761-770.
Swanson, JW, Swartz MS, Van Dorn RA, Elbogen EB, Wagner HR, Rosenheck RA, Stroup TS, McEvoy JP, & Lieberman JA. A National Study of Violent Behavior in Persons With Schizophrenia. Arch Gen Psychiatry, 2006;63:490-499.
Jeff, thank you for the rigorous response and thoughtful reply. I, and my readers, very much appreciate you engaging in this kind of public dialogue to improve our understanding of your study.
1. You’re correct and I did not mean to say what I wrote. I got the 42% number from the number of patients’ families you were able to interview at baseline:
This goes to the self-report validity more than anything else. If you found that those 617 participants’ families baseline reporting was consistent with the self-report from the participants themselves, then I would grant that in this study, the violence self-reporting was reliable.
You basically have two datasets, though. 42% you have self-report + family report = violence & other aggressive acts, and then 58% of self-report only. I looked but couldn’t find where in the study if you examined these two groups for any differences?
2. Generalizability. Yes, I saw that you compared the demographics between the two groups. I also saw, and you failed to mention, that you didn’t mention in the study whether any of the differences were significant or not. For instance, the CATIE population had seemingly significantly less racial diversity and seemingly significantly higher college attendance. I don’t know what to make of those two characteristics in relation to violence, but I only note that there are some differences between the two groups.
I should also note, since you mentioned SCAP, that “SCAP is a prospective, naturalistic, non-randomized research initiative studying clinical, functional, and economic outcomes in schizophrenia.” Non-randomized being the key point here. Comparing the CATIE sample with another non-randomized sample for research purposes is far from ideal, wouldn’t you agree?
3. I think there was some confusion in my descriptions of data, which led you to believe I was comparing ECA with CATIE data. I am not, but others apparently are (including the NIMH and TAC). I would and did not compare ECA with CATIE data, because, utilizing two different measures, I don’t believe the results are directly comparable.
As for what the Steadman et al. (1998) was or wasn’t, I believe their results speak for themselves:
4. We’re certainly in agreement with regards to violence. Violence results in injury. Battery includes violence, as it is commonly defined, as well as “other aggressive acts.” The MacArthur Interview specifically defined “other aggressive acts” as “battery that did not result in physical injury.” So you can see that while the common definition of violence includes injury, the MacArthur researchers specifically put behaviors that don’t result in injury into a separate, less serious category.
I think it’s important to note the lack of measuring of intent, as well, since we’re speaking of definitions and bringing in the law. Physically hurting someone without intent is not battery. These studies don’t ask or otherwise measure intent, and yet still use legal terms to try and describe behavior and “violence.”
As you know, the more detailed one can be one’s definitions up-front, the less room there is for wiggle and reinterpretation of results. Your study did not find 19% of people with a mental disorder diagnosis committing serious violence, as others have reported. It found just over a 3% serious violence rate, which is comparable to Steadman’s serious violence only findings.
5. I appreciate your understanding that the real reason studies like this get the spotlight shown on them is because sometimes others pull out only the data they want to illustrate a point, even if their comparisons are invalid. This is certainly no fault of your research methods, but it was made much easier when you changed the definition of the second MacArthur category (and on second thought, for understandable reasons give then field you’re working in).
Again, thank you for the detailed followup and reply.
I found your site on technorati and read a few of your other posts. Keep up the good work. I just added your RSS feed to my Google News Reader. Looking forward to reading more from you down the road!