Ah, Pediatrics. You publish such ridiculous studies sometimes. We called you out for the flawed study on ‘Facebook depression’, a shoddy study which should have never made it past your reviewers without some serious work.
Now you’re in the news again for a study about SpongeBob SquarePants, the apparently evil cartoon that will turn 4-year-old’s minds into mush after just 9 minutes of viewing. While you also published a somewhat more balanced commentary article alongside the study, nobody seemed to notice it.
And why would they? This study was a siren call to over-generalize and suggest we have found one of the enemies attempting to influence our children. And he wears square pants.
The study itself is short and fairly direct (Lillard & Peterson, 2011). A group of 60 4-year-olds were randomly divided into one of three experimental groups. One group watched 9 minutes of the cartoon SpongeBob SquarePants, another watched a slower-paced cartoon on PBS, and the third group sat drawing. (Why the experimenters didn’t allow the kids to watch the full 11-minute episode of the cartoons is left unexplained, but could’ve negatively or positively impacted the final results; we just don’t know.)
Then the children completed four tasks, three of which are designed to measure executive brain function — such as attention, working memory and problem solving — and one was a delayed gratification task.
Here’s what the researchers found:
The fast-paced television group did signiï¬cantly worse on the executive function composite than the drawing group.
The difference between the fast-paced and the educational television groups approached significance, and there was no difference between educational television and drawing. [emphasis added]
Compared to drawing, kids in the SpongeBob group did worse when the researchers measured these executive function areas — attention, working memory, and problem solving.
But compared to the kids who watched the other cartoon, there was no statistical difference between the two groups of kids. When a researcher says something “approached significance,” that’s a squishy research term to say, “Well, it’s not significant, but it’s darned close.”
Unfortunately, in research, “darned close” doesn’t count. Either something’s significant or it’s not. And even if something is “approaching” statistical significance, that may not mean anything in real life. Statistical significance doesn’t always directly translate into actual deficits in a person — something the kid or anybody else would even notice or affect their actual real-world efforts.
Figure 1 in the study says it all:
It’s not just that SpongeBob impacts kids’ attention and memory abilities immediately after watching the show — apparently so does watching the other cartoon. Only drawing helps a kid with these executive function skills.
But this is glossed over in what the researchers choose to focus on in their discussion section. In fact, they contradict their statement that I quoted above:
Children in the fast-paced television group scored significantly worse than the others despite being equal in attention at the outset, as indicated by parent report.
No, they didn’t. According to your data, the children in the fast-paced television group did worse — but not significantly so — than the children watching the slower-paced cartoon.
The limitations of the study weren’t even mentioned in most media reports I’ve read. They include the small number of subjects studied, and limitations the researchers did note: “only 4-year-olds were tested; older children might not be negatively influenced by fast-paced television. [… We also] do not know how long the negative effects persist or what the long-term effects of habitual viewing include.”
Indeed. If the effects wear off in 30 minutes, it would hardly represent any cause for concern — much less national news media attention. It would be the same as noting people’s pulse rates, distractibility and jumpiness seem to rise immediately after watching 9 minutes of a horror movie. But then they settle down as soon as a person gets reoriented toward the environment around them.
I am just splitting hairs? Perhaps. But it’s also important to note when researchers don’t quite tell the whole truth in their own studies, and how publishers, like the American Academy of Pediatrics, don’t seem to much care.
Reference
Lillard, A.S. & Peterson, J. (2011). The Immediate Impact of Different Types of Television on Young Children’s Executive Function. Pediatrics. DOI: 10.1542/peds.2010-1919
12 comments
I’m glad I’m not the only one who found this study to be a complete fraud. First of all, who’s to say the children watching Spongebob weren’t already slower? The study doesn’t say that the children were all tested on their abilities prior to conducting the study. Every child in the study should have then been given tests after completing all three activities on separate occasions and before each activity they should have been given similar tests to provide a control. What were all of these kids doing before the testing began? Watching Spongebob at home? They said themselves that they didn’t know how long the effects would last. Additionally has anyone watched Calliou? He’s a hooligan and a brat that gets sway with everything. Talk about bad programming.
Kathe, if you were to read the actual study, which I recommend you do, you’ll see that this was a RANDOMIZED trial. The original sample of children was divided into the three subgroups by CHANCE, essentially through a lottery. Therefore, there’s no reason to believe there were appreciable differences among the three groups prior to the sessions of drawing or watching the TV segments. This is the normal procedure in medical/psychological experimentation.
I am so tired of this so-called science being passed off as fact. Its manipulative to say the least. Headlines intended to shock, and terrible science put forth as medical “truth” to the lay. Seriously where are the ethical Boards when you need them?
The failure here is two-fold:
1. Pediatrics, as a journal, apparently has a pretty low threshold in terms of what they approve to publish in their journal. That lowers my expectations for any studies appearing in the journal in the future. A good set of reviewers would’ve noted this discrepancy and, at the very least, asked the authors to tone down their findings in line with their data.
2. Because of the extreme competition and compressed timeline for publishing online, mainstream media (and sometimes others) regurgitate news releases uncritically, without bothering to check into what seem like incredible or over-generalized findings. It takes an additional 10-15 minutes to track down a study and browse through it quickly looking for these things. I guess that’s 10 or 15 minutes journalists just don’t have these days.
First, Dr. Grohol, I want to commend you on the sharpness of your critical eye in spotting the researchers’ lapse when they claimed, at one point, that the children in the fast-paced group did “significantly†worse than the “others‖ incorrectly including the educational TV group. I smiled at your blunt but entirely appropriate rebuke of “No,they didn’tâ€. To have to remind the researchers of the non-statistical significance of their own findings is ludicrous! And were the peer-reviewers watching cartoons themselves while vetting this study?
And “Hail to you, Dr. Grohol†for noticing the “apparent†gap between the drawing and the educational TV groups on the backward digit and Tower of Hanoi tests, and for then pointing out that it suggests the culprit may be TV-watching in general, and not poor put-upon SpongeBob.
But I would like to respectfully dissent from your categorical rejection of scientists’ even mentioning studies that merely “approach†statistical significance—for several reasons.
First, as long as one emphasizes that the finding has not met the pre-designated benchmark of significance, but merely has come close, it permits possibly important though still unconfirmed findings to become known. And isn’t that what you did with respect to the backward digit and Tower of Hanoi results? You properly acknowledged there was just an “apparent†gap, and then benefited your readers enormously by using it to raise the very real and important possibility that TV-watching itself might be responsible and not SpongeBob. I would have hated for you to have deleted that insight from your post just because the gap you noticed was merely “apparent†and not of demonstrated statistical significance.
Second, as I’m sure you know, Dr. Grohol, the choice of a p-value of .05 as the threshold of significance is not based on any incontrovertible, fundamental mathematical or philosophical or statistical “truthâ€. SOME fairly low p-value must be chosen as the criterion for the rejection of the null hypothesis, but MUST it be .05? Not at all. “.05†was clearly selected at least in part because it’s a nice round number. There is no qualitative difference between .045, .05, and .055; any could have been chosen as the cut-off point. In this light it seems to me that a study finding that “approachesâ€.05 shouldn’t be dismissed as lacking all evidentiary meaning, nor should its discussion be discouraged.
Third, in the past decade or so, there’s been an increasing insistence by scientific journals, especially medical journals, that ALL statistical analysis be two-tailed, even if the hypothesis of the experimenter is clearly one-tailed. Why this development has occurred is a fascinating and little-known story, but its relevance to this discussion is that many genuinely one-tailed hypotheses that have (significant) p-values between .026 and .05 are forced into statistical NON-significance by the doubling of the p-value compelled by many two-tailed statistical analyses. Whether the study at issue falls into this category I don’t know, but it very well might.
Fourth, with the growing prominence of Bayesian analysis, and especially meta-analysis, we’re well past the days of a single study or two being considered on their own. Appropriately, today, no hypothesis is considered proven just because it has one highly significant study (and significant replication) supporting it–the totality of evidence must be examined in a meticulous meta-analysis. Likewise, a hypothesis that does not have EVEN ONE statistically significant study supporting it but has a meta-analysis that is very highly significant can be considered so well-confirmed that it forms the basis for medical practice. Scientists therefore take studies that “approach†statistical significance in a particular area very seriously, because they might be hinting at a truth that will only be fully revealed by meta-analysis.
For these reasons, I think the discussion of low p-valued results that fall short of statistical significance is often entirely justified.
In closing, let me note, Dr. Grohol, that I very much enjoy your analytical and probing posts, and I hope you continue your passionate advocacy of your positions.
Dr. Grohol wrote:
“Indeed. If the effects wear off in 30 minutes, it would hardly represent any cause for concern — much less national news media attention.â€
If young children were spending, on average, 9 minutes per day watching TV, then yes I would agree with you. But, in reality young children are spending way, way more time than that watching TV.
http://www.sciencedaily.com/releases/2010/10/101028073743.htm
As I’m sure you know, “Neuroplasticity is a non-specific neuroscience term referring to the ability of the brain and nervous system in all species to change structurally and functionally as a result of input from the environment.†– Wikipedia
With short-term effects of a tiny exposure being so negative for four-year-olds, it becomes very, very important to look at, and find out the longer-term effects of longer exposure to TV.
A number of correlational studies have been done, finding a link between excessive and TV and attentional and behavioral problems. These studies have been dismissed because they were correlational. Now, finally a non-correlational study is done with interesting results, more of these types of studies should be done. It is interesting that when similar studies showed that video games increased certain cognitive abilitiesthere was no controversy.
Here is another non-correlatinal study:
“Middle-class 6-year-olds matched for sex, age, pretest WPPSI IQ, and TV-viewing time were blindly assigned to a restricted TV-viewing group or an unrestricted group. Restricted parents halved subjects’ previous TV-viewing rates and interacted 20 min./day with subjects for a 6-week period. Unrestricted TV parents provided similar interactions but did not limit viewing. Results tentatively suggest that TV restriction enhanced Performance IQ, reading time, and reflective Matching Familiar Figures scores.”
http://www.sciencedirect.com/science/article/pii/0193397380900611
—–
Terry — I agree in general with your observations, but since we don’t know the data about how long the effects last, I can’t say whether time watching directly impacts long-term attention or learning.
If you read the study you cited, you’d see there were some interesting findings in there — findings the authors don’t really emphasize. One, their significant findings applied only to performance IQ — not verbal IQ. The main dependent measure also gives us a full scale IQ score, which for some reason was not mentioned by the researchers (likely because it was non-significant). So the only way the researchers could find some significance in their data was to drill down into a specific component of IQ, and then even further into specific subtests of the measure.
Last, they note a pretty serious set of limitations and problems with their data in a footnote in the study:
Indeed, a followup study to confirm this study’s results would’ve been handy. 31 years later, I could find no evidence a followup or confirmatory study was ever conducted. Meaning we would have to take the results of this small study with a grain of salt.
Dr. Grohol wrote:
“Indeed, a followup study to confirm this study’s results would’ve been handy. 31 years later, I could find no evidence a followup or confirmatory study was ever conducted. Meaning we would have to take the results of this small study with a grain of salt.â€
Agreed! TV correlational studies have generally shown more of a negative correlation between attention and self-regulation abilities (as opposed to IQ) and excess TV watching. So I think it would make sense to do a followup study, but looking more at the effects of restricting TV on self-regulation and the ability to focus.
In fact it would make sense to do a whole slew of laboratory studies looking at the effects of TV on attention and self-regulation. Again a number of studies did find a correlation between attention problems and excess TV, yet there have been (up until the SpongeBob study) no laboratory studies to find out, much more explicitly, the effects of TV on attention.
This would be like doing a bunch of studies finding a correlation between lead and exposure and decreased IQ, and then dismissing the studies as purely correlational and then not doing any kind of followup to find out for sure whether or not lead exposure is dangerous to young children.
Terry
The study claims that
“Children in the fast-paced television group scored significantly worse than the others despite being equal in attention at the outset, as indicated by parent report.”
This is backed up statistically at the very beginning of that paragraph when they report
“Combining across these groups and using age as a covariate, there was a significant main effect of intervention on the composite EF score (P = .01,
pn^2 = .15).”
So while you more-or-less correctly note that there is not a significant difference between the Spongebob group and the educational group (depending on the threshold of significance), there *is* a significant difference between the Spongebob group and the other two groups. Their claim stands.
What I found interesting was that, in the discussion section, the researchers note that encoding of novel events may also contribute to the poor performance of the Spongebob cohort. This strikes me as something that could actually be beneficial to children and it would be interesting to see it separated out from pacing in a future study.
Kevin, the researchers were wrong, and your reasoning is fallacious. Here’s why: If you have a very statistically significant result, where let’s say Group A is higher in performance than B, you can pair all sorts of other groups with A and still have a statistically significant overall result when comparing A and its new partner with B. But if you know that the new group paired with A, call it C, is NOT itself significantly higher than B, and you also know that adding C to A does NOT enhance the statistical significance of the difference with B (versus A alone compared with B), then it’s misleading to refer to the “others†(meaning A and C) as being significantly higher than B. (In the case at hand, the drawing group is A, the fast-paced group is B, and the educational TV group is C.) In fact, it’s entirely possible to imagine situations where A is so much higher than B that even though C is LOWER than B the aggregate A+C is still “significantly†higher than B. I think you would admit that to then refer to A+C as significantly higher than B would be an egregious misrepresentation, even though technically not a false statement.)
Kevin, you commented in your final paragraph, “What I found interesting was that, in the discussion section, the researchers note that encoding of novel events may also contribute to the poor performance of the Spongebob cohort. This strikes me as something that could actually be beneficial to children and it would be interesting to see it separated out from pacing in a future study.â€
Kevin, your reaction was precisely my own INITIAL reaction. In fact, I actually started to comment on that in my previous post. But then, upon reflection, and upon a closer reading of the remarks of the researchers, I realized that my first response was likely not valid.
Basically, the researchers conjectured that because the SpongeBob cartoons contained fantastical elements that caused the children to create new neural circuitry (as they continually “oriented†themselves to the completely novel situations of the cartoons), that would lead to ‘cognitive depletion’ and a diminution of executive functions. At first I thought, “New neural circuitry! Wow! That sounds terrific! Why are the researchers acting as though it’s worthless if not downright bad?†But I quickly realized that new neural circuitry devoted to meaningless things (such as recognizing which of the unfamiliar creatures in the cartoons are good and which are bad) has no direct benefits and probably few indirect benefits to the child. Yes, they’ll likely be better at the “intellectual challenge†of understanding fantastical cartoons, but will there be a more general beneficial impact? Given limited capacity for forming new neural circuits (remember ‘cognitive depletion’!!), whatever capabilities exist should be spent on new neural circuits for concepts in language, mathematics, science, etc.
I think most parents wouldn’t mind if the child’s executive functions diminished for a while after developing new neural circuits for doing subtraction–but after SpongeBob?
Before this study ever came out my wife and I decided to “ground” our 8 year old son from watching Sponge Bob – one of his favorite programs – for bad behavior bordering on ODD. When his viewing was limited his behavior would become better and then we would allow him to watch again. The cycle would start all over again. I don’t know that I buy into the idea that Sponge Bob causes ADHD but there might be something to the program negatively affecting those children who have ADHD and related issues. I would not be so quick to discard this thought.
In your opinion, is watching Spongebob, or any other show, actually beneficial? My 6 year old tends to get a lot more combative and disrespectful after watching Spongebob. After watching too much TV (which he prefers Spongebob over anything), he tends to get in more trouble at home, and school.
Comments are closed.