We love a usability study as much as the next person. But we love well-designed, elegant studies that rightfully point out their own limitations and are printed in peer-reviewed journals most of all. We have less love for studies that act as propaganda, or researchers who draw conclusions not supported by their own data.
Bad research can be found anywhere — even by consultants who make their living from doing research on usability. Well-meaning folks like Jakob Nielsen for one. Recently he released a study on his website on the usability of the Kindle, the iPad, a PC and a book for reading a piece of short fiction.
After noting there was no statistical difference between reading on the Kindle or iPad, and then noting that the data did not reach statistical significance for the iPad versus a book, the authors of this report still declare that “Books Faster Than Tablets.”
Well, yes, that’s true on the face of it. But the point of research and statistical analysis in the first place is to go beyond what seems to be true and see if the difference is meaningful or not. After all, data may look like they mean something, but if the statistics don’t back it up, then the appearance of meaning is just an illusion. One that shouldn’t be emphasized in one’s sub-titles, since it’s misleading.
In fact, the data from this particular study found that only reading on the Kindle was statistically different than reading a book. But that’s a far less sexy conclusion than the broader, “Books Faster Than Tablets.”
Alternative explanations for the results were not offered in this study, a glaring omission. The most obvious alternative explanation for the results found is simply that folks have a great familiarity and experience with reading the printed word in a book — often times decades’ worth! On the other hand, the time and experience spent reading from a newer device — whether it be a Kindle or iPad or even the PC — has been extremely limited. In any case, people have not spent decades reading books from one of these devices. So it could simply be a matter of familiarity — books are easy, familiar ways to ingest reading material. The Kindle and iPad are not (yet).
This alternative explanation also explains the difference between the Kindle and iPad data findings (the Kindle was statistically significantly slower than a book, while the iPad was not). The iPad uses a screen that most people are already familiar with, and Apple’s usability expertise in these kinds of hand-held devices is well-recognized. The Kindle, on the other hand, uses a screen technology few people have ever encountered. And while far more usable in everyday situations (such as reading outdoors or not worrying about having to recharge it everyday), the Kindle is simply a more unfamiliar device using unfamiliar technology.
Limitations of the study include its design and subject pool. While 24 subjects may give us sufficient power for statistical analysis, people’s reading styles are different enough that a larger experiment might give us more robust and conclusive results. Asking a person to read the same 17 minute story four times in a row also seems less-than-ideal. While the order of devices in which they read on was randomized, I can’t help but think how boring and monotonous it would be to read the same story over and over again, 4 times in a row.
Offering different types of reading material — both fiction and non-fiction — would also rule out any possible effects that the type of content being used as a measure is not inadvertently affecting reading speed too. Hemingway — the author used in the current research — is a great author, but I’d hardly say he’s to everyone’s liking or taste. Reading material you have little interest in may also impact reading times.
What this study demonstrated was that different table computing devices for reading a work of short fiction don’t significantly impact reading times for those with no specific familiarity with the devices (except on the Kindle, where reading time was significantly slower). Not surprisingly, the iPad, Kindle and book all had similar user satisfaction scores, demonstrating that the devices have no significant usability issues that detract from an end user’s satisfaction of using them.
Read the full study: iPad and Kindle Reading Speeds (Jakob Nielsen’s Alertbox)
13 comments
The most likely “culprit” in my opinion is not that either the kindle or the iPad are less familiar, but that both devices let you look up a word you are not familiar with by tapping the word (in the case of the iPad) or navigating to the word via the joystick (in the case of the Kindle), so that people reading these devices are much more apt to stop and “look up” a word than those who are using a book.
Another perfectly good and legitimate hypothesis that could readily explain the researcher’s findings. Yet not mentioned in the original report.
Those were some good points-especially the methodology that included reading the same story 4 times in a row.
I didn’t read the actual study, but thought maybe technology is affecting research quality? Perhaps researchers are trying to shorten the length of articles, and tend to not elaborate as much? just throwing that out there.
Also, it could be that the researchers did not want to invest too much in theisstudy and instead wanted to do more of a brief pilot study to find the most appropriate ways to better structure future research in this area-including adapting the methodology. Maybe they didn’t have a good foundation to start with and realize they will learn from the possible mistakes or limitations.
I still prefer reading hard copies over any type of computer screen or device although it is not good for trees. Computer monitors of various sorts bother my eyes and overall, I don’t feel well after looking at a screen for considerable lengths of time. So I still print things out before I read them and prefer books over electronic words.
The links to look up words, as pointed out by DS, come with benefits, and although not directly related to this particular study, I find it distracting when using the internet as I can never quite finish reading what I started because I click on links that lead away from the material I sought to read in the first place.
There is a major issue with the kindle for academic use: you cannot reference the page number as you would in a printed text. Is there an option to set the page numbers (will most likely affect font and image size) in iBooks to match a printed version of the book? Thank you.
In response to Michelle, there is a way to reference location in a Kindle book. There are location numbers at the bottom of the screen. The location number for a particular quote in the book will be the same for every Kindle version regardless of the text size. Unfortunately, iBooks does use page numbers which are meaningless because they depend on text size.
I wonder whether the study figured in page turning. It is more frequent and takes a litle longer on a device.
I really wonder about the page turning question also. I have a kindle and a Sony reader. I probably read 4-5 books a week, including paper ones, the LA Times on my door step every day, the NY Times on my Kindle and a ton of documentation online. I know the Kindle is slower because of the page turning but also much easier to read than a regular book because I have terrible vision and even with contacts and glasses on top of them, regular text is still difficult for me to read. Because I use the magnification on any electronic device, the number of pages (and hence, page turns required) is much higher.
Surely, God loves p <= .06 as much as he loves p <= .04
Interesting perspective. I think your points about the limitations of the subject pool are stronger than your point about the p-values and rejecting the Null.
I wrote about it here
http://www.measuringusability.com/blog/books-tablets.php
Comments are closed.