I hate to give press to a “research firm” that doesn’t know the first thing about reporting statistics or basic methodology in their own “research” report. I guess that’s what happens when you get a bunch of people together who are mostly technologists, not statisticians or social scientists.
This past week, MyType, a Facebook personality application that takes your data and then sells aggregated reports based upon your answers to their quiz, released a report about the iPad. They suggested that iPad owners and those looking to buy one were “selfish elites” while those who were iPad critics were characterized as “independent geeks.”
You can already tell that this isn’t exactly going to be a scientific analysis, right?
First off is MyType’s reporting of its results. What isn’t included in the actual report are the actual statistical tests used, statistical significance (p values), and numbers of subjects in each group studied. You know, the basic kinds of data other researchers are looking for in which to evaluate the results presented. Without that basic information, this kind of “research” is no better than an undergraduate’s paper for their Psychology 101 class.
A person by the name of John Hall called them out in their comments section, at which point they started giving broad statements about p values (but again, no specifics, breaking each analysis down, which is what you’d expect in market or scientific research). Dr. F tried educating the MyType people about basic statistical theory, since they apparently don’t know much about how to design a good research study.
What’s also clear from the comments section is that someone is editing the comments to remove things they later regretted saying. John Hall quotes apparently Tim Koelkebeck saying that because he has a computer science degree (apparently a bachelor’s), that somehow qualifies him in research design and methodology as well as statistics. Odd.
But the biggest giveaway that this isn’t a serious research study is the adjectives chosen to describe the two groups of people. They took some arbitrary personality traits to describe each group. They could have just as easily said that non-iPad owners are neurotic, insecure, proud and careless. Is that a legitimate characterization of non-iPad owners, though? Of course not, that’s only what their tiny, biased dataset shows. Given that doesn’t even pass the smell test, it would’ve made most researchers pause to wonder if they were going down the wrong road.
Going by their own data, they could have characterized iPad critics to be intolerant, dislike charity and friendship, aggressive and cynical teenagers. Wow, no surprise there — you’d have to tease out whether gender or age were skewing these results (and offer an alternative explanation to the traits noted). The “researchers” could have also described iPad owners as family-oriented, smart, high-achieving adults who are both imaginative and sophisticated.
But they didn’t. They tried to summarize a bunch of disparate traits into catchy marketing phrases to make news headlines — phrases that were neither particularly accurate, nor particularly scientifically valid. That didn’t stop dozens of legitimate, mainstream news outlets like CNET, ZDNet, and even Wired from reporting on the results without raising an eyebrow about the lack of scientific rigor or method.
Read their “conclusions:” iPad Personality Clash: Elites vs. Geeks
15 comments
For the record, I do not own an iPad nor have any intention of buying one. I do, however, own an iPhone.
I am actually horrified that you take mytype at all seriously and anything more than warranting no more attention than is paid to the weeties box over breakfast….it doesn’t make the news
Hi John,
And how professional is journalism that doesn’t even try to reach out to its subject? That’s what we’ve been getting from the press, including here: a bunch of articles (including all the ones you quoted) written up without speaking a word to MyType.
I put up a blog post, not an academic paper. Yes, I believe our methodology was more or less sound and that the data is reasonably rigorous. Did I write up our methodology in painstaking detail? No, if you take a look at the blog, it’s our second post. Next time we’ll better plan for all of this attention. I’m going to write up our methodology in detail shortly.
The psychological measurements are based on (did you read the post?) the Big Five and Schwartz’s Value Survey, both very well-respected instruments. I assume you’ve heard of them.
Also, you mention “teasing out” the affect of gender and age on the personality results, when we stated multiple times that we weighted the sample by gender, age and personality in order to minimize this effect. (Again, did you read the post?)
And sure, the “selfish elite vs independent geek” in the title is an oversimplification, but it does point to main themes in the data. As blogger yourself, I hope you can appreciate the need to simplify and intrigue (or provoke) in a title. My assumption was that our readers would not react to the title in such simplistic ways, which MyType readers (I know them in the comments) haven’t. It’s only when the study gets grossly summarized by other journalists without the context that readers are misled.
I’d be happy to discuss our survey methods and results with you further, if you’re interested. Or you can just pile on inaccuracies in this game of blogger telephone. Some of your criticisms are valid, but I think if you actually spoke to me you’d find that others are unfair.
By the way, I like psychcentral a lot and believe we can be a good source of material for you. We strive to be rigorous, resources permitting, and we work with very respected psychologists like Sam Gosling and Jon Haidt.
-Tim
oh and whether Iphone or Ipad…only IDIOTS buy them you don’t need a survey to know that….
all you have to know is that apple has patented a system to lock devices during advertising and force response from the user once the advertising media is completed in order to unlock it again….
which will be released to all internet enabled apple products when they roll it out…..the only apple products in this house are Ipods….and they don’t access the internet.
I would not waste my money on an Iphone or Ipad there are cheaper more powerful devices on the market since long before the Ipad or Iphone rocked up.
I read the report twice, front to back, searching for the basic statistics one would expect from any kind of market research firm. Lots of pretty graphs in that report! And it looks so slick!! But sorry, none of that messy “data stuff” that actually makes the rest of the report worth reading… So while I appreciate your defense of your work, it still lacked the foundation we would expect from something purporting to give some type of insight into iPad owners or critics.
I didn’t ask for the psychological measurements, I asked for the statistical methods performed. Those are not in the report.
How exactly did you “weight” the age factor to take into account the age biases in the samples? There are many ways to do this, some right, some not so right.
When you make the basis of your company on selling its research acumen to future customers, and then come out with something that clearly demonstrates a lack of scientific and research grounding in basic fundamentals, yes, you’re going to be called on the carpet for it. It’s really not my job to provide free statistical and research design consulting about someone else’s research, is it? Maybe you should have hired one less engineer and one more statistician or someone with a 10 year research background.
Now, if you were releasing this is as just a “fun” observation instead of a serious attempt to break into market research based upon your unsuspecting customer base, you probably would’ve received somewhat of a different reaction from people who do real research for a living.
This is the first sensible article written about this travesty of a survey about iPad users. I own an iPad. However I am NONE of the things they say I am. I drive a Kia, work for Lowes, make less than 25k a year, people tell me I am friendly and likable, I am also a hard worker. So where they get off passing judgement on us by use of a FACEBOOK survey (insert laughter here) is beyond me. I use facebook but am not addicted to it. It’s only a form of communication for me. If anything this survey is ONLY a partial way of looking at the type of people FACEBOOK iPad users are… NOT ALL iPad users. Even then it’s a stretch. The iPad users I know are all different sorts of people. Some nice, some not, some have money, most do not, etc etc. And I didn’t buy the iPad as a status symbol or anything… I just thought it was cool and would get a great deal of use out of it. It’s also helping to save me from putting a whole lot of excess usage on the iMac… thus prolonging the life of my iMac. Btw this was posted using my iPad. It is a truly useful device. Not an elitist status symbol.
John,
We’re a small startup company following the “lean startup” model (probably much smaller than you’re thinking, based on the hiring a statistician comment). This is a great principle for startups that says you should first launch an ok product to see if there is any interest rather than expending all of your limited time and funding trying to launch the perfect product that you’re not even sure you can sell. If you get interest in the ok product, then you iteratively improve. If you don’t, you move on to the next idea.
So we put out an ok product. The methodology was reasonable, but we never claimed it was comparable to an academic paper or a full blown market research firm effort. With that said, we did collect a ton of data and I think that our sample, despite how many people criticize it, is much better than the college student samples of many academic papers and the paid volunteer respondents for many market research firms. 20,000 is also a much larger number than the typical study.
How did we weight for age? We broke our sample down into age, gender and personality strata. For instance 18 year old women with a given personality “type” are one stratum. We then weight respondents in each strata so that our total sample is reflective of the US population, based on census data for age and gender, and very solid personality distribution data (which is kind of our secret sauce). If 18 year old women with personality type X are 2 times more common in our sample in the US, then they are each weighted as 0.5 people. If 45 year old men with personality type Y are 50% as common in our sample as in the US, then they are each weighted as 2 people. We didn’t have to use very large or small weights with any of the strata, the vast majority were between 0.5 and 2.
I realize this is a fairly simplistic normalization technique, but I argue that it provides reasonable enough rigor to make the results worth paying attention to. We also used methods to remove “bad” respondents. And, like I said, our psychological and demographic measures are very sound.
The problem I have with your post is that, in the absence of methodological information, you simply assume MyType is completely incompetent and/or fabricating without even making an attempt to ask us first. Posting about our lack of methodological details is understandable. Your headline and much of your article, though, is a bit unfair. In many ways I feel like we’re on the same side. MyType wants to get serious psychological research in front of the masses, rather than buried in academic journals. We can’t get there by headlining an article with an academic paper title. We’re walking a fine line, I know. I’d appreciate constructive criticism, though, rather than a complete tear-down before we even get off the starting blocks.
Thanks for your response, I look forward to your additional thoughts.
-Tim
Tim, according to your own website, you have two PhDs on your “scientific advisory board” (Sam Gosling, PhD and Jonathan Haidt, PhD). If they aren’t advising you on how to do and report on basic statistics expected in something purporting to be a research report, you should find some academics who can. And since reports like yours are certainly nothing new, you have a historical archive of literally thousands of such reports you could have drawn on for examples from firms who’ve been doing this for decades.
I said nary a word about your competence. My complaint is that MyType, like many startups, are more interested in getting publicity first (by contacting media outlets about your report), and getting it right later. You still believe you have it mostly right. Well, you had the right idea, arguably. But you got so much of everything else wrong. And instead of just coming out and saying, “Wow, we really messed this up, and learned a lot through the process,” you continue to defend the work with phrases like “reasonable enough rigor to make the results worth paying attention to.” Admitting mistakes and being modest about things can go a long way to cultivating professional relationships with others who are in a position to help.
“Lean startup” model and launching an ok product is great for technology. It’s absolutely a wrong model for research though, since data can be manipulated quite easily through using the wrong statistical methods. There are entire books written on this topic — it’s no wonder you got called out on it.
Good researchers/statisticians, like good Facebook app developers, generally don’t work for free or goodwill. You would no more assume that a course or two I took in pascal programming 25 years ago means I should be writing C++ programs or developing web apps, than I would assume someone who’s had a statistical course(s) is qualified out of the gate to do good quality research. That’s why people get PhDs in this stuff — it’s hard to get right, and do it well.
Having said that, though, there is a lot of potentially inexpensive talent available if you just look for it. A PhD grad student can be a great resource if they’re already publishing (and most in the social sciences are). And then hiring a freelance consultant — a statistician or experienced researcher — to walk through their methodology before you get too far is money well-spent. I bet there’s probably more than a few researchers out there who might even do it just for fun, given what you guys are trying to do. For instance, a few of them replied to your blog comments already — reaching out to them privately might not be a bad idea.
And one last piece of advice — nobody hires research firms pursuing an agenda or with an obvious bias. Researchers are valued (and ultimately prized) for their objectivity going into the data from the start. Your report did not read like an objective research report, but rather like a biased op-ed that was looking for data to support an existing opinion or belief. Even if that wasn’t your intent, that’s the way it came across to many people who read it.
Great read and comments, John. I’m glad that someone took the time to write this up.
John, thanks for taking the time to address the many flaws with MyType’s conclusions and methodology. However, after reading the back and forth you had with Tim here, as well as his own remarks in the comments section of his own website, I’m convinced that he just doesn’t get it, nor does he seem to want to get it either. I suspect that he was ill prepared to defend MyType’s published data – which any grad student could have told him is the first thing you prepare for; defending your conclusions.
John,
I perhaps should not have pulled Sam and Jon into this, as they were not at all involved in the making the iPad Opinion Profile. They advise on which existing surveys provide the best measures of personality, values, morals etc, help me get in touch with the survey creators, and ensure that the surveys are effectively implemented.
I think underlying our misunderstanding is the fact that you’re evaluating us as a “research firm” when we never claimed to be one. Wired labeled us a research firm, remember they didn’t speak to us before publishing their erroneous article. We’re a web app, read our about page. We produce valuable data and could become the basis of a research firm. But we’re not a research firm now, and the iPad Opinion Profile is not a “research firm” product. It was no more or less than the basis of a blog post.
When I defend our efforts as reasonable, I mean that they are reasonable for a blog post. I think we did a decent job of publishing interesting, worthwhile information. I don’t see how one could argue that the basic correlations we highlighted – wealthy people are more likely to own the iPad, people who value self-direction are more likely to be a critic, etc. – are worthless. Sure, our figures may be off some due to methodological issues, but that’s not a big deal to me in this context. No one is hinging on the exact numbers, it’s not science. I think our basic methodological tools put us in the right ballpark. If you think I’m wrong about even that, I’d like to know.
Our next profile will have improved methods and methodology reporting. We’ll be taking another step in the research firm direction, but still not labeling ourselves one. We will, as planned in the event of a “successful†initial opinion profile, contract a statistician. If anyone reading this is interested, please contact us.
We ultimately intend to partner with a research firm that knows better than we what do with our data, freeing us up to focus on the features and user experience of our web app.
As for your agenda/obvious bias comment, I honestly didn’t have one. I did not intend the label “elite†to be taken as a pejorative, it’s defined in all positive terms: high income, high education level, high power, high achievement and sophistication. I think the word “elite†sums up all of these strong correlations well. Why did I choose selfish? After defining the elite label, selfishness was the only other thing that stood out to me. The list of strongest correlations for iPad Owners, focusing only on demographics (except race), interests, and valid psych measures that were not already “claimed†by the elite label is: business, low self-direction, 40-49, 1 child, male, finance, low benevolence, 3 children, imaginative, low altruism. The themes that stand out to me in this list are: business and finance, 1 child and 3 children, and low benevolence and low altruism. The first pair seems related to being an “eliteâ€. The second pair was weakened by the fact that the correlations with 2 children and 4+ children are weak. The final pair is strengthened by the fact that iPad Ownership also substantially correlates with being detached and highly valuing stimulation and hedonism (both fairly self-centered values). So I chose the final pair and summarized them with the word selfish.
Probably a longer explanation than you cared for, I know, but these comments are getting some media attention on their own, I thought some readers might appreciate the details.
Thanks again,
Tim
Tim, if it walks like a duck, quacks like a duck and looks like a duck, it’s a duck. Your PDF is entitled, “iPad Opinion Profile,” and subtitled, “The Personality, Values and Demographics behind consumer opinions of the iPad.” It is organized exactly like a marketing research report. It has an executive summary. It has dozens of colorful graphs. And it uses a significant amount of research terminology throughout the report. You went to a lot of trouble to produce something for just a “blog post.”
I write about data and studies and even in-house surveys we do from time to time for blog posts. I never generate PDFs that look like research reports, because that would be deceptive.
You can backtrack now and say you’re not a research firm nor did you ever intend to publish this as a research report, but nobody writing a simple blog post goes to the bother of putting together a paper PDF to support a “web app” unless they’re trying to look like something they’re not. Furthermore, one doesn’t spend as much time as you have after-the-fact trying to defend your methods (even editing your initial blog post to add them into it) unless you have something more to lose. If you’re just a “web app,” you shouldn’t care — people are still going to use your web app.
Besides, what’s an “opinion profile” — isn’t that just a fancy way of saying you’re conducting an opinion poll with built-in psychographics based upon your Facebook data? Whether you acknowledge it or not, surveys are research.
As for your subjective descriptions, I doubt few serious researchers looking at what data you provided in the PDF would come away with the same interpretations you did. Walking us through your analysis process doesn’t make it any less flawed.
But you summed up what this report is right here:
Survey research actually is science. And the sooner you and your “web app” company get a clue about that most basic understanding, the better for your next clients and report.
Of course survey research is science. I was only making the statement that the iPad Opinion Profile should not be evaluated as science. (But again, so that I do not get quoted out of context: the data is still useful.)
I see a progression:
Anecdotes -> Informal Poll/Data Mining -> Market/Academic Research, with shades in between. All levels are useful, for what they are. Something like the OkTrends blog (http://blog.okcupid.com/), for instance, falls between the second and third. I find what they publish to be very insightful, but I don’t take it as science. The iPad Opinion Profile falls in that range as well, admittedly to the left of the typical OkTrends blog post. I’d like the MyType blog to be as good as OkTrends in the coming months. I’m not aiming for it to be like Forrestor’s blog.
I see how the PDF report could be misleading. It was generated to make clear to potential customers that our data can be provided as a service. In lean startup fashion, the profile is basic and lacks many elements of a full-blown research report. We now have companies contacting us to purchase an opinion profile, and we’ll make clear to them the nature of the data and the profile, i.e. that they shouldn’t be comparing our report to a Forrestor or Nielson report. And of course our price is a small fraction of what a market research firm would charge.
The PDF wouldn’t have been so misleading for you and others if you chanced upon it. Given the way Wired and other press described it, though, you came with certain expectations, expectations that our blog alone does not create.
I care to defend our methodology for the iPad profile because we have a reputation to defend. Not a reputation as a market research firm, but a reputation as an honest company. Many readers and bloggers, like you, assumed the worst about us and claimed or implied that our entire iPad dataset is worthless.
Of the many criticisms across the web, yours was the most reasonable, so I chose to defend MyType here. I acknowledged most of your criticism as valid, if one thought the iPad Opinion Profile was supposed to be science, and mainly have taken issue with your critiquing us for not being something that we never claimed to be.
After going back and forth, a new disagreement seems to have emerged: you seem to think there is no place for survey insights that are not extracted via the highest methodological standards, and I do. I think people enjoy reading about them and companies will pay for them. And I think that they represent real value. They’re not at the scientific end of the continuum of value, but they’re past the point of “worth something”.
Of course when I say “not extracted via the highest methodological standards”, I do not mean that insights with no methodology are worthwhile. I’ve described our level of methodological rigor, and I believe it’s enough to make our data worth something.
Thanks for doing an in-depth research, John. Good thing someone still do their research well. 🙂
Comments are closed.