Archive for 'Quant Methods'

[Ed. 9 September: Now that events are unfolding a bit more slowly, and people have had a chance, for the most part, to learn about the basic facts, I've moved my on-the-fly updates to the bottom of the post, so that the original text is easier to find. I will try to update all the broken links next week.]

Original Post

A couple of people have sent me the link to this debacle of two researchers attempting to study what they call the “Cognitive Neuroscience of Fan Fiction” (further historical background here and here, collated links there, and information about the original research (which somehow doesn’t get around to mentioning that the research is designed – not for academic publication – but for a popular book whose working title is Rule 34: What Netporn Teaches Us About the Brain) in the researchers’ background information).

As someone looking on from outside the fan communities directly involved in this mess, the whole thing unfolds something like a live action version of the phenomenon Justin Kruger and David Dunning discuss in their “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” (1999, Journal of Personality and Social Psychology, Vol. 77, No. 6., p. 1121-1134).

Kruger and Dunning are interested in whether, below a certain level basic competence, it becomes very difficult for people to improve their skills – because they are, in fact, too incompetent to be able to tell the difference between competence and incompetence in the first place. They take as their point of departure the story of hapless bank robber McArthur Wheeler who – some of you will remember from my previous post on this article – robbed two banks in broad daylight without any disguise and, when arrested almost immediately based on the bank security footage, burst out: “But I wore the juice!” Mr. Wheeler was evidently under the impression that, by rubbing lemon juice on his face, he could conceal himself from security cameras (Kruger and Dunning 1999: 1122).

Assuming this mess is not some sort of elaborate research-themed performance art, or the result of a revenge-fuelled identity theft, researchers Ogi Ogas and partner Sai Chaitanya Gaddam are trying their best to demonstrate to the world that they are something like the academic research equivalent to Wheeler. They have blundered into an online community whose members write and read, among other things, erotically-themed fan fiction, and have presented community members with a poorly-designed questionnaire (now taken down, but for a while being modified on the fly as people lined up with complaints about the research design – participants have posted screenshots and a text version of the survey after its initial modifications – note that a number of the final option responses and some other warnings and qualifications seem to have been added in response to criticisms of the survey in its original form – the modifications are often palpably different in style from the original text).

Among many other problems, the questionnaire asks respondents to provide sensitive information about sexual habits, desires and fantasies, in a setting where the questionnaire could be accessed by minors, without – as far as I can tell – having vetted the research design with their university’s IRB (the researchers are currently being hounded across several websites with demands to answer the question of whether they did, in fact, submit the project for ethics review – while answering other questions, they have steadfastly ignored this one: quick suggestion that, if the researchers don’t mean to imply the answer is ‘no’, then they should probably address this question very explicitly, very soon). [Side note: there's a nice critical discussion of the limitations of IRB's that's been sparked by this whole mess: here.]

In the ongoing discussions now sprawled across a number of sites, the authors continue to dig this initial hole deeper by using terms regarded as offensive by members of the community (and, in one case, defending this because these are the terms that are standard in the sex industry – as Marx might say: !!!), by blithely demonstrating their own participation in widely-criticised assumptions about sexuality and presuppositions about gender, by demonstrating ignorance of basic facts about the community that could be gleaned from a quick skim of community sites, and by insisting on knocking back well-reasoned and absolutely on-target critiques by arguing that they are not doing “social research” and are not actually interested in the community anyway, other than as an example of a much more general phenomenon (these last, the researchers seem to believe, get them off the hook on ethical and basic research design requirements).

I’m not going to write my own critique of this mess: the community has already done this, eloquently, thoroughly – and, given the circumstances, with admirable patience. I am always warning my students when I teach research methods that something like this can happen – that this is why I’m so harsh on their research designs. Welcome to my new case study. I’m serious. I’m thinking of assigning parts of this trainwreck when I teach research methods next term.

I’m posting on this mainly because I’m wondering why the researchers have not apologised far more abjectly for having blundered into a community so ill-prepared – and possibly having ignored basic legal requirements and professional ethical standards governing their research. I am wondering if they are simply failing to register how devastating are the critiques being made of their work – perhaps because they are assuming these critiques have arisen defensively, due to strong affective attachments and loyalties within this particular community – or perhaps because they have “othered” this community so much that they aren’t sufficiently open to how badly they are being schooled here. Sai Gaddam’s university website suggests a potential vulnerability in this regard – let me quote from the source (apologies: I owe a poster in the original discussion a hat-tip for drawing attention to this, but unfortunately I’ve lost track of the comment – if you want to make yourself known, I’ll add a link):

My research interests have evolved over the years I have spent in the Ph.D program, but my derision for my subjects remains a constant. Well, not really, but this quote does make me smile.

The individual I chose as my principal subject for the experiments … was an old toothless man, with a thin face, whose features, without being absolutely ugly, approached ordinary triviality, and whose facial expression was in perfect agreement with his inoffensive character and his restricted intelligence.

The Mechanism of Human Facial Expression — Guillaume de Boulogne

So, for what it’s worth: I don’t belong to this community, but the criticisms being made of your ill-conceived research are excellent. Listen to them. You have tried wearing the juice. They’ve seen through it. It wasn’t the disguise you hoped it might be.

Updates:

[Ed. 7 September: Still no time to update the broken links below, but wanted to point to the discussion at metafilter, for those interested. ETA: and Neuroanthropology weighs in! - Twice!]

[Ed. 4 September: If people aren't aware, Ben Goldacre from Bad Science has referenced SurveyFail on Twitter, linking here and also to Alison Macleod's fantastic overview at The Human Element. Rushing at the moment - apologies for not responding yet to comments.]

[Ed. 4 September: Another day, a few more broken links. Ogi Ogas and Sai Gaddam seem to have had their websites removed from Boston University - not surprising, given the report that they are not affiliated with the university for purposes of this research project. Gaddam's blog has also been made private. The links I have below off their names therefore no longer point anywhere. Again: my schedule's too hectic to fix this right now, so just noting the problem. Some limited information about Ogas is included in his Wikipedia page, as a backup link... If the old Boston University pages end up being included in any of the screencaps collections currently being collated online, I'll restore links to those once I have time.

Broken link clean-ups still days in the future, I'm afraid...]

[Ed. 3 Sept: Folks, just a note that the researchers have taken down their site - after an amazingly offensive final blowup that, honestly, must be seen to be believed... This will break a lot of the links I've posted below. I'll try to clean these up later, but for the time being, there's are a number of good summaries of the whole incident - now christened SurveyFail - see especially Yonmei's post at Feministsf.net, as well as a report of a response from the IRB at their university, which has disclaimed any affiliation with the project and asked the researchers not to use their uni emails or web addresses in conjunction with this activity. (My favorite part of the linked IRB discussion was the report that, when the IRB office was contacted directly: "Their exact words 'I had a feeling it would be about that.'") Links cleanup might have to wait a couple of days - schedule is awful at the moment...]

“…there is no reason to suppose that an inhabitant of Mars would see us more ‘objectively’ than we, for instance, see ourselves.” ~ Karl Popper

Popper, K. (1976 [1962]), “The Logic of the Social Sciences”, The Positivist Dispute in German Sociology, p. 92.

I’ll try to write something substantive again later in the week – at the moment, I’m absolutely drowning in marking, which leaves me no time to have interesting thoughts, let alone pull them together into something others might want to read… For my own reference as much as anything else, I’ve tucked below the fold a sort of “Research Proposals for Dummies” piece I wrote this week for my quant methods students. It’s very, very, very simplistic – among other things, because it’s written for second-year undergrads, many of whom have no intention of going on to research careers – but some of my Research Strategies students also found the material helpful as a very basic breakdown and explanation of the strategic intent of the sections of a proposal. The piece might be useful for someone needing similar material for their own students, and not wanting to start utterly from scratch, but wanting to riff off of someone else’s basic structure.

Note that, because this piece was written in relation to a specific assessment, much of the material is obviously not relevant to a standard proposal (and I’m too lazy and too busy – hmm… can one be both? Evidently so… – to rewrite this as a more general piece right now). Note also that I wrote this at 3 a.m. – caveat emptor.

If anyone does convert this into something less assessment-specific – or improve it in all the various other ways it needs to improved – I’d consider it a great kindness if you’d share a copy of your revised version with me.

I’m currently waiting to find out whether my research methodology empire will be extended this term, to cover a quantitative research methods course for second-year undergraduates – a teaching stint that would itself be regarded as preparation for assisting with a rethink of our second-year methodology course offerings, which are currently split between one term of “quant” and one term of “qual”. No one likes the split and yet, for various reasons (some programs only want their students to take one or the other course, and some programs are still running their own independent courses, etc.) thinking through whether and how to integrate these courses will be quite complex. While my responsibility for this course is still somewhat hypothetical, the beginning of the term is rapidly approaching, and I’ve begun half-preparing – mainly by soliciting ideas from folks who have taught into the course in the past, or who are interested in teaching into it this coming term.

One of the stories that seems to crop up in relation to past iterations of the course is the difficulty obtaining an interesting dataset on which students can practice the more statistical concepts covered in the course. Past iterations of the course appear generally to have given students some overarching policy problem – drug use in youth culture is a theme that has been mentioned often – and then set them loose on a dataset to test various hypotheses against the data, and to reflect on the policy implications of their results. Apparently, however, we have struggled to obtain relevant Australian data sufficiently robust for whatever exercises the students have been asked to perform. Instead – at least one year – we used a UK dataset, but were still asking students to reflect on Australian policy concerns.

When I heard this, I grimaced a bit, and said, “No – I’d really rather, if the point is to reflect on local problems, we use relevant local datasets. Otherwise it will confuse the students – and convey the wrong message, I think, about the need to look into these problems empirically – we don’t want to give the impression that just any old data will do…”

“Oh, no -” my interlocutor clarified, “the students didn’t know they were using UK data. We went in and edited the dataset – we changed all the names of British counties to the names of Victorian communities. It took forever! So, as far as the students were concerned, they were working with Australian data. They never knew.”

Now let me get this straight: We give students a term-long assessment task, oriented to get them to test their assumptions about an Australian policy issue (I’m not clear whether this was on the drug use topic, or on something else) – but we cook the data!!! Oh sure, the data are true for somewhere – and the same sorts of skills and reasoning would apply, regardless of the dataset – I do understand the reasoning behind the assessment task. But still… I have these images of students coming out of this course, getting into debates with friends and family years from now, and going, “Well, you know, I actually researched this issue at uni, and apparently the trend is…” What will the students do, when they run into conflicting empirical data at some later point? How will they make sense of it all?

Why not just tell students you’re using UK data? Or making the data up? Surely we don’t think our students are so fragile that this would cause them to disinvest completely from the task? Maybe it’s just me, but I find this absolutely riotous (and, on another level, want to go curl up in a small dark hole – but that’s what the new office is for…).