Archive for 'Qual Methods'

[Ed. 9 September: Now that events are unfolding a bit more slowly, and people have had a chance, for the most part, to learn about the basic facts, I've moved my on-the-fly updates to the bottom of the post, so that the original text is easier to find. I will try to update all the broken links next week.]

Original Post

A couple of people have sent me the link to this debacle of two researchers attempting to study what they call the “Cognitive Neuroscience of Fan Fiction” (further historical background here and here, collated links there, and information about the original research (which somehow doesn’t get around to mentioning that the research is designed – not for academic publication – but for a popular book whose working title is Rule 34: What Netporn Teaches Us About the Brain) in the researchers’ background information).

As someone looking on from outside the fan communities directly involved in this mess, the whole thing unfolds something like a live action version of the phenomenon Justin Kruger and David Dunning discuss in their “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments” (1999, Journal of Personality and Social Psychology, Vol. 77, No. 6., p. 1121-1134).

Kruger and Dunning are interested in whether, below a certain level basic competence, it becomes very difficult for people to improve their skills – because they are, in fact, too incompetent to be able to tell the difference between competence and incompetence in the first place. They take as their point of departure the story of hapless bank robber McArthur Wheeler who – some of you will remember from my previous post on this article – robbed two banks in broad daylight without any disguise and, when arrested almost immediately based on the bank security footage, burst out: “But I wore the juice!” Mr. Wheeler was evidently under the impression that, by rubbing lemon juice on his face, he could conceal himself from security cameras (Kruger and Dunning 1999: 1122).

Assuming this mess is not some sort of elaborate research-themed performance art, or the result of a revenge-fuelled identity theft, researchers Ogi Ogas and partner Sai Chaitanya Gaddam are trying their best to demonstrate to the world that they are something like the academic research equivalent to Wheeler. They have blundered into an online community whose members write and read, among other things, erotically-themed fan fiction, and have presented community members with a poorly-designed questionnaire (now taken down, but for a while being modified on the fly as people lined up with complaints about the research design – participants have posted screenshots and a text version of the survey after its initial modifications – note that a number of the final option responses and some other warnings and qualifications seem to have been added in response to criticisms of the survey in its original form – the modifications are often palpably different in style from the original text).

Among many other problems, the questionnaire asks respondents to provide sensitive information about sexual habits, desires and fantasies, in a setting where the questionnaire could be accessed by minors, without – as far as I can tell – having vetted the research design with their university’s IRB (the researchers are currently being hounded across several websites with demands to answer the question of whether they did, in fact, submit the project for ethics review – while answering other questions, they have steadfastly ignored this one: quick suggestion that, if the researchers don’t mean to imply the answer is ‘no’, then they should probably address this question very explicitly, very soon). [Side note: there's a nice critical discussion of the limitations of IRB's that's been sparked by this whole mess: here.]

In the ongoing discussions now sprawled across a number of sites, the authors continue to dig this initial hole deeper by using terms regarded as offensive by members of the community (and, in one case, defending this because these are the terms that are standard in the sex industry – as Marx might say: !!!), by blithely demonstrating their own participation in widely-criticised assumptions about sexuality and presuppositions about gender, by demonstrating ignorance of basic facts about the community that could be gleaned from a quick skim of community sites, and by insisting on knocking back well-reasoned and absolutely on-target critiques by arguing that they are not doing “social research” and are not actually interested in the community anyway, other than as an example of a much more general phenomenon (these last, the researchers seem to believe, get them off the hook on ethical and basic research design requirements).

I’m not going to write my own critique of this mess: the community has already done this, eloquently, thoroughly – and, given the circumstances, with admirable patience. I am always warning my students when I teach research methods that something like this can happen – that this is why I’m so harsh on their research designs. Welcome to my new case study. I’m serious. I’m thinking of assigning parts of this trainwreck when I teach research methods next term.

I’m posting on this mainly because I’m wondering why the researchers have not apologised far more abjectly for having blundered into a community so ill-prepared – and possibly having ignored basic legal requirements and professional ethical standards governing their research. I am wondering if they are simply failing to register how devastating are the critiques being made of their work – perhaps because they are assuming these critiques have arisen defensively, due to strong affective attachments and loyalties within this particular community – or perhaps because they have “othered” this community so much that they aren’t sufficiently open to how badly they are being schooled here. Sai Gaddam’s university website suggests a potential vulnerability in this regard – let me quote from the source (apologies: I owe a poster in the original discussion a hat-tip for drawing attention to this, but unfortunately I’ve lost track of the comment – if you want to make yourself known, I’ll add a link):

My research interests have evolved over the years I have spent in the Ph.D program, but my derision for my subjects remains a constant. Well, not really, but this quote does make me smile.

The individual I chose as my principal subject for the experiments … was an old toothless man, with a thin face, whose features, without being absolutely ugly, approached ordinary triviality, and whose facial expression was in perfect agreement with his inoffensive character and his restricted intelligence.

The Mechanism of Human Facial Expression — Guillaume de Boulogne

So, for what it’s worth: I don’t belong to this community, but the criticisms being made of your ill-conceived research are excellent. Listen to them. You have tried wearing the juice. They’ve seen through it. It wasn’t the disguise you hoped it might be.

Updates:

[Ed. 7 September: Still no time to update the broken links below, but wanted to point to the discussion at metafilter, for those interested. ETA: and Neuroanthropology weighs in! - Twice!]

[Ed. 4 September: If people aren't aware, Ben Goldacre from Bad Science has referenced SurveyFail on Twitter, linking here and also to Alison Macleod's fantastic overview at The Human Element. Rushing at the moment - apologies for not responding yet to comments.]

[Ed. 4 September: Another day, a few more broken links. Ogi Ogas and Sai Gaddam seem to have had their websites removed from Boston University - not surprising, given the report that they are not affiliated with the university for purposes of this research project. Gaddam's blog has also been made private. The links I have below off their names therefore no longer point anywhere. Again: my schedule's too hectic to fix this right now, so just noting the problem. Some limited information about Ogas is included in his Wikipedia page, as a backup link... If the old Boston University pages end up being included in any of the screencaps collections currently being collated online, I'll restore links to those once I have time.

Broken link clean-ups still days in the future, I'm afraid...]

[Ed. 3 Sept: Folks, just a note that the researchers have taken down their site - after an amazingly offensive final blowup that, honestly, must be seen to be believed... This will break a lot of the links I've posted below. I'll try to clean these up later, but for the time being, there's are a number of good summaries of the whole incident - now christened SurveyFail - see especially Yonmei's post at Feministsf.net, as well as a report of a response from the IRB at their university, which has disclaimed any affiliation with the project and asked the researchers not to use their uni emails or web addresses in conjunction with this activity. (My favorite part of the linked IRB discussion was the report that, when the IRB office was contacted directly: "Their exact words 'I had a feeling it would be about that.'") Links cleanup might have to wait a couple of days - schedule is awful at the moment...]

So today was the formal logic lecture in our newly designed social research course. In the spirit of the best pedagogical traditions we established in our quantitative methods course last year, my esteemed colleague L Magee set out to instil in our students the virtues of rigour and precision with a thorough discussion of the connections between logical operators, variable types, and research methods. For reasons that quite elude me, the normally intrepid LM seemed however to stumble when it came to explaining to our students the culminating point of one of his slides, which confidently informed:

Interval – constant intervals between values.

Consider temperature:

Arbitrary starting point

But degrees are constant and fixed units

Values are additive: 10 degrees + 10 degrees = 4 days

I’m not clear what the problem with this conclusion is meant to be? Why else were you recommending Lewis Carroll during this lecture, were it not to equip our students to parse conclusions such as this?

So it’s been a somewhat gruelling couple of weeks, getting everything together for the two methods courses I’m covering this term. The undergraduate course, since it’s new and we were designing it from scratch, took the lion’s share of the time – although it was nice to be able to work alongside the ultra-competent (and ubiquitously sardonic) L Magee in pulling everything together.

The course is large enough, and massive lecture halls scarce enough, that we have to deliver two different iterations of the lecture – so we had one go at the lecture Monday morning, and will get another try on Friday. Most of the first lecture, necessarily, dealt with housekeeping and course mechanics. I’m currently trying to gather my thoughts from the lecture on Monday – I always learn something about the tacit logic of my own stuff, when I present it – there were elements of the tacit logic underlying the structure of the lecture, elements of grasping why I wanted to organise my bits the way I did, that I sort of “got” only when listening to myself give the lecture… We’ll see if this improves the reiteration of the lecture to be delivered on Friday.

I thought I’d post a few notes here on the lecture and course concepts – with the caveat that I’m always a bit cringe-y when I expose pedagogical material publicly. There’s a strong exhortative dimension to teaching – things get simplified, not simply or straightforwardly to make them easier to learn, but with the goal of trying to rouse something, trying to pass along a certain contagion about why this stuff can actually be exciting (er… realising that what I’m about to write may not… er… have that effect on terribly many people – I don’t claim to be a rousing lecturer – quite the contrary (really strongly the contrary) – my skills lie much more in leading discussions – but there is still an element, in lecturing, of wanting to communicate affect, and not simply content, of wanting to share, somehow, that the very abstract sort of material that I generally teach, can be deeply meaningful in its strange way – and something about what I do, to try to communicate this, never seems to me to translate well when I write about what I did – instead, what comes through is the simplicity of the content, all the ways I would qualify it, all the ways I disagree with it… And yet… There’s a reason that stuff gets left out of the lecture in the first place… A reason that doesn’t prevent me from being fairly self-conscious about reproducing lecture concepts outside the shelter of the lecture hall…)

The course is titled “Social Research: Qualitative”, and the structure of classes here gives you twelve weeks to somehow meet whatever expectations such a title engenders. Last year, the staff member who took this course decided that twelve weeks simply wasn’t enough to give a meaningful introduction to something as broad as “social research” and so decided to drill closely down into one research method (discourse analysis), to try to give the students some in-depth experience with mastering a particular method, from which they could hopefully extrapolate when orienting themselves to other methods in the future. In earlier years, the course has been taught with a heavy NVivo focus, with all the students doing the same research project – again with the thought that students could extrapolate their experience with that project, into other sorts of research they might conduct in the future.

We’re equally dismayed, I suspect, by the jarring disjoint between the expansive course title, and what can reasonably be covered in twelve weeks in an introductory methods class for second-year undergraduates. Something – lots and lots of somethings – have to “go”, to make the course possible.

We’ve channelled our dismay, however, in a slightly different direction: while it is certainly true that any individual student is going to do some specific form of research project, and neglect others, we’ve decided not to pre-dictate either the method or the project itself. There will no doubt be plenty of project-deflection over the term, as students choose topics that are too vast – or illegal, dangerous, or inappropriate in other ways… ;-P But in principle we’ve left it to the students what they want to study, substantively, and how they intend to study it. Although we will do a bit with “method” in the “to do” list sense in this course, we’ve decided instead to focus on the most basic elements of the research design process – becoming curious about something, asking a question, looking around to see whether anyone else has ever asked something similar, trying to figure out what you need to do, to answer the question you’ve asked, and then being accountable for your question, what you’ve done to answer it, and the answer you’ve put forward, in a public sphere.

This approach means that we can’t dictate method, because we’re telling the students in a very strong way that their method has to derive in some quasi-logical way from their question. And we can’t dictate question because… well… we’re telling students that research is about straddling that strange space between personal curiosity and public accountability – and it’s a bit out of place to tell other people what they ought to be curious about… ;-P

So we bookended this first lecture with two videos, designed to mark out two possible extremes in conceptions of social research. After some brief transitional comments, we opened with the first six minutes of this video of the Milgram experiments:

What the students saw, was a man in a white lab coat take an authoritative role in a highly artificial experimental setting, where the stated purpose of the exercise was to test a hypothesis in carefully controlled circumstances. I did warn the students there was more to this experiment that met the eye (we’ll return to this video again later in the course) – but the parting image they were left with was of what looked to be a research subject with a heart condition, strapped to a chair, awaiting progressively nastier electric shocks if he failed in a memorisation task… (They laughed… Hmmm… I responded by telling them we would trial this method in their tutorials…)

So this is one extreme – not, in this case, for the distressing nature of the experiment, but for the highly artificial, controlled, hypothesis-testing orientation of the study. The video with which we ended the session was this one, on Sudhir Venkatesh’s anthropological work on a Chicago gang (the embedded video below is only an excerpt – the full program is here):

Venkatesh’s piece was chosen for a sort of maximal contrast to the fragment of the Milgram video that we showed: a research scenario in which the field strikes back, takes its researcher captive in the most literal possible sense, rejects the researcher’s “expert” knowledge, and tells the researcher how to conduct the (radically uncontrolled) study.

We will do other things with these and other video materials through the course but, for purposes of this introductory lecture, the point was simply to mark out two extreme points, suggest that there is a continuum of possibilities between them – and that all of this, the whole continuum, could be defended as some form of “social research”. A continuum of social research along which the students would have some opportunity to begin situating themselves in the course of the term.

In terms of other content, this blurb from the course guide gives the gist of how we are approaching the course:

Many people, when they think about research, think of something done in a special sort of place, like a laboratory, a library, or a “field site”, by a special sort of person, like an academic expert who has spent years acquiring a vast specialist knowledge of what they are studying, and on a special sort of topic, which is important enough to count as a “research question”. Thought of this way, research can seem a bit intimidating and removed from our other concerns: we can struggle to think of ourselves as being the kind of people who might do research – surely we aren’t qualified or we don’t “know enough”? We can struggle to imagine what research might look like, if carried out in the sorts of settings where we spend our personal and professional time – surely research doesn’t tackle the sorts of experiences we have in our everyday lives? We can doubt whether our questions and concerns are “important” enough to count as research questions – surely research investigates something more removed from our everyday experiences or personal passions?

While it is very common to think of research as this kind of specialised, rarefied expert activity, this image of research is highly misleading. Research, at its most basic, involves cultivating the very opposite of expertise: it entails a process of opening ourselves to what we don’t know – of taking seriously our own curiosity and desire to learn more – of asking questions. Because research crystallises around a question, the research process is driven precisely by our lack of expertise – by what we need to learn. In the research process, we all position ourselves as explorers and investigators, rather than as people who already possess some kind of mastery over our subject matter. Because research operates in this space of exploration and uncertainty – because it takes the form of quest to learn something new – it is impossible to have all the skills and knowledge you will need for the research process, before you undertake the research itself.

While some of us may have a bit more practical experience with research than others, all of us have some experience with the core skills required for the research process: we have all been curious, asked questions, set about finding answers, and debated with other people about each step in this process. On one level, then, we are all “researchers” in at least an informal sense. At the same time, no specific research project – formal or informal – begins with a special creature called a “researcher” who already possesses all the skills and knowledge required to carry out a research project, before they start asking questions and working out how to answer them. Researchers are created, not born. And what creates them is nothing more than the process of actually doing research. You become a researcher: you do this by carrying out research. All the skills that research requires, and all the things you need to know to do research successfully, are learned through the research process itself.

This doesn’t mean that formal study is not essential to the research process: it is. It means that this formal study more closely resembles the process of apprenticing in a craft, than it does the process of committing to memory some fixed body of information. Research is a practical activity – an art, albeit one undertaken with a scientific spirit. Every question, every method, every researcher brings something subtly different to the research process – meaning that research is never learned ​abstractly, as a skill that could be pursued separately from its various practical applications. Instead, your research question is what drives your formal study, providing a meaningful context within which you can work out what sorts of formal knowledge and skills you need to have, why you need to have them, and how you can learn them most efficiently. Your research question therefore grounds other sorts of study you undertake – which is why we will start this course, not with an abstract set of knowledge or skills we think you need to memorise, but with activities that will help you work out a research question that can organise the rest of your work in this course.

From this starting point, we will then guide you as you undertake a quick apprenticeship in the major stages of the research process. There are of course many different types of research. The research carried out by a journalist, an activist, a market researcher, a government, or an academic researcher will differ in significant ways, for example, due to the different end goals and audiences for the research. Nevertheless, certain elements are common to any sort of research process. Those common elements will provide the focus of our work in this course.

So it’s looking as though L Magee and I (with the assistance of others who just might be lurking, and who are welcome to out themselves if they so choose ;-P) will be responsible for a second-year undergraduate social science methods course, with an emphasis on qualitative research. This is the sister course to the quantitative methods sequence we took on a couple of years ago. This time around, we have considerably more time to design and prepare for the course (a colleague laughed when I said this, but in relative terms, this is true… I tend to get… rather little notice of what I will be teaching…). My hope is that this wealth of preparatory time might help us avoid the occasional difficulties we experienced last time around, particularly in coming up with concrete examples that are… pedadgogically appropriate…

Which brings me to a bleg. If anyone has favourite websites, readings, syllabi and other materials they would like to share, for an introductory social science qualitative methods course: love to see them. More specifically, I am particularly interested in developing very bounded, small-scale projects or activities that will help students obtain a hands-on feel for particular methodologies: the tentative course design concept is that, over the term, students will design and carry out one primary research project – but that, before they do this, they will first do a series of micro projects/activities that will give them at least a gestural sense of a range of research methods.

If anyone has ideas that have worked well for them in the past, materials, readings – would be much appreciated if you could share…