I’ve been watching with some amusement the evolution of Scott Eric Kaufman’s meme experiment, which I mentioned here the other day (anyone who hasn’t yet linked back will… I don’t know… have some horrible chain-lettery thing happen to you very soon…). For those following from the sidelines, Scott has posted an update on the quantitative success of his experiment. Personally, I’ve been following the unintended qualitative dimensions of the project – specifically, the number of people who’ve evidently decided to help out because, well, they’re so damned irritated about how badly they think the whole thing has been designed.
Sarapen opened with a thoughtful and even-tempered critique (including a literature review, even), but not all participants were as kind. Scrolling through the comments on Scott’s blog, you see a very large number of methodology criticisms, pointing Scott to things he hasn’t controlled for, noting problems in capturing the relevant data, criticising what he believes the data will show, complaining that he hasn’t sufficiently defined his hypothesis prior to the experiment, accusing him of stacking the experimental deck, arguing that the experiment can’t possibly be expected to follow the course of “wild” memes, contesting the finer points of whether “meme” was the right word for this – and, my personal favourite:
Based on my experience with technorati, when they pick up a link can be highly variable and not well-correlated with the actual time that link is created (to the point of being off by days). Your methodology is already crap just on technical grounds, even before taking into account all the objections above. Try using a web bug or something like it next time.
I suggested to Scott that he put all of this criticism to productive use at the MLA conference – prove the value of internet academic discussion, by challenging his panel audience to see whether they can come up with as many reasons that his methodology is “crap”. ;-P Since making this comment, though, I’ve begun to wonder: perhaps we’re looking at the birth of a new kind of PhD student performance art – the methodology slam. Someone stands up on some obscure corner of the net, calls out their research methodology, and asks a friend to tell a friend… Perhaps the results can be submitted as part of the portfolio to the committee approving candidature – a new criterion before you can call yourself ABD…
Rough Theory 
Interesting metapost. Perhaps part of the reason for the hostility toward Scott’s project (including mine) is a sense of discipline-poaching—he is basically a humanities person groping for the science-associated prestige of “hard numbers” without actually learning from disciplines that have well-developed methodolgies for how to obtain and work with “hard numbers.” The problem is that his numbers are actually really “soft,” padded by mushy disclaimers and pleas of time and resource limitation, etc. I am not against inter-disciplinary endeavors per se, but I do think it is problematic to undertake them without first understanding what one needs to learn from disciplines in which one is not a specialist.
Actually, I applaud your suggestion that a methodological discussion is very much in order here. But insofar as the average MLA audience member is probably not from the hard sciences, social sciences, or philosophy, I’m not sure that would be quite the right venue to really get to the heart of the problem. As you suggest here, ironically but I’m going to take you at your word, the internet methodology slam should indeed be an online-genre and–why not–expose ivory-tower bound humanities people to the rigor of a much broader review process. Maybe it would help!
Sorry your post was initially held up – sometimes my spam filter is a bit overzealous.
I actually do take quite seriously the potential for interdisciplinary feedback that can arise from exposing work online. What amused me about this incident was that I suspect that, if Scott had just posted something titled “MLA Blogging Paper: Methodology Proposal”, and had asked for feedback, he might have gotten a few stray comments from the regulars at his blog. Instead, he tries to do something else entirely, and the methodology critiques come pouring in.
As it is, having read bits and pieces about the background for this paper over the past couple of months, my guess would be that the methodological issues with Scott’s experiment are probably not that pivotal for the way he intends to use his data. Obviously Scott can speak for himself, and I’m not across this project in any detail, but my impression was that the claim related more to the hierarchical nature of information transfer across the net – that the net is not as much of a horizontal free-for-all as it’s sometimes presented as being in popular discussion. This is a point that can be established in a variety of ways – I’m sure Scott must be familiar with this – and my sense was that he was just looking for an offhand illustration that would be entertaining to discuss during his presentation. But perhaps I have the wrong idea…
None of this, though, detracts from the basic concept that this kind of online exposure could be enormously useful for interdisciplinary critique – if we can figure out how to get one another to pay attention…
Pingback: Crooked Timber » » Son of “The MLA Meme.”