Schooling for Scandal

Ever since the news broke about Science pulling a political science paper after graduate students called attention to problems with its methodology and one of its authors asked for it to be retracted, I’ve been mulling over how to talk about this episode with students. It says something about how peer review works, but doesn’t always. It suggests things critical readers should look out for when skimming a methods section. It illustrates how even highly reputable publications (perhaps especially those with the greatest prestige) can be seduced by a startling finding that will attract attention. It raises questions about the importance of replication and shows that students can shake things up by raising questions about a study that was not only published in a major journal but was reported in lots of news outlets – even This American Life gave it airtime. But here’s the problem I run into: how can you raise these issues without making students completely cynical?

Too often we err in the other direction. Telling students to limit their search to peer-reviewed articles and giving them a checklist so that they can identify what a peer-reviewed article looks like oversimplifies the process of evaluation and suggests that all articles dressed that way are equally good or at least “safe” to use.  The often-used “CRAAP Test” (a checklist that asks students to evaluate currency, relevance, authority, accuracy, and purpose of a text) doesn’t always help because it’s not easy for non-experts to assess authors’ credentials or evaluate the accuracy of an article on a topic that’s fairly new to them. Years ago Marc Meola urged us to “chuck the checklist” and instead engage students in comparing sources and corroborating them in context, advice that's sound even with beginners. With more advanced students, I want to get deeper into the culture of scholarship and talk about where research comes from and why sometimes important findings are hidden and why the most prestigious of journals often publish research that is flawed. The trouble is, I don't want to undermine their confidence in the scholarly record itself. With all its flaws, it’s still a pretty robust system for sharing ideas that can help us understand the world better.

One thought I’ve had is to have students trace the history of a retraction such as this one (or the notorious vaccine paper, the influential economics paper on austerity that had flaws found by graduate students, or the Facebook study that unexpectedly creeped the public out in a big way and raised questions about informed consent). Find the original paper, see what news outlets covered it and whether they reported the research accurately, find out how the study was challenged and on what basis, and do an analysis of what factors played into the controversy. Then have them interview a faculty member about their experiences publishing research (something I already do pretty regularly) so that they get a perspective on how it normally goes. I hope that by learning how someone they know experiences the peer review process and digging into what happens when it goes wrong, students will be able to see some of the complexity of the culture surrounding scholarly work and where the failure points may be - without leaving them in a crisis of faith. 

This is obviously not something you can do in a one-hour “here’s how the library works” session, but it might work in a semester-long course I teach or with classes that spend several sessions or a long lab period in the library.  Has anyone else tried this? Any words of advice?