The Use of Checklists in Research

Kaitlin Gallagher is a PhD Candidate specializing in Biomechanics at the Department of Kinesiology at the University of Waterloo in Canada, and a permanent author for GradHacker. You can follow her on twitter at @KtlnG.

We may not like to admit it, but many of us can describe a time when we’ve made a mistake during the progress of a study. These mistakes can range from mixing up wires or forgetting to turn on an amplifier to forgetting to collect an essential piece of information that either requires additional processing time or prevents you from analyzing a certain variable altogether. Increased computing power and technological advancements have also made it easier than ever to collect data. We can collect five measures simultaneously in one study and hundreds of trials in no time at all. But where does this leave us now? We must set up all of this equipment and make sure it works together, monitor it as well as our participant or specimen, and somehow sift through all the data post hoc. Even with a detailed lab notebook, its no wonder problems can arise. Even just writing this makes me feel…exposed, as if I’m the only one who struggles with this. It seems so simple, how can I not get it perfect every time? I always thought that I just had to work harder to not miss small steps, but maybe I just needed a different, yet structured, perspective on how to manage such a high volume of complex information.

My interest in general checklists above and beyond the detailed lab notebook began after reading The Checklist Manifesto by Atul Gawande, a surgeon and Harvard Professor (he also is the author of a New Yorker column on the same subject). The purpose of this book is to describe how a basic checklist can help us perform complex tasks consistently, correctly, and safely. Much of the book is told from the point of view of eliminating errors during surgery, but Gawande also draws on stories on how checklists have benefited those in construction, aviation, and investing.

Gawande explains that we are up against two things when performing either a high volume of simple tasks or a variety of complex tasks. The first is that human memory and attention can fail you, especially when a bigger issue arises. This could be your participant being late and your data collection program freezing, making it easy to forget that you haven’t performed a baseline test. The second thing is that we skip tasks even when we remember them because nine times out of ten that step doesn’t matter. Never check to make sure your wires are plugged in correctly?  If you’re the only one working in the lab maybe it doesn’t matter, but if multiple lab mates are cycling through the lab, this could be a bigger issue.

This week I made my first checklist for setting up one of my thesis data collections. I listed specific essential tasks and supplemented them with common errors I had either made or had encountered in the past. After making this specific checklist, I decided to see if I could make a general list that could be applied to all studies. Surprisingly, it was easier to do than I thought, although I’m sure it isn’t perfect. I was able to group many of my tasks together under one common point. What is not easy so far is trusting and not deviating from the checklist. It’s been easy to throw the checklist to the side when I get frustrated. In more stressful situations or even when things are running smoothly, I may forget that I’ve come up with a structured way to make sure I’m managing my data collection in the best way possible.

I started to brain storm other areas that checklists could be beneficial. In the construction industry, Gawande explains that checklists are used so that key points are discussed between those in different aspects of the building process. For research, are there things that you always need to talk about with other experimenters or your supervisor when it comes to a study? Maybe a checklist can help there too. Also, when editing manuscripts or proofs, you could have a structured set of points to assess such as, “check to make sure data in tables/figures is correct” or “make sure reference list is up to date”. These all seem so basic, but if taking the time to go over them and know that once you’ve handed in the manuscript that these things have definitely been checked, it could prevent you from having to submit an erratum due to something like an improper figure.

That brings us to the last point about checklists - they DO NOT replace knowledge. An investor interviewed for the book said it best when describing that the checklist is “not a fail safe thing…you still need expertise and insight into the process to be able to ultimately perform each step correctly”. These checklists wouldn’t help me if I didn’t know what I was doing to begin with. Rather than being a “Step by Step to Collecting Data”, people can perform a task however they want and the checklist makes sure that in the end that task was performed correctly.

I’m going to continue making checklists for various parts of the research process and see how things go. In a future post, I hope to update you on my use of checklists and whether I feel they’ve helped or hindered the process for me.

What are your thoughts on data collection errors and how to resolve them? Leave your thought in the comments below.