Introducing 'Rethinking Research'

How do we know which scientific results to trust? Research published in peer-reviewed academic journals has typically been considered the gold standard, having been subjected to in-depth scrutiny -- or so we once thought. In recent years, our faith in peer-reviewed research has been shaken by the revelation that many published findings don’t hold up when scholars try to reproduce them. The question of which science to trust no longer seems straightforward.

Concerns about scientific validity and reproducibility have been on the rise since John Ioannidis, a professor at Stanford School of Medicine, published his 2005 article “Why Most Published Research Findings are False.” Ioannidis pointed to several sources of bias in research, including the pressure to publish positive findings, small sample sizes and selective reporting of results.

In the years since, a wave of scholars have dug deeper into these issues across a number of disciplines. Brian Nosek at the Center for Open Science and Elizabeth Iorns of Science Exchange spearheaded attempts to repeat past studies in their respective fields, psychology and cancer biology, with discouraging results. Economists encountered trouble in merely repeating the analyses reported in papers using the original data and code.

By 2016, when Nature surveyed 1,500 scholars, over half expressed the view that there is a significant “reproducibility crisis” in science. This crisis comes at an uncomfortable time, when some skeptical voices question even well-grounded scientific claims such as the effectiveness of vaccines and humans’ role in climate change.

Given this hostility, there’s a concern that reproducibility issues may undermine public confidence in science or lead to diminished funding for research. What is clear is that we need a more nuanced message than “science works” or “science fails.” Scientific progress is real, but can be hindered by shortcomings that diminish our confidence in some results, and need to be addressed.

There has been plenty of coverage about the reproducibility crisis and its implications (including debate over whether to call it a crisis) in both scientific publications and mainstream outlets like The New York Times, The Atlantic, Slate and FiveThirtyEight. But somewhat less attention has been paid to the question of how to move forward. To help chip away at this question, we’re publishing a series of articles from researchers leading initiatives to improve how academics are trained, how data are shared and reviewed, and how universities shape incentives for better research.

Read the rest of this essay here.