Debunking Bad COVID-19 Research

To understand and prevent the spread of the COVID-19 pandemic, researchers are working at a rapid clip.

As funders scale COVID-19 research grants and expedite application processes, publishers too are trying to move quickly to ensure that academics, policy makers and the public can access the latest research developments in a timely fashion.

This rush to disseminate information is exposing cracks in the scholarly research system. Academic journals have not been fast-moving historically, and traditional peer review can take months. To make research findings available quickly, many researchers are publishing versions of papers that have not yet been peer reviewed on preprint servers such as arXiv, bioRxiv and SSRN.

Preprint servers play an increasingly important role in the scholarly publishing landscape. They are a popular platform for researchers to get early feedback on their research. They are also a space where researchers can publish research products and data sets not typically published in traditional journals. The process is fast -- publication of open-access research that anyone can read is immediate.

The downside of this open publication system is that sometimes controversial or poor-quality research can garner a lot of attention on social media or in news articles, said Stefano Bertozzi, professor of health policy and management at the University of California, Berkeley, School of Public Health. In the clamor for information about COVID-19, it is easy for misinformation to spread online, he said.

To combat this, MIT Press and the Berkeley School of Public Health are launching a new COVID-19 journal, one that will peer review preprint articles getting a lot of attention -- elevating the good research and debunking the bad.

The Rapid Reviews: COVID-19 journal will be led by Bertozzi, who will serve as the first editor in chief. Unlike a traditional journal, authors will not submit their work for review. Instead, the Rapid Reviews team will select and review already-published preprint articles -- a publishing model known as an overlay journal.   

Article authors will be notified if their research is selected for review, but the Rapid Reviews team will not wait for their permission to publish public reviews of their work. These reviews will be published as stand-alone articles using open-source publishing platform PubPub. The authors of research papers deemed to be high-quality will be invited to publish their work with Rapid Reviews in a more conventional journal format.   

The work of screening potentially thousands of preprint articles is not going to be easy, acknowledges Bertozzi. He plans to collaborate with researchers at the University of Washington and other institutions that have already developed processes to sift through the vast volume of pandemic research published every day. A network of student screeners from UC Berkeley and other institutions will assist in this process. Artificial intelligence will also be used to identify potentially noteworthy articles.  

There are already more than 5,800 articles relating to the pandemic on bioRxiv and medRxiv, said John Inglis, who co-founded the preprint servers. “I do think it necessary to peer review at least some of the many thousands of pandemic-related preprints that have appeared,” he said in an email.

“I applaud this initiative by MIT Press. It is one of several efforts the research community has mounted to provide rapid review or commentary on COVID-19 preprints,” said Inglis, who is also executive director and publisher of the Cold Spring Harbor Laboratory Press. “It’s important that reviews are done quickly, also by experts, since we’re in a public health crisis where new and reliable information is vital to the management of patients and communities more broadly,” he said.  

The academic scope of the Rapid Reviews journal is broad and not confined to biomedical research, said Bertozzi. The speed of the review process will depend on the availability of subject matter experts. Bertozzi hopes for quick turnarounds -- publication of reviews within a week or two of Rapid Reviews selecting the articles would be the ideal, he said.

Funding for the journal is being provided by the Patrick J. McGovern Foundation, which has supplied a $350,000 grant. Vilas Dhar, a trustee of the foundation, described the Rapid Reviews initiative as a “breakthrough in academic publishing, bringing together urgency and scientific rigor so the world’s researchers can rapidly disseminate new discoveries that we can trust.”

“We are confident the RR:C19 journal will quickly become an invaluable resource for researchers, public health officials, and healthcare providers on the frontline of this pandemic,” said Dhar in a news release. “We’re also excited about the potential for a long-term transformation in how we evaluate and share research across all scientific disciplines.”

The formation of the journal, which will publish its first reviews in early July, was incredibly fast, said Nick Lindsay, director of journals and open access at MIT Press. Conversations about the need for peer review of COVID-19 preprints began in March, and support has been substantial, he said.

“We want to ensure that clinicians and researchers have trusted information they need to make crucial decisions. We also want to ensure we don’t have any more misinformation seeping out into the mainstream and being consumed by the public in a way that it really shouldn’t,” said Lindsay.  

Open postpublication peer review is a good solution to help rapidly correct errors and improve research, said Jessica Polka, executive director of ASAPbio, a nonprofit promoting transparent and innovative science communication. Polka noted that a group of leading open-access publishers recently called for traditional peer-review practices to change in order for important COVID-19 research to be reviewed and openly published as quickly as possible.

There are several overlay journals that review preprint articles, said Daniel Himmelstein, a biodata scientist at the University of Pennsylvania. In the past he has reviewed articles for initiatives such as biOverlay and published links to his reviews in the comment section of bioRxiv. “I like reviewing things that are already online because I can immediately post my review and engage in a conversation with the authors,” he said. 

“It’s great to see the first steps being taken to develop an ecosystem where authors can publish their research openly and people can post their feedback publicly,” said Himmelstein. “I think ultimately this will be much better than our current peer review system which is often done in a private way where the public never gets to see the assessment.”

It is possible that the reviews conducted by Rapid Reviews may be considered good enough to replace the reviews conducted by traditional journals, said Bertozzi. “It will be interesting to see the degree to which journals will use our reviews to accelerate their publication processes, that would be another real impact we could have,” he said.  

For traditional publishers, speeding up prepublication peer-review processes is challenging. In recent months, more than 20 research papers on COVID-19 published in peer-reviewed journals have been taken down from public view, according to science publishing watchdog blog Retraction Watch. The reasons behind these retraction decisions are not always explained. Most of the time publishers retract articles following scrutiny and criticism of research methods and data. Sometimes authors themselves request retractions of their published works.

Two controversial COVID-19 research papers were retracted from high-profile, well-regarded journals earlier this month. Both The New England Journal of Medicine and The Lancet retracted studies after questions were raised about the reliability of a company-owned private database of electronic health records used as underlying data in both papers. These retractions illustrate that even the most well-resourced journals can get peer review wrong, Bertozzi said.

“These are very prestigious journals with egg on their face for an inadequate peer review process,” Bertozzi said. When you’re rushing, mistakes happen, he said. “I’m sure we’ll have egg on our face, too, before too long.”

Emphasizing speed in the peer-review process can lead to errors in judgement, but Bertozzi hopes that by encouraging reviewers to attach their names to publicly published reviews, transparency and accountability will promote thoughtfulness and care. Bertozzi hopes that each preprint article selected for review will be reviewed by at least two experts in the field. The reviews will themselves be reviewed to ensure they meet certain quality benchmarks, Bertozzi said.

Most scholarly journals have a system of prepublication peer review that is slow-moving and offers limited incentives to researchers asked to be reviewers. Reviewers are not typically paid, nor are their names typically attached to reviews. The peer-review process typically happens behind the scenes, meaning reviewers are not recognized publicly for their work. The Rapid Reviews team members hope that their journal can demonstrate a new model for how to handle research during a crisis.

“I like the idea and think it could be extremely useful,” said Paul Ginsparg, professor of physics and information science at Cornell University and founder of the arXiv preprint server. The initiative is likely to face challenges, however, he said. Fast turnarounds by reviewers are “highly desirable but not clearly sustainable,” he said.

“The most expert reviewers are the people working in the area, who are also incredibly busy, unsurprisingly,” said Ginsparg. “Depending on the number of reviews, they’d burn out fairly quickly, and if the reviewer pool grows too large, then the expertise is diluted.”

For the Rapid Reviews reviews to gain visibility, it would be beneficial for preprint articles to include links to published reviews. Early discussions with staff at leading preprint servers about this have been positive, said Lindsay. But not all servers have the capability at this point to include links to Rapid Reviews reviews beneath published articles.

If the servers want to publish links to reviews under preprint articles, it would be technically straightforward, said Ginsparg, who coined the term "overlay journal" in the mid-'90s.

“The issue, of course, is whether more of these overlays will arise, and then preprint servers will have to make decisions about which to accommodate,” he said.

“Another issue is author buy-in,” Ginsparg said. “Some might not be thrilled to have a negative review posted prominently, and it might be necessary to warn them at the point of deposition that a review might be posted.”

Ginsparg asked whether authors might be able to post a rebuttal, and if they republish the preprint with revisions, whether the article will be re-reviewed. Authors will be able to respond to reviews, said Bertozzi in response. If authors decide to publish a revised preprint, it is possible the research may be reviewed again. The Rapid Reviews team will work with preprint servers to ensure that different versions of preprints and corresponding reviews are clearly labelled and discoverable.

Trying to manage different versions of articles and reviews could get complicated, said Ginsparg. “I can think of a number of problematic scenarios, all ultimately requiring additional human labor,” he said. “This is fine in principle but costs money at both ends. These things can work with volunteer labor for a while, but in the long run, there’s no free lunch.”