‘You Must Retract This Paper’

A  new paper in Nature Communications concludes that informal female mentorship in academic collaborations is, by certain measures, bad for scientists. The journal is now reviewing how and why the paper got published. 

Some have accused the paper’s many critics of being afraid of uncomfortable findings. Those critics find that allegation risible, pointing to what they describe as serious methodological and analytical problems within the paper itself -- many of which reviewers flagged prior to publication.

“Let me blunt: For the good of the global scientific community and for the reputation of Nature Communications, you must retract this paper,” Leslie B. Vosshall, investigator at Howard Hughes Medical Institute and Robin Chemers Neustein Professor at Rockefeller University, wrote to the journal last week. “The general consensus among hundreds of colleagues who have read and commented on this paper in large group email threads and on Twitter is that it is deeply methodologically flawed, and with the potential to inflict serious harm on the global scientific community.”

Over all, Vosshall said, “I find it deeply discouraging that this message -- avoid a female mentor or your career will suffer -- is being amplified by your journal.”

In response to Vosshall’s message and others like it, the journal quickly added an editor’s note to the paper saying, “Readers are alerted that this paper is subject to criticisms that are being considered by the editors. Those criticisms were targeted to the authors’ interpretation of their data that gender plays a role in the success of mentoring relationships between junior and senior researchers, in a way that undermines the role of female mentors and mentees.”

Nature Communications is now “investigating the concerns raised and an editorial response will follow the resolution of these issues.”

On social media, the journal thanked “those who've contacted us about this issue,” adding that it “strongly believes in & supports equality and diversity in research.”

The paper’s authors say they welcome the journal’s review. They suggest that some of their conclusions have been misinterpreted.

Bedoor AlShebli, assistant professor of computational social science at New York University’s Abu Dhabi campus and lead author of the paper, said via email that the study, supplementary materials and public review document already address some of the questions raised.

She forwarded a statement she co-authored with her campus colleagues, Kinga Makovi and Talal Rahwan, saying, “We highlight that the elevation of women in science depends on the achievement of at least two objectives: retaining women in scientific careers -- for which female mentors are indispensable, as explicitly mentioned in our paper -- and maximizing women’s long-term impact in the academy.”

Quoting their paper, AlShebli and her co-authors said that “the goal of gender equity in science, regardless of the objective targeted, cannot, and should not be shouldered by senior female scientists alone, rather, it should be embraced by the scientific community as a whole.”

As for the journal’s investigation, the authors “believe that free inquiry and debate are engines of science, and welcome the review,” which “will lead to a thorough and rigorous discussion of the work and its complex implications.”

What the Study Says

For their paper, AlShebli and her colleagues studied a variety of dynamics within three million mentor-mentee research pairings, including the “possibility that opposite-gender mentorship may actually increase the impact of women who pursue a scientific career.”

Working with the Microsoft Academic Graph data set, which includes information about citation networks, they relied on a computer program to recognize scientists’ gender by their first names. They then distinguished between junior and senior scientists based on their “academic age,” as measured by years since first publication: in the first seven years, they’re considered junior scientists. After that, they’re considered seniors.

Whenever a junior scientist published a paper with a senior scientist, the authors considered him or her to be a “protégé,” and the latter to be a mentor, as long as they co-authored at least one paper with 20 or fewer authors total and shared the same discipline and U.S.-based institutional affiliation. The study spanned the disciplines of biology, chemistry, computer science, economics, engineering, geology, materials science, medicine, physics and psychology, across 100 years of research.

To support their definition of mentorship -- that it isn’t just co-authorship -- AlShebli and her collaborators included survey results from 167 respondents to a survey they originally sent to a random sample of 2,000 scientists in the data set. These respondents generally agreed with that premise.

Looking at measures of mentorship quality across the sample, the authors isolated the “impact” of the mentors prior to mentorship, or the average number of yearly citations. This, they say, shows how much of a “big shot” the mentors are.

The second measure of mentorship quality considered is the average “degree” of the mentors prior to mentorship, where the degree of each mentor is calculated in the network of scientific collaborations up to the year of their first publication with the protégé. The authors call this the “hub” experience, as in how much of a “hub” each mentor is in the collaboration network.

AlShebli and her colleagues measured the mentorship “outcome” for the protégé in a similar way, during their senior years without their mentors.

The study found that an increase in “big-shot experience” with a high-impact mentor is significantly associated with an increase in the postmentorship impact of protégés by up to 35 percent. The hub experience is associated with an increase in the postmentorship impact of protégés by up to 13 percent.

Focusing much of their remaining attention on “big-shot” mentors, AlShebli and her colleagues found that having more female mentors is associated with a decrease in outcomes, of up to 35 percent. This was especially true for female protégées.

Looking at how mentorship affected outcomes for the mentors themselves, the researchers also found that by mentoring female instead of male protégés, “female mentors compromise their gain from mentorship, and suffer on average a loss of 18 percent in citations on their mentored papers.” Male mentors don’t appear to be significantly affected, either way.

The “why” behind these findings lies beyond the scope of the study and merits further research, the paper says. But it challenges past research finding that female mentors improve career outcomes for female mentees, saying that such research compares having female mentors to having no mentors at all -- instead of comparing outcomes of having male or female mentors.

Indeed, AlShebli and her colleagues wrote that their study “suggests that female protégés who remain in academia reap more benefits when mentored by males rather than equally-impactful females.”

The authors vaguely address sexism as a possible driver of their findings, saying that “historically, male scientists had enjoyed more privileges and access to resources than their female counterparts, and thus were able to provide more support to their protégés.” But they say it’s possible that female mentors now are “serving on more committees, thereby reducing the time they are able to invest in their protégés,” or “taking on less recognized topics that their protégés emulate.”

Moreover, the authors say, “Our findings also suggest that mentors benefit more when working with male protégés rather than working with comparable female protégés, especially if the mentor is female.”

‘Diminishing’ Female Mentors

Negative responses to the paper appeared soon after it was uploaded, including one on SkepChick saying that AlShebli and her collaborators had two choices: “simply present the data” or “wrap the data in a disgusting agenda aimed at diminishing and belittling the contributions of female mentors.”

The writer of the SkepChick piece -- a professor of science who publishes under the name Isis the Scientist, and who declined to share her full name and institutional affiliation for this story, to keep her job separate -- said that AlShebli and her colleagues chose the latter. What tipped her off to the paper’s real agenda? She quoted this part of the abstract: “While current diversity policies encourage same-gender mentorships to retain women in academia, our findings raise the possibility that opposite-gender mentorship may actually increase the impact of women who pursue a scientific career. These findings add a new perspective to the policy debate on how to best elevate the status of women in science.”

No, Isis wrote, “These findings do not add a new perspective.” Instead, “They reinforce a perspective that so many of us have fought our entire careers to challenge -- that the best path to success is by working with a big name, white male faculty member, despite the well-recognized impact it has on retaining women in science.”

Undergirding this perspective is “mediocre science, which is always the best sort of science for reinforcing discriminatory agendas,” Isis wrote. Among the problems with the science itself, is the gender identification program used for scientists, which failed to correctly identify 21 percent of the 73 names she tried on her own.

“This is a huge potential source of error and important when we start thinking effect sizes which are <10%,” she wrote. “Further, I’d argue that any study of gender that does not consider non-binary gender, trans-gender, etc. is bullshit en face.”

Like many of the critics of the paper, Isis also criticized the paper’s definition of “mentor,” saying that the survey of 167 scientists on mentorship represented not even 1 percent percent of the data set, and that only about half of respondents said they received any substantial mentoring from senior authors on their papers.

Citing a common critique of citation counts alone as a measure of scientific impact, Isis said that to equate that paper’s definition of impact with “‘quality of the scholar’ is laughable. It reflects only the thing that defines it -- the number of times a paper is cited across a 5 year period.”

Joshua Miller, a postdoctoral associate and conservation genomicist at the University of Alberta in Canada, raised similar points in a separate critique of the paper on Twitter. Miller said Saturday that his original concerns were about interpretations and conclusions about the data -- specifically the idea that the lower citation counts in some of the gendered pairings means that female mentees should seek male mentors, and female-female mentorship pairs should be discouraged.

Calling the notion “absurd,” Miller said the results instead “point to systematic barriers and implicit biases faced by female researchers that can and should be addressed.” Beyond that, he said, the paper's definition of mentorship is essentially co-authorship and its definition of mentees' success -- number of citations on papers not authored with the mentor -- is “very narrow.”

Regarding how the researchers determined gender, Miller, said the authors admitted they couldn't determine a gender for 48 percent of the first names tested on Genderize.io, as these names were unclassified or unknown. That method also "erases trans and nonbinary people,” Miller added.

On gender, Isis said it’s “not a secret that men are cited more than women. It’s not a secret that there are factors that keep women from publishing (cough, COVID, cough). The quality of a scholar branches beyond a 5-year publication period. Teaching, grants, talks, translation to practice, and impact on people.”

Isis “would never have stayed in science were it not for my female mentors who demonstrated a quality and creativity of scholarship that extended beyond this lone index,” she continued. “To distill quality down to a single metric and diminish this important role of women mentors feels like a knife in the heart.”

Carolyn R. Bertozzi, Anne T. and Robert M. Bass Professor of Chemistry and professor of chemical and systems biology and radiology at Stanford University, who also publicly criticized the paper, said via email that “this ‘big authorship data’ analysis stumbles on what we already know about gender bias in academia, then stupidly advises that the key to women’s success is to avoid female mentors and instead align with privileged males.”

Lara Mahal, Canada Excellence Research Chair in Glycomics at Alberta, told Inside Higher Ed that when she was starting out as a graduate student at the University of California, Berkeley, a faculty member advising her said that she shouldn’t join Bertozzi’s lab there because Mahal had worked for another female professor as an undergraduate -- Rebecca Braslau at California’s Santa Cruz campus -- and that it “wouldn't be good for me to work for two women in a row.”

Mahal said she ignored the “quote-unquote advice” but never forgot it.

She became Bertozzi’s first Ph.D., having completed work for their first paper for Science by the end of her first year in the program.

“Up until recently, I had hoped that this kind of thing was getting better,” she said. But between a recently deleted Angewandte Chemie paper purporting to show that diversity negatively affects the field of chemistry -- over which many international advisory board members resigned -- and this new paper, she said, it appears “these attitudes have never really gone away.”

Some of the other problems with the paper, according to Isis from SkepChick and others: there is no study control for the fact that women are less likely to be full professors or overburdened (not necessarily by choice) by service obligations, and more.

In essence, Isis wrote, “Other studies of women mentors have truly quantified mentorship. This is not what their study shows. The study shows that publishing with a man is beneficial and that the least citations arise when two women publish together as end authors. No f-cking shit.”

Backlash to the Backlash -- and Back Again

These critiques aside, some scientists find the response to the paper chilling to controversial research. One wrote on Twitter, for instance, that “We are allowing Twitter mobs to force journals to review already peer-reviewed and accepted scientific papers because they hate the results.” This will “inevitably lead to the complete distrust of our scientific institutions,” he said.

Tania A. Reynolds, an assistant professor of psychology at the University of New Mexico, wrote on Twitter that “Many people are calling to have this paper retracted, but these findings are quite in line with extant work on female-female competition,” including the “queen bee” phenomenon. “If there is something undermining female-female mentorship,” she said, “we should investigate why.”

Reynolds told Inside Higher Ed that if she worked for Nature Communications, she’d request that scientists publish critiques of the paper and “save retractions for cases when there are data fraud issues or coding errors rendering the results invalid.” People may critique the operationalization of informal mentorship or the interpretation of findings, she said, but “I don't see those potential flaws as fatal.”

Beyond that, she said, the paper "should be treated as an opportunity to examine further the causes of these biases, if they are indeed there. If female scientists benefit from working with co-authors of both genders, we should know that.”

Not all critics want the paper retracted, but they are united in asking how it passed peer review. They cite the paper’s peer-review file, in which reviewers asked many of the same questions being posed now.

Anonymous Reviewer No. 1 wrote, for instance that mentorship is an understudied aspect of research, but that the paper as is “contains a number of major shortcomings.” The reviewer said the database is known to have “many problems with author disambiguation and tracking of citations,” and that the authors “use co-authorship as synonymous of mentorship which is not well justified as there are many more reasons to be a co-author than to be a mentor.”

The conclusion that "gender homophily in mentor-mentee relationships has negative effects for females ignores the historical aspects of this relationship," the reviewer said, "as men have enjoyed significant advantages and access to resources for their mentees.” There are “societal aspects in the data that cannot be ignored no matter how clever the matching method is for doing causal inference on observational data.”

Prior to any publication, “this paper needs major clarifications and revision, the least of which is to tone down the claim that they are analyzing ‘mentorship’ to something more accurate such as co-authorship," the review says.

The file-drawer effect, in which "positive" and even flashy findings that support a hypothesis are more likely to be published than "negative" ones, is well known. And few studies have unimpeachable methodologies and interpretations. But the outstanding criticism of this paper is that the authors made major leaps between arguably shaky data and their conclusions, on a topic of serious importance -- and that Nature Communications gave them a microphone. Perhaps the lasting conversations about this incident will involve questions about whether scrutiny of research prior to publication should increase with its real-world implications.

Mahal, of Alberta, said, “I don't think people realize how much this can affect women's careers. If ambitious students and postdoctoral fellows are told that working for women is going to prevent their careers from taking off, it has an impact. That can harm careers from the get-go.”

That said, Mahal doesn’t want the paper to be retracted, “because that just sweeps this under the rug and allows it to hide in the shadows.” Instead, she said, the editors at Nature Communications need to append an analysis of this paper to it, so that when it's distributed, "the flaws in the logic and the awfulness of it is made clearly visible.”

“We already knew there was a citation bias against women, and this article simply doubles down on that -- using it as a reason to undermine women as mentors rather than a real issue of the system," Mahal said.