Mentor or Risk Rejection

Grant applicants to the National Science Foundation lately might have noticed a new bit of commentary from reviewers: Requests for more hard data on how they’re going to mentor their students, and improve as mentors going forward.

The possible trend – so far anecdotal – was recently noted on the popular blog Female Science Professor. The anonymous moderator, who is a tenured professor in the physical sciences at a large research institution, said she and a handful of colleagues had noticed such language in their grant proposal reviews of late -- particularly within the last year. All were rejected.

Here’s a cited example from the blog:

“A highly qualified Ph.D. student has already been identified for this research but the mentoring of this student and an undergraduate is largely assumed based on prior experience of the [principal investigators]. The PIs have records of successful advising but should include in the proposal a more intentional discussion of how they plan to train the next generation of scientists.”

The NSF review continues: “The mechanism for success is not explained and there is no plan for assessing success of their mentoring. How will successful training of the graduate student be determined other than by the record of publications, presentations, and completion of the thesis? Although the research is potentially transformative and this is an excellent team of researchers, because of these shortcomings in the broader impacts I have given the proposal a lower rating.”

In a separate proposal review that included just one week of pay for a soft-money research scientist in whose lab students would do some work, reviewers wrote that the “[d]escription of the mentoring of the postdoc is not well developed. There is no mention of career counseling.”

The review said that mentoring in “professional activities such as writing proposals and papers is confined to discussions and support for participation in conferences and workshops” and that there was “no mention of how the postdoc will be mentored to collaborate with diverse groups of researchers and students.”

In a third example, reviewers of a proposal that included support for undergraduate research wrote that the principal investigators had “a long record of success in advising undergraduate students in research” but hadn’t included evidence “for how the field of research on undergraduate research will be advanced.” 

Female Science Professor wrote that it’s no longer “enough to have a record of success advising grad students, undergrads, and postdocs in research -- you have to understand and explain your advising techniques and you have to have a plan for assessing and improving.”

She said that even data related to student degrees, publications, conference publications and employment upon graduation weren’t “sufficient” for some reviewers.

“They want something different,” she said. “Apparently, unless you change something, you are not improving and therefore are not being transformative, or something.”

Numerous academic commenters offered their opinions and personal experiences with the topic, many confirming that they were being expected to have more detail on mentoring, and many doubting the appropriateness of that criterion when applying for research grants.

"I think the magic word is ‘portfolio,’ ” wrote Nicole and Maggie, a pair of semi-anonymous professors who moderate a blog called Grumpy Rumblings of the (Formerly!) Untenured. “I'm not entirely sure what a portfolio is, but the assessment police seem to like it even if they can't explain what one is either.”

Several other commenters, including Christie Rowe, an assistant professor of earth and planetary sciences at McGill University, attributed the possible trend to the NSF “splitting hairs” between good proposals in post-recession, sequestration-era research funding.

But others were more accepting of the NSF feedback.

“The funding agencies want to know that the funding will benefit trainees who are going to stay in the system and have success themselves as future PIs,” one anonymous commenter wrote, attributing the feedback to the state of the academic job market. “I completed two postdocs before leaving academia, and I think I would have felt better (and perhaps fared better) if there were very specific mechanisms in place to ensure that I was getting the mentoring that is foundational to future success.”

Good mentoring doesn't ensure success, the commenter wrote, but it can't hurt.

Another commenter said the reviewers were looking “for a plan/ideas that could be shared with others to improve training and benefit multiple students and advisers.”

In an email interview, Female Science Professor said that it’s “useful” that NSF asks principal investigators to pay attention to advising and mentoring, and to demonstrate that they’ve “put some thought” into it for the proposal. But, she said, “the problem comes when there is a focus on assessment, not just on having a mentoring plan or a history of successful advising.”

The professor said she considers her advising to be successful "if my students progress in a timely way towards their degree (eventually completing the degree), publish papers and give conference presentations, obtain skills that are useful for their thesis research and for their post-graduate careers, and then move on to a career that uses their Ph.D. training.”

Now, she said, “I am being told in proposal reviews that that is not enough because I don't specify well enough in proposals my plan for future mentoring.”

Some of the reviewers’ idea of assessment seems to be “very particular,” yet undefined -- a kind of "moving target,” she added.

Still, Female Science Professor said that she recently got back a proposal that contained no advising feedback – meaning that the feedback “trend” still depended on individual reviewers.

But other scientists outside of the blog have noticed similar NSF feedback.

Selmer Bringsjord, chair of the department of cognitive science at Rensselaer Polytechnic Institute, said he believed there was trend at least "firmly taking root” – although he noted that his evidence, too, was anecdotal. (He hasn’t seen such language in his own proposals, he said, but in those that come across his desk as department chair mentor to more junior faculty members.)

Frederick Dahlquist, chair of chemistry and biochemistry at the University of California at Santa Barbara, also said it was his “sense” that mentoring plan requests were becoming more common in reviews of NSF proposals.

Despite the rejections cited on the blog, Bringsjord said he wasn’t worried that the new criteria would “derail” the research agendas of faculty researchers with “superlative records” of co-authoring strong papers with students over a period of years.

Female Science Professor said she wasn’t sure about the reasons behind the mentoring push. But she guessed that NSF’s increasing emphasis on the broader impacts of funded research “gives reviewers extra ammo for shooting down a proposal.”

She added: “I hasten to mention that I support the idea and practice of broader impacts as long as the expectations of what an individual PI can reasonably do as part of a grant are not unrealistic.”

Last year, the NSF revised its longstanding Merit Review Criteria for grant proposals. One of the biggest changes was a new emphasis on the “broader impacts” of the project.

Previously, project descriptions needed to include a discussion of the research's broader impacts on society as part of the overall narrative. Now “broader impacts” merits its own, separate discussion within the narrative and two other parts of the proposal.

A spokeswoman for the NSF said there wasn’t any specific new push for data-driven mentoring in particular. But some of the NSF’s broader impacts relate, at least in part, to improved mentoring. They include “development of a diverse, globally competitive STEM workforce; increased partnerships between academia, industry, and others; improved national security; increased economic competitiveness of the United States; and enhanced infrastructure for research and education.”

Even before the broader impacts changes, NSF signaled an increasing focus on mentoring. In 2010, it adopted a postdoctoral mentoring policy first outlined in the 2007 America COMPETES Act for global competitiveness in science, technology, engineering and math (STEM). Under the policy, requests for funding for postdoctoral researchers must be accompanied by a Postdoctoral Researcher Plan. The plans must include a description of anticipated mentoring activities, such as career counseling, training in preparing grant proposals, publications and presentations, and guidance on how to collaborate with diverse groups of researchers.

Dana Topousis, NSF spokeswoman, said the agency places "importance on this topic because of the need to develop the next generation STEM work force."

As some commenters guessed, increased competition for funding also could be an issue. The NSF’s budget dropped to $7 billion to $6.9 billion from 2012 to 2013, with much of the loss due to sequestration. Total grants awarded dropped from 11,500 in 2012 to 10,800 in 2013, although several hundred more proposals were reviewed for merit last year than the year before.