NSSE's Validity Questioned

Ask many a college president about assessment of student learning at his or her institution, and the first thing you hear is “Nessie,” as the acronym of the National Survey of Student Engagement is pronounced. The increasingly popular survey is offered to students at hundreds of colleges -- asking students questions about their in-class and out-of-class experiences, interactions with faculty and peers, and much more. The theory behind NSSE is that such measures of engagement are crucial to student success.

Amid all this praise, a small but growing number of researchers are questioning whether there really are positive correlations between NSSE measurements and the outcomes that tend to be most important to colleges: retention and graduation. Two new studies raising such doubts will be released this week in New Orleans at the annual meeting of the American Educational Research Association.

One study -- by two Auburn University education professors, David DiRamio and David Shannon -- found no relationship between total NSSE scores in the various subcategories of the survey and either grade-point averages or graduation rates. It did find positive outcomes in these areas associated with a few NSSE questions.

A second study -- by Amy Korzekwa and Scott C. Marley of the University of New Mexico -- examined the relationship between NSSE sub-scores and found them only "minimally predictive" of first-year grades and "not at all predictive" of third-semester retention.

Each of these studies was conducted at a single unnamed university. They follow on 2009 work by Stephen R. Porter, an associate professor in Iowa State University's educational leadership and policy studies department, that questioned whether students responding to NSSE surveys (or many similar surveys) really understand the questions sufficiently well to provide meaningful answers. Other studies, of course, most notably a national study by researchers at Wabash College, have found links between the qualities NSSE measures and academic success.

The potential significance of the new studies is summed up in the DiRamio-Shannon paper, which notes the way college administrators and trustees have come to rely on NSSE as an all-purpose measure of quality, when it is not as focused on solely academic measures as some might imagine.

"A trustee," they write, "could be listening to a glowing report about how well her institution is doing in terms of student engagement, not realizing that high NSSE scores might actually be negatively associated with outcomes she considers desirable for the institution and its students. After all, doesn't engagement take away from the 'time on task' that a student would normally devote to coursework and studying?"

Such a statement is consistent with the message of the new book Academically Adrift, which argues that academic rigor, not engagement, is the key to improving student learning.

The DiRamio-Shannon paper notes a key paradox with regard to how NSSE is used. Many of the college presidents who praise it link it more closely to academic achievement than NSSE officials ever have. Citing the lack of any statistically significant relationship between NSSE scores and either grades or graduation, the authors note, "NSSE was never designed to measure these types of student outcomes, although it would certainly be encouraging if measures of engagement had some relationship to the sorts of outcomes that are of concern to senior administrators and governance officials."

In fact, the most encouraging news for institutions that rely on NSSE is that these studies don't find negative correlations between engagement and the academic achievement measures studied.

But both studies conclude with essentially discouraging statements about NSSE's value in measuring anything beyond engagement levels. DiRamio and Shannon write that NSSE is "murky and ambiguous at best" as a predictor of grades or graduation. Korzekwa and Marley write that when it comes to student persistence to a third semester "the predictive validity of NSSE is questionable," and that "considering the widespread usage of the NSSE these results are of considerable importance."

Jillian Kinzie, associate director of the NSSE program, said that it was important not to read too much into studies that are conducted at one institution, and she noted that the Wabash study -- more favorable to NSSE -- is from a broad range of institutions. But Kinzie added that she and her colleagues are pleased to see more scholars looking closely at NSSE, as attention will promote better use of NSSE results.

She also noted that NSSE is always making adjustments and is planning a major revision in the coming years, for which these findings may be helpful. "We take these results seriously," she said.