Evaluating Evaluations

To the many professors who say that student course evaluation data play too big a role in promotion and tenure decisions, a new study might come as both good news and bad. On the one hand, faculty evaluations seem to be getting more holistic, with deans gathering data from an increasing number of sources to assess professor performance. On the other hand, student evaluation results are playing a role in promotion and tenure decisions at more and more colleges.

The study, out in the American Association of University Professors’ journal Academe, also suggests that collegiality as a criterion for tenure and promotion is on the decline, and that value increasingly is being placed on research and publication -- even for professors at teaching-oriented liberal arts institutions. Service work and student advising matter more, too.

The study says it remains to be seen whether such changes make for more meaningful evaluations or better teaching, but that the stakes are higher than ever before.

“Years ago, the process of faculty evaluation carried few or none of the sudden-death implications that characterize contemporary evaluation practices,” the study says. “But now, as the few to be chosen for promotion and tenure become fewer and faculty mobility decreases, the decision to promote or grant tenure can have an enormous impact on a professor’s career.”

It continues: “At the same time, academic administrators are under growing pressure to render sound decisions in the face of higher operating costs, funding shortfalls, and the mounting threat posed by giant corporations that have moved into higher education. Worsening economic conditions have focused sharper attention on evaluation of faculty performance, with the result that faculty members are assessed through formalized, systematic methods.”

The study was conducted by Peter Seldin, professor emeritus of management at Pace University, and J. Elizabeth Miller, associate professor faculty and child studies at Northern Illinois University. Data are based on survey responses regarding promotion and tenure criteria from deans at 410 four-year liberal arts colleges. Researchers compared the results to similar data from 2000 to see how teaching evaluation practices had changed over time.

Seldin said that the data study is part of a larger longitudinal study going back decades, to his dissertation. He originally tracked four-year liberal arts colleges to keep his sample size small, he said, guessing that research and publication would be even more central to faculty evaluations at research institutions.

Perhaps surprisingly, Seldin and other experts said, little other research exists on the topic of faculty evaluations. The data in the study, which is called "Changing Practices in Faculty Evaluation," apply only to tenure-track and tenured professors.

The study’s first set of data show what percentage of deans consider various criteria to be “major factors” in evaluating overall faculty performance. Almost all deans said classroom performance was still the most important factor.

But research, publication, campus committee work and student advising have all increased in importance, and length of service in rank isn’t as important as it used to be. “Personal attributes,” which Seldin said were synonymous with the vexed notion of collegiality, have also declined in importance.

% of Colleges Considering Criteria to Be
“Major Factor” in Evaluating Faculty Performance

The study attributes the rise of research and publication to a desire to increase a college's prominence as a means to secure more public support.

One North Carolina dean wrote on the survey: “Most of our budget comes directly from the state legislature. They want faculty to publish, and present papers at professional meetings so our college stays visible.” A New York dean said, “High visibility is the name of the game today. It’s important to stay in the public eye.”

Seldin and Miller say such remarks "lend credence to the oft-heard observation that faculty members are paid to teach but are rewarded for their research and publication."

The authors link the increased importance of campus committee work to a "decentralization" of non-teaching duties, and the greater focus on student advising to retention, based on the idea that building relationships with students leads to student success.They also link the declining focus on "personal attributes" to more inclusive campuses. A Texas dean wrote, "We no longer expect conformity. Today, diversity is more highly valued." 

Who Evaluates Teaching?

A second data set suggests how deans evaluate teaching in particular. The study finds that the information-gathering process is becoming more “structured and systematic” and that colleges are “reexamining and diversifying” how they evaluate teaching.

Comparing data from 2000 to 2010, student ratings have increased in use and now factor into personnel decisions at 94 percent of colleges surveyed, from 88 percent in 2000. But at the same time, the sources of teaching evaluation data are becoming more diverse. Classroom observations and self-evaluations have increased, as have faculty committee evaluations (although deans and department chairs still do much of that work).

% of Colleges That “Always Used” Sources
of Information in Evaluating Teaching Performance

Seldin attributed the diversification of data to the rise of faculty teaching portfolios. Twenty years ago, he said, fewer than a dozen colleges used portfolios, as described in his books on the topic. Today, more than 3,000 colleges and universities use them, he said.

But deans’ opinions of student evaluations remain high, according to survey comments. A dean in Texas wrote, for example, that “[s]tudents are the most accurate judge of teaching effectiveness.”

Seldin said the biggest takeaway of the study is that “evaluation is changing. Student evaluations are still the most relied-upon source of information, but at the same time, even greater importance is being given to other types of information.”

Instead of the whole picture, student evaluations are now “one piece in a mosaic,” he said.

Henry Reichman, professor emeritus of history at California State University at East Bay and chair of the AAUP's Committee A on Academic Freedom and Tenure, said via email that deans are right to value student opinion.

However, he said, “It is essential that student evaluations not become popularity polls or simply reward the least demanding and easiest-grading teachers. In this light, I fully agree with those deans who opined that direct classroom observation is the best way to evaluate teaching.”

But Reichman stressed that such evaluations should be made by fellow faculty members, not by one or two administrators, which could allow for “meddling.” 

Jeffrey Buller, a dean at Florida Atlantic University who has written extensively about faculty evaluations, said the study focused perhaps too much on “input metrics,” or how well faculty members are perceived to teach, rather than “output metrics,” such as various measures of student success.

He also said research was about more than increasing a college's profile, noting the value of "the way in which research informs pedagogy."

Reichman said the decline in attention to "personal attributes" is “most definitely encouraging, since these should play no part at all in professional decisions. But it is nonetheless disappointing that nearly a fifth of deans still report taking such extraneous considerations into account.” (AAUP opposes the use of collegiality as a criterion for tenure.)

Robert Cipriano, professor emeritus and former chair of the department of recreation of leisure studies at Southern Connecticut State University, and author of Facilitating a Collegial Department in Higher Education: Strategies for Success, took a different view. He said that in his research, 20 percent of chairs have reported leaving a position due primarily to an less-than-collegial colleague, and that for that reason personal attributes may have a legitimate role to play in personnel decisions.