Measuring Other Outcomes

As the old saying goes, money can’t buy happiness. And yet, in measuring alumni success and satisfaction, colleges – often prodded by those seeking to hold them accountable – typically look at two things: whether their former students are gainfully employed, and whether they’re making a decent salary.

A new project announced today, led by Gallup and debuting at Purdue University, aims to change that. Focusing on a set of factors that are shown to correlate with “a great life,” the survey of 30,000 graduates annually will provide data on how alumni of groups of colleges (public or private institutions in certain states, for instance, or athletic conferences) are faring and how they compare to national averages. The final product will be a benchmark for student success against which any campus can measure its own graduates, if it works with Gallup individually.

At a time when students and parents are increasingly saddled with debt and President Obama wants to tie federal financial aid to colleges’ performance on a new ratings system, Gallup and other proponents of the project say it will increase institutional accountability and provide a more holistic look at the returns on an education.

The Gallup-Purdue Index "contains strong elements that make it superior to many other existing and proposed outcome measures," Association of American Universities President Hunter Rawlings said in a statement, and "[avoids] the myopic notion that financial outcomes like earnings constitute the primary value of a degree."

Anthony Carnevale, director and research professor at Georgetown University’s Center on Education and the Workforce, said the project is a step up from tired studies on student engagement and completion.

“Ultimately, the purpose of a college education is not to get you a job and make you money,” said Carnevale said. “The purpose is to allow you to live more fully in your time.”

Purdue President Mitch Daniels called the survey "a matter of responsibility" in an interview.

"We owe it to potential students, we owe it to businesses who might recruit our students, to be able to say something with statistical confidence about the quality of our graduates,” Purdue President Mitch Daniels said in an interview. “As each of us compares our own graduates to the new national norms and averages that we’ll learn about, we expect – at least over time – to derive some lessons that we can bring back to the campus and try to do an even better job.”

However, others are skeptical about the data’s value.

“What you have is a thermometer with no theory of action behind it,” said Mark Schneider, vice president of College Measures at the American Institutes for Research. “If I have a fever, nothing here is going to tell me how to deal with the fever. All it’s going to do is tell me I have a fever.”

The “largest representative study of college graduates in U.S. history,” the Index will begin with a survey of that university’s alumni. But Gallup will supplement those findings by asking the 1,000 people it surveys by phone every evening if they would also submit to questioning about their education and current situation. The inaugural results will be reported in the spring, by state, Carnegie Classifications and athletic conference. 

While Purdue is the only university contracting with Gallup so far, the project’s organizers hope to expand to include many campuses. Gallup hasn’t yet talked to other institutions about joining, but “there’s definitely interest out there,” said Brandon Busteed, executive director of Gallup Education.

The survey’s line of questioning goes beyond job placement and salary, also inquiring about work place and community engagement, personal relationships, physical fitness, sense of purpose and happiness, and economic management and stress.

A sampling: Do you like what you do at work each day? Is it what you’re best at? How would you rate your life satisfaction on a scale of one to 10? Do your friends and family give you positive energy each day? Are you recognized for improving your community?

“No one is going to suggest that colleges and universities are responsible for 100 percent of your great job and great life,” Busteed admitted, “but obviously, if you go to college and you get a degree, the odds are you increase the probability of having a good outcome.”

Gallup and Purdue are right to be looking at these questions, said Mike Cahill, director of career services at Syracuse University.

“I think some of the metrics that we’re used to using – salaries, prestigious jobs – I don’t think they go quite far enough, and I think this is a study that does,” Cahill said. “It’s certainly something that I would entertain bringing in.”

But Schneider, whose organization works with states to measure job outcomes, stressed that he hasn’t seen how Gallup “validated” the measures with which it’s gauging alumni success (Gallup says it’s through years of surveys on workplace engagement and well-being). But even if they are legitimate indicators of colleges’ performance, he said, the means of comparison between institutions is not specific enough to be meaningful.

“When we start rolling up things to that level of aggregation, I think we’ve actually lost a lot of the most valuable information that we need,” Schneider said. “Why would I want to know Indiana versus Illinois, when the fact of the matter is that it’s variance that really matters?” In other words, when each campus has its own unique systems and attributes, how can one identify what it's doing wrong when comparing itself to a large, generalized group?

Regardless of whether the data are useful for comparison, Cahill, of Syracuse, said it should be helpful for individual campuses.

“The expectation that any quick changes are going to come along – probably not. But at least it will raise the issue,” he said. “To add some data to it is going to be critical.”