Gaming the Formula

Most states now have some form of performance-based funding for higher education on the books. Policy makers use performance formulas to prod public colleges to be more efficient in their spending and, in some cases, to work harder to improve student retention and graduation rates.

A new academic paper, however, found an early indication of an unintended consequence of the policies: less revenue from Pell Grants on a per-student basis, which could indicate that colleges are seeking to game performance-based funding by enrolling fewer low-income students.

A growing body of research has tried to determine whether state funding formulas actually are improving graduation rates or other student outcomes, with mixed results. But the new paper by Robert Kelchen (pictured below) and Luke J. Stedrak, both assistant professors of education at Seton Hall University, may be the first to explore whether the policies have affected colleges’ revenue and spending patterns.

The analysis drew data from multiple sources, including federal databases, to track institutional finance patterns over the decade ending in 2013. It examined public four-year institutions and community colleges across the nation, with a sample of 1,649 public colleges. (Many public technical colleges that grant certificates were excluded.)

Overall, colleges subject to performance funding received $30-40 less in federal Pell revenue per full-time student, said Kelchen, a dip of up to 2 percent across all students. The revenue gap applied both to community colleges and four-year institutions.

“These results suggest that public colleges that face at least some funding being tied to outcomes do change their expenditure patterns and potentially even the composition of their student body,” the paper said.

The research also found an increase in institutional grant aid at four-year colleges in performance-funding states, relative to other colleges. For example, four-year colleges that were subjected to performance funding in the previous year spent $43 more in institutional (or unfunded) grant aid, which was about 5 percent of the median college’s total spending on institutional grant aid.

Grant aid from college coffers often is merit based, as New America detailed in a recently released paper. However, some colleges focus institutional aid on needy students.

As a result, the finding in the new research paper “suggests colleges may be trying to recruit more students from higher-income families via the strategic use of institutional financial aid,” Kelchen and Stedrak wrote.

Kelchen cautions that more research is needed to get a better sense of what may be driving the changes in revenue and spending.

“We need a lot more work on the nitty-gritty on the formulas and how they’ve changed over time,” he said.

In addition, Kelchen said many states, such as Ohio, use performance-funding formulas to encourage public colleges to enroll and graduate more low-income students, not fewer.

“Are the incentives in performance funding really driving it?” he said. “Are colleges responding rationally to the incentives they’re given?”

The study included some encouraging findings. For example, it found evidence four-year colleges that were subject to performance funding received somewhat more in state appropriations in the period after performance funding was adopted, although that amount faded during a longer time frame.

In addition, community colleges in performance-funding states spent slightly more on student services, the paper said, which could help more students graduate.

More Money, More Data

In the past it has been hard to use data to gauge how performance funding is working, in part because most states weren’t tying much money to formulas. Most colleges in the study had between 1 and 5 percent of total state appropriations tied to various performance metrics, the study found.

That’s changing, however, as more states follow Tennessee and Ohio in linking larger portions of budget contributions to performance formulas.

“We’re just starting to get to that point,” said Kelchen. “We have states tying 25 percent of funding.”

An annual study by HCM, a public policy and advocacy consulting firm, tracks various performance-funding policies that are emerging in states. Last year HCM found 35 states either developing or using formulas that link support for public colleges to student completion rates, degree production numbers or other metrics.

More has happened since 2013, the last year of data included in the new research by Kelchen and Stedrak. And many of the performance formulas were relatively low stakes during that time period, given the money in play. But Kelchen said that makes the substantial gap in Pell revenue more interesting -- because the decline could worsen in subsequent studies.

The variation is wide among performance formulas. And, as the paper notes, even determining which formula counts as performance funding is difficult. That also means it's hard to measure the impact of these policies in the aggregate, said Mamie Voight, director of policy research at the Institute for Higher Education Policy, who has written about performance funding.

"Each state’s system is different in terms of which metrics it uses," Voight said via email, "so performance-based funding could have one effect in one state and an entirely different effect in another state, largely because of what the performance-based funding formula prioritizes."

Kelchen said several qualitative studies have described how colleges are responding to performance funding, and possibly gaming outcomes. For example, he pointed to research by Kevin Dougherty, an associate professor of higher education at Columbia University’s Teachers College.

That possible response by colleges -- seeking to become more selective -- underscores the need for policy makers to be thoughtful as they design performance-funding systems, Voight said.

"It’s crucial that performance-based funding formulas -- or any accountability system, for that matter -- incorporate measures of access and success for low-income students and students of color to prevent unintended consequences," she said.

Trey Miller, an economist at the RAND Corporation, also has researched performance-based funding. He said the amount of the Pell revenue decrease identified in the new study was small yet statistically significant.

Gaming the formulas is one possible interpretation, Miller said. But there are others. For example, he said, increasing enrollment levels of undocumented immigrant students -- who cannot access federal aid -- could affect the numbers. And colleges could be using merit-based institutional aid for other reasons than responding to performance funding.

“We need to do more research about what’s going on,” he said.

Kelchen agreed, and said one remaining question is whether colleges are enrolling fewer Pell-eligible students or enrolling the same number of students who received less money.

“There’s a lot more work to be done,” said Kelchen.