Underrepresented Students, Unintended Consequences

For the last five years, efforts to assess and "shame" selective colleges based on their enrollment of low-income students have worked -- perhaps too well, two top economists of higher education assert in a new paper.

The study, a version of which was published late last week in Education Next, was conducted by Caroline Hoxby and Sarah Turner, economists at Stanford University and the University of Virginia, respectively. (Another version of the paper was released today by the National Bureau of Economic Research.) The studies assert that recent efforts by researchers and think tanks to rank colleges and universities on enrollment of low-income students, while well intentioned, have both unfairly judged some colleges' performance and led institutions to alter who they enroll in ways that disadvantage some low-income students, even as they help others.

"By engaging in what I would call public shaming around what I would call bad measures, the well-intentioned research leads to defensive and reactionary responses by institutions," Turner said in an interview. (Note: This paragraph has been updated to make clear that Turner, not Hoxby, made the statement.)

The two researchers propose a different way of measuring colleges' success in enrolling low-income students that takes into account both their missions and their demographic pools.

Pressuring Selective Colleges to Enroll Low-Income Students

Hoxby and Turner themselves played key roles in starting the campaign they now assert has produced unintended consequences.

Then a professor of economics at Harvard University, Hoxby was a co-author (with Christopher Avery, a Harvard professor of public policy) of an influential 2012 paper (commonly referred to as the "undermatching study") showing that many low-income, high-achieving students apply to no selective colleges, and that many such colleges were searching for such students at too narrow a number of high schools.

Hoxby and Turner then teamed up on a series of studies showing that adopting a set of relatively simple and inexpensive methods could get many more of those talented, low-income students to apply, enroll and succeed at highly competitive colleges.

Numerous other researchers have picked up this line of research in the last five years, perhaps none more visibly than the scholars associated with the Equality of Opportunity Project (now Opportunity Insights at Harvard University), whose work -- assessing colleges on how successfully they enroll and graduate students from the lowest economic quintile and propel them up the economic ladder -- captured the public imagination as few scholarly studies of higher education do. The study tends to be associated most clearly with the lead researcher, Raj Chetty of Stanford University.

Collectively, this line of research (embraced by policy analysts) has put significant pressure on highly selective colleges to enroll and graduate more low-income students. Institutions are now regularly rated and ranked by how many students they enroll who are eligible for federal Pell Grants for needy students, a ready demarcation of low-income status, and the Equality of Opportunity Project's data on intergenerational mobility (brought to a large public audience by The New York Times) has effectively shamed some institutions into changing their admissions practices and policies.

The problem, Turner and Hoxby argue, is not that the research has prompted colleges to change their behavior; the researchers wanted to prod institutions to enroll more students from low-income backgrounds. The problem has arisen, they say, because the use of what Hoxby calls "bad measures" has led colleges to adopt policies that "move the numbers up rather" than "thoughtful, proactive efforts to … increase the representation of low-income students broadly."

They raise two main issues.

First, they lay out (using what they call "proof by contradiction") how applying a national measure (like the bottom-quintile rankings used in the Equality of Opportunity data) makes some universities look better or worse than they actually are doing because their circumstances differ.

The University of Connecticut fares much worse than the University of Maine does in rankings of the share of their students who are from low-income families, Hoxby and Turner note. But the scholars instead construct what they call the "relevant pool" for each university, which includes all students from the state whose scores on either the SAT or ACT put them in their flagship’s “core” preparation range, from the 25th to 75th percentiles of their students’ scores.

Using that measure, students from the lowest percentiles of Connecticut's income distribution are overrepresented at the University of Connecticut (the red line in the chart below shows equal representation for each income level), while low-income students are underrepresented compared to the state population at the University of Maine. Students from higher income levels are more underrepresented at UConn than at Maine.

Though it gets less attention in their paper, the widely embraced tactic of judging colleges by how many Pell Grant-eligible students they enroll also comes in for criticism from Turner and Hoxby.

The Pell Grant is in many ways a logical way of judging whether students hail from a low-income background, because it is a national standard that is easily tracked.

But like any clear standard, using it also creates the possibility that institutions seeking to make themselves look better will take aim at it in ways that create problems.

That's what has happened in this case, Hoxby and Turner argue.

They examined data for two institutions (that they chose not to name) that have been credited with significantly increasing the number of Pell-eligible students they enroll.

The two charts below show how the adjusted gross family incomes of the two colleges' enrolled students changed from 2008 (when pressure was just beginning to build on selective colleges to enroll low-income students) and 2016. In 2008 (the chart with the blue bars), the colleges' enrollment of students just below and above the Pell Grant threshold (the red line) was roughly in proportion to what one would have expected based on what the researchers describe as the colleges' "relevant pool" based on their geography and mission.

By 2016, in contrast, the colleges were admitting almost twice as many students with incomes just below the Pell threshold as their relevant pools would have predicted, while they were admitting far fewer (still needy) students with incomes above the Pell threshold.

These data showing a "large discontinuity" between students just under and just over the Pell Grant threshold pretty clearly suggest, Turner said, that these institutions are "actively targeting Pell Grant recipients." Colleges like these are almost certainly giving significant financial support to those students to ensure they enroll, and because their financial aid dollars are limited, they are in all likelihood giving less financial aid to those low- and middle-income students who the graphs show to be enrolling at those colleges in smaller numbers than they were before.

But "if you’re a student who’s just above [the Pell threshold], you still need just as much financial aid as somebody who is just below," Turner said. "We know that there’s distortion going on there" in the colleges' aid policies.

Hoxby and Turner say they're not trying to cast aspersions on the work done by the researchers who have focused on Pell eligibility or the bottom income quartile as benchmarks -- especially since many of them are former students of Hoxby's, she notes.

But by embracing flawed measures and deciding to "do something that's a little bit sexy or prurient by putting people in rankings," Hoxby said, these analyses "may not actually be helping the situation, but making the situation somewhat worse. These measures are not measuring what they’re supposed to be measuring."

Staying Out of the Fray

A spokeswoman for Opportunity Insights -- whose work is most directly called out by Hoxby and Turner -- said the organization's policy was not to comment on studies by other researchers. She also said the organization will be publishing new research on this topic in the coming months.

Several other researchers who specialize in higher education access issues said they preferred not to comment about the dispute between two well-regarded groups of researchers.

One, however, said the Hoxby and Turner papers give short shrift to the "heightened sense of awareness of issues of inequality in access, with positive policy implications," that has been created by the researchers Hoxby and Turner criticize.

One prominent effort that has flowed from the data produced by Opportunity Insights and other researchers is the American Talent Initiative (ATI), which aims, by 2025, to increase by 50,000 the number of low- and moderate-income students enrolled at colleges and universities with high graduation rates (70 percent or greater six-year rates).

Joshua Wyner, who heads the Aspen Institute's College Excellence Program, which jointly administers ATI with Ithaka S+R, said in a prepared statement that "the issues Hoxby and Turner raise are important."

ATI members generally measure their progress to expanding access by counting increases in Pell enrollment, he acknowledged. But "most are also paying attention to moderate-income student enrollment just above the Pell threshold. In fact, when ATI was created two years ago, several members noted that they wanted to protect against a barbell effect -- meaning the enrollment of high-income and low-income students with little in between.

"As a result, most ATI schools now voluntarily submit to Aspen and Ithaka data on enrollments not just by Pell but also by income quintile."

Stephen Burd, a policy analyst at New America whose own work has built on the studies of Opportunity Insights scholars and other researchers, said in a series of tweets that Turner and Hoxby appeared to be protecting colleges from criticism that they are overenrolling wealthier students.