Diversity and the Rankings

It is rankings season again, and despite objecting to their validity and worrying about the incentives they create for colleges and universities, we pay attention nonetheless. And, since they aren't going away, I would like to propose one small change: factor the socioeconomic diversity of a college's student body into the rankings formula.

Consider the top-ranked colleges and universities as a case study. They proclaim in their mission statements and through their financial aid policies that they want to attract the strongest students from all different backgrounds regardless of their families' incomes -- both in support of principles of equal opportunity and fairness, and because a diverse student body improves the quality of education. These institutions commit significant resources to need-based financial aid, approaching as much as a third to a half of their operating budgets. These institutions' success in attracting a diverse student body could easily be incorporated into the data series used by U.S. News & World Report to rank our institutions.

And, for those of us who don't think any one ranking makes any sense, reporting these data would still serve the purpose of informing prospective students and their families about how institutions are doing in accomplishing one of their stated objectives. U.S. News reports separate rankings for socioeconomic and racial diversity, but the magazine's main ranking clearly receives the most national attention. Leaving diversity out of its main ranking formula suggests this criterion is of secondary importance, in contrast to the importance diversity receives in the mission statements of these institutions.

The socioeconomic diversity of the student body at an institution would be expected to be related to the selectivity of the school and the academic credentials of the entering students. It is an unfortunate characteristic of our society that achievement in high school correlates with family income. The possible explanations range from prenatal care to quality of high schools to SAT prep. Whatever the causes, the data are clear. To take this into account in a ranking such as U.S. News’s, one option is to include a measure of socioeconomic diversity relative to what one would expect, given the selectivity of the school. This would be similar to U.S. News's current treatment of actual graduation rates compared to predicted rates based on the students' test scores and institutional resources.

Here is one way to do this. We have the Department of Education data on the share of students receiving Pell Grants and we have data from the Common Dataset on the share of students receiving need-based financial aid at each of the schools ranked by U.S. News. Pell Grant recipients come from families with incomes below $40,000, which is approximately the bottom 40 percent of the income distribution in the United States. We also have data that report the share of students in each income quintile by SAT/ACT results for all high school seniors who took the test for one year. While not a fully adequate measure of academic credentials of high school graduates or the admissions selectivity of the schools, test scores do provide one possible measure. They can be used to examine the income distribution of students qualified to be enrolled at colleges with different levels of selectivity.

For example, our data indicate that for students with SATs (and ACT equivalents) of 1420 or above, 12.8 percent come from the bottom 40 percent of the income distribution. If a college with SAT selectivity at this level has proportionally fewer Pell Grant recipients than this, then it is not doing well on socioeconomic diversity relative to what one would expect given the SAT/ACT scores and income profile of students in the country. If such a college has more than 12.8 percent enrollment, then it is overachieving. So we suggest that the share of Pell grant recipients in a school’s student body, relative to what would be predicted given the school’s selectivity, would be a new variable in the overall U.S. News ranking formula.

Similarly, the total share of students receiving need-based aid at each college can be compared to what would be expected, again given the national data on the distribution of students by income and SAT scores. In our analysis we use the share of students from the lower 80 percent of the income distribution at each SAT level as the expected proportion of students on need-based aid, given the selectivity of the college. Again, having a larger share of students on need-based aid than the data would predict given a college's selectivity would count as doing well. While the Pell Grant variable we proposed would capture a college's success at recruiting lower income students, this financial aid measure would address all families except those with incomes high enough to pay a college’s full price.

The table below reports the U.S. News rankings (published in August 2010) for the top 15 national liberal arts colleges, as well as the rankings of these colleges on our predicted Pell Grant variable and need-based financial aid variable. Looking at Pell Grants relative to predictions, Amherst College (second in the overall U.S. News ranking of colleges) comes in first place with 20 percent of its first-year students receiving this aid, well above the predicted 14.9 percent given Amherst’s SAT profile. Vassar (tied for 12th overall in U.S. News) is second on the list, exceeding what the data would predict by 4.3 percentage points. Even though Vassar, like Amherst, had 20 percent of its first-year students receiving Pell Grants, the data indicate that Vassar would have a higher percentage of students on Pell Grants due to the college's slightly lower level of admissions selectivity. Pomona College (sixth overall in U.S. News) moves to third in this socioeconomic analysis, exceeding predicted enrollment of Pell recipients by 3.1 percentage points, with U.S. News top-ranked Williams fourth by this measure, exceeding the data prediction by 2.3 percentage points.

When measuring the percentage of students receiving need-based financial aid, the order shifts further. Vassar moves to the top with 63.5% of its freshmen receiving the school’s need-based scholarship aid, exceeding the data prediction by 3.1 percentage points. Amherst follows, with its 59.6% of students receiving need-based aid, exceeding the data prediction by .9 percentage points. All the other schools among U.S. News’s top national liberal arts colleges enroll below their predicted level. Wellesley and Smith are the third and fourth on the need-based aid list, below their predicted enrollment levels respectively by 2.6 percentage points and by 5.5 percentage points. Williams is seventh, beneath its predicted level by 7.8 percentage points.

Since the ranking results differ between the U.S. News measures and these new socioeconomic measures, adding either or both of these new measures to the U.S. News formula could change its results, depending on the relative weight assigned to these new variables among all of U.S. News’s criteria. Given the emphasis on the importance of diversity in the mission statements of these institutions and the amount of resources spent on financial aid (easily exceeding that spent on faculty salaries at many of these institutions), one could argue for a significant weight and therefore changed rankings. While I've used the selective private liberal arts colleges to make the case for valuing diversity in the rankings, it is equally important for public colleges and universities, including the land grant and flagship institutions. Access and equal opportunity are core commitments of these institutions, and recognizing their accomplishments in this area in the rankings is equally as important as for the private, nonprofit sector of higher education.

What would be the advantage of adjusting the rankings in this way? The existing rankings create incentives for the allocation of resources at colleges and universities. (Whether colleges actually allocate resources in response to these incentives is hard to prove or disprove, but it seems quite likely that it happens at some schools at some points in time, despite proclamations to the contrary.) Currently, schools do not receive much credit in the rankings for expenditures on financial aid and success at diversifying their student bodies. (One could argue it might be included in the broad “reputation” measure, but this is indirect and not possible to confirm.)

During the recent economic downturn, many of these schools had to make difficult budget decisions. It is clear that many schools looked to reducing financial aid to help solve their financial difficulties. In trying to make the best possible choices, decision-makers would have considered the effect on their educational missions of cutting from financial aid relative to the effect of cutting from other places -- faculty salaries, building maintenance, or the quality of the food.

Cutting financial aid may seem the easier choice because in part it affects potential students who might have been part of an academic community, but who now won’t be. They are anonymous. With a new socioeconomic variable in the core U.S. News rankings formula and a significant weight on the variable -- given the commitments colleges make to diversity and equal opportunity -- institutions would gain a powerful incentive to live up to their commitments and to these otherwise anonymous students.

Top Liberal Arts Colleges in U.S. News -- and by Socioeconomic Diversity