Imperfect Accountability

WASHINGTON -- One by one, as the U.S. Education Department under Secretary Margaret Spellings intensified its pressure on higher education to be more accountable to the public, groups of colleges unveiled voluntary efforts in which their members would collect and publish data designed to answer that call. The National Association of Independent Colleges and Universities put forward its University and College Accountability Network (known as U-CAN) and the country's two major groups of public institutions unveiled their Voluntary System of Accountability, on a site they called College Portraits.

"U-CAN profiles also include information identified by policymakers as important for accountability," the independent college group writes on its Web site. "Congress and the U.S. Department of Education have called for more consumer information to help the public evaluate and choose colleges -- a goal that NAICU strongly supports."

"The College Portrait is a source of basic, comparable information about public colleges and institutions presented in a common, user-friendly format,... designed to provide greater accountability through accessible, transparent, and comparable information," the Association of Public and Land-Grant Universities and the American Association of State Colleges and Universities say in describing their accountability system.

Nice rhetoric -- but the groups' accountability systems/networks don't live up to it, researchers from Education Sector and the American Enterprise Institute say in a report released today.

"[A] close examination of these two prominent efforts reveals serious flaws that undermine their utility as engines of accountability," Chad Aldeman of Education Sector and Andrew Kelly of AEI write in "False Fronts? Behind Higher Education’s Voluntary Accountability Systems." "For these efforts and others like them to improve consumer choice and exert meaningful pressure on schools to improve, they need to be more complete, comparison-friendly, and designed to highlight institutional differences.

"If existing flaws are not resolved, the nation runs the risk of ending up in the worst of all worlds: the appearance of higher education accountability without the reality. As such, policy makers and consumers should not be persuaded that these systems satisfy the need for increased transparency and accountability in higher education until their flaws are addressed."

The authors are particularly dismissive of the independent college group's U-CAN effort, which it says "provides almost no new information about costs, student experiences, or learning outcomes to parents and prospective students." While the site allows students and others to compare multiple colleges based on certain student and institutional characteristics (including college prices), "it does not obligate institutions to gather or reveal any data that are not already available elsewhere."

U-CAN, the report says, "is best cast as a pre-emptive attempt to fend off federal and state regulators, not a sincere effort to compel institutions to focus on consumer needs."

NAICU officials, not surprisingly, took umbrage at the groups' conclusions, noting that U-CAN has grown from 440 college and university profiles in 2007 to 708 now.

"The study criticizes the U-CAN's repackaging of existing data, without acknowledging that consumers historically do not know where to find this information in a consumer-friendly format," said David L. Warren, the group's president. "This is what U-CAN aims and succeeds in delivering. The report also fails to note that U-CAN provides original data on institutional net tuition -- information that was a precursor to the new federal requirement." U-CAN emerged in response to pleas from key members of Congress "for a simpler, less confusing way to get the vast reservoir of existing information on cost out to families in an institutionally comparable format," not in response to Spellings' call for learning outcomes data, Warren said.

"The wide diversity of private higher education's institutional missions makes a standard measure, or set of measures, of outcomes impossible. For this reason, U-CAN does not -- and will not -- prescribe one-size-fits-all learning measures for participating colleges," he said.

The authors spend a vast majority of the report detailing imperfections of the College Portraits Web site -- which is in many ways a tribute to the two public college associations that put together the Voluntary System of Accountability. "There's a whole lot more to the VSA than there is to U-CAN, and given some of the issues they tackle, we applaud them for it," Kelly said in an interview. "But while we applaud them for what they've done so far, that doesn't mean we should be satisfied with it."

The report critiques a range of structural and other limitations in the system's approach that impair its usefulness for consumers and policy makers alike. Most fundamentally, College Portraits does not allow users to compare institutions to one another based on a set of characteristics of their choosing, "nor can they easily rank schools on any of the criteria that they might want to."

Although the report notes that data from each institution's College Portrait page resides on the college's own Web site -- which was true when the project began -- that is no longer the case, said Christine Keller, executive director of VSA. The data is now collected on a centralized Web site and the groups could present it in a way that allows users to compare institutions (as the report's authors suggest) if there was demand for it, Keller said, but "there hasn't been a great outcry from our users that it's something they would really need and want."

Kelly and Aldeman praise the public colleges' voluntary system for doing something others have criticized it for: requiring all participating institutions to use one of a handful of measures of student learning outcomes. "Colleges and universities are generally loath to submit to to this kind of standardized testing for fear of how their results might compare to those of their peers," the authors write. They also praise VSA for collecting and publishing data on "student engagement" that many institutions use but many fewer publish.

But they point out that the way that the accountability system uses data from the National Survey of Student Engagement results in relatively little variation among institutions, and note that institutions use drastically different "cost calculators" that make any attempt to compare price information difficult.

The fact that some of the most selective and highly visible public universities in the country (University of California campuses at Berkeley, Los Angeles and San Diego, the University of Michigan, and Georgia Institute of Technology, to name several) and numerous colleges with low graduation rates have opted out of the VSA lead the authors to suggest that such accountability systems should be mandatory instead of voluntary.

They argue less for a federal mandate (though "it's not as if the government mandating that schools report data is a new concept," said AEI's Kelly) than for states to ensure (as North Carolina does) that all of their public institutions submit their information. "If the higher education market is to exert pressure on poor-performing institutions, consumers must have the necessary information to make informed choices and 'vote with their feet,' " the authors write. "Though social and political pressure to join voluntary systems might succeed in the long run, statutory or regulatory pressure from state legislatures to increase transparency could pay more certain and immediate dividends."

Keller of the VSA said officials at the two public college groups were fairly heartened by the think tanks' report, which she said recognized "that the VSA is a pioneering higher education accountability project."

She said the authors made some valid and helpful points about how the College Portraits site might be improved, but dismissed the need for more mandates to participate -- "which is exactly what we were trying to avoid by developing the voluntary system" -- and urging patience instead.

"With two-thirds of our membership participating, and upwards potential, I think we're doing pretty good job representing the public colleges and universities to the public," she said. "We need to continue to work on this, but we've got a good faith effort going."

Aldeman acknowledged that the existing systems were works in progress, and noted that he and Kelly had weighed in not only to critique the existing accountability projects but to shape those still in the works -- to be put forward (at some point, presumably) by groups of research universities and community colleges.