Accountability System Launched

A new way for students and their families to compare colleges -- and for legislators and others to evaluate them -- was unveiled Sunday with the start of a campaign to get institutions to sign up to use it.

“College Portrait,” as the effort is called, is a template for information that public, four-year institutions will provide online in an easily comparable way. Some of the information -- statistics on the student body, figures on college costs -- is fairly commonly found (if not always in comparable ways) on colleges’ Web sites today. But the program also includes a new method for measuring graduation and retention rates and, controversially, a requirement that institutions that choose to participate conduct and release results from standardized tests as a means of measuring the learning that goes on at their institutions. Those tests would be administered to small, representative cohorts of students -- possibly 100 or fewer freshmen and a similar group of seniors -- and would not be generally offered or required of all students.

College Portrait was released at the annual meeting of the National Association of State Universities and Land-Grant Colleges, whose members – along with those of the American Association of State Colleges and Universities -- developed the system. Association leaders have viewed the effort as a way to respond to the Spellings Commission and other demands for greater accountability for higher education, but to do so in a way that was more sophisticated than a legislatively designed system might be. And one emphasis of the effort has been the importance of such a system being voluntary (College Portrait is part of what the associations call the Voluntary System of Accountability) and designed from within higher education, rather than imposed by others on colleges and universities.

Peter McPherson, president of NASULGC and the prime mover behind the effort, was blunt in an interview about why colleges should embrace this process -- or risk having federal officials come up with another system. "If we can't figure out how to measure ourselves, someone else will figure out how to measure us," he said. "It's inevitable."

A key part of the push for more accountability in higher education -- at least as voiced by the Bush administration -- has been on the need for comparative data and College Portrait would provide that. But one question mark about the effort has been whether any voluntary program would attract enough participation to enable comparisons to be made. At the NASULGC meeting, in New York City, organizers noted that they had pledges of participation -- even before Sunday’s official invitation for participations -- from such prominent and large higher education systems as the California State University, University of North Carolina and University of Wisconsin systems, as well as the Universities of Iowa and Tennessee.

But what NASULGC leaders didn’t announce was that the University of California’s nine universities have all decided not to participate, citing the testing requirement as something that “usurps the role of campus and departmental faculty in assessing student learning.”

California’s decision raises the question of whether a system that will allow for comparisons of Chapel Hill and Madison but not Berkeley or UCLA will have the national value that its supporters hope. McPherson said that in any voluntary effort, some colleges would opt out, and he predicted that in the end, participation would be "wide and deep."

College Portrait has three parts: student and family information, student experiences and perceptions, and student learning outcomes.

The student and family information section includes a variety of standard demographic categories, and also two features that proponents are particularly excited about. One is the College Cost Calculator, which will allow prospective students to answer a few basic questions about family income, number of siblings in college, etc., and find out roughly how much aid they would receive and what they would need to pay to attend. The idea is that many students don't realize how much help they can receive.

The other is the Student Success and Progress Rate, which aims to provide an alternative to the federal graduation rate, the commonly used statistic that provides information on the percentage of first-time, full-time students who graduate within 150 percent of the expected time (so three years for an associate degree or six years for a bachelor's degree). Many educators, especially at institutions that serve nontraditional or non-elite students, have questioned the value of the statistic.

College Portrait would handle this issue by providing information on how many students graduate within four or six years, how many are still enrolled, how many have graduated from another institution, and how many are still enrolled at another institution. In addition, the same information would be provided for those who transfer into each institution. Hallway buzz at the NASULGC annual meeting was particularly positive about this feature, as many public universities feel that the traditional federal graduation rate measure doesn't reflect the realities of the way students move from institution to institution.

For the student experiences and perceptions category, participating institutions will be required to report data from one of four surveys: the College Student Experiences Questionnaire, the College Senior Survey, the National Survey of Student Engagement or the University of California Undergraduate Student Experience Survey. The survey results focus on six areas: group learning, active learning, experiences with diverse groups of people and ideas, student satisfaction, institution commitment to student learning and success, and student interaction with faculty and staff. While many colleges have responded to calls for accountability by touting their scores on student engagement surveys, such data has historically only been released selectively (except by community colleges) so the College Portrait requirement could make the availability of such information much more widespread.

The testing requirement has been by far the most contentious issue in the new program. Many professors fear that standardized testing is already overused in American education, believe that it tends to favor the wealthy over the disadvantaged (students or colleges) and that the testing-driven No Child Left Behind has been a disaster. The letter from Robert C. Dynes, president of the University of California, telling College Portrait organizers that the system would stay out of the project, lavished praise on the effort -- except for the testing requirement. "The university has concluded that using standardized tests on an institutional level as measures of student learning fails to recognize the diversity, breadth, and depth of discipline-specific knowledge and learning that takes place in colleges and universities today," Dynes wrote.

Most presidents here said that they hadn't started formal discussions of the testing requirement with their faculty leaders, but expected testing to be the key area of concern. College Portrait supporters have indicated that they will lobby hard -- they sent officials to Iowa City last weekend to brief faculty senate leaders gathered from several large public universities.

The testing requirement in College Portrait will have two parts. In one part, colleges will provide data from state licensure examinations and other tests taken by graduates of certain programs so that prospective students can see, for example, the passage rates on the tests required to become a nurse or a teacher in the state.

Participating colleges will also be required to use one of three measures of broad cognitive skills, such as critical thinking, analytic reasoning and written communication. The three measures are the Collegiate Assessment of Academic Proficiency (from ACT), the Measure of Academic Proficiency and Progress (from the Educational Testing Service) or the Collegiate Learning Assessment. Students would take the exams as freshmen and seniors so that results would indicate both test scores and gains during undergraduate education.

In a reflection of how sensitive the testing issue is, the plan is to give colleges up to four years to determine the best way to carry out the testing requirement. During that time, colleges may opt not to report test results.

McPherson said that the only requirement from College Portrait is that the testing group be reflective of all undergraduates and that the testing cohorts reflect the standards of the test makers with regard to such thorny issues as how many need to take the test, what incentives may be offered and so forth. Officials of the CLA, for instance, say that they can get 95 percent reliability with sample sizes as small as 100 freshmen and 100 seniors -- even at large public universities. In terms of what's acceptable to get students to take the test seriously, ETS has a handout of practices it considers acceptable, including some sticks (requiring it as a condition of registration or receiving a diploma) and carrots (priority status for dormitory space, parking privileges for a month or semester, pizza, or a reception with the president, among others).

To those who are dubious of testing because of No Child Left Behind, McPherson stressed that this was different. "We're not testing everyone," he said. He also noted that the long phase-in for testing reflects the desire to let universities experiment and come up with a good system, not to force practice into place immediately.

Mary Evans Sias, president of Kentucky State University, spoke at the meeting about having initial fears about testing, and then getting over those fears. Kentucky State is a historically black college, and Sias said that as part of its mission, it admits students with test scores and educational backgrounds that would cause them to be rejected at many other institutions. Sias said she feared being judged by those scores. But the "value added" approach to testing -- in which scores of both freshmen and seniors will be reported -- is actually a potential boon to colleges like hers, she said, because she will be able to demonstrate the progress her institution makes with its students.

Bruce L. Mallory, provost of the University of New Hampshire, asked the panelists who spoke in favor of the new system whether they had considered that the press might take the information and devise new rankings or use the information to bolster current rankings. While the panelists said No, emphatically, Mallory wasn't convinced. A phone interview Sunday night with Robert Morse, who heads the U.S. News & World Report college rankings, suggests that Mallory was on to something. Morse, who praised the College Portrait project for providing meaningful information to students and parents, said that he hasn't seen the final version, but that it might create "an opportunity" to add to the data his magazine uses. The magazine might, for example, take information that the College Portrait has on a college-by-college basis and group it together for "comparative purposes." Morse said it was too early to see if the new data could contribute to actual rankings, but that there are "certainly possibilities" for the magazine.

The overall theme from supporters of the new system was that, whatever concerns people have, they need to get over them and focus on the bigger picture, which is avoiding a federal system of defining how assessment should take place. McPherson said that "if we wait for the perfect test, we'd be waiting 10 years from now." He predicted that the endorsement of this process by NASULGC and many of its members would help assure that tests are improved. "You need to take a step like this to force the pace," he said.

And speaker after speaker said that the pace couldn't be allowed to slow down. William E. Kirwan, chancellor of the University System of Maryland, said that with the Spellings Commission talking about new assessment systems, the leaders of the state university group "came to the rescue" with this effort.

Charles B. Reed, chancellor of the California State University System, said he was excited about the new system and said that legislators and the public will embrace it. "The more information we provide to the public, the better off we're going to be," he said. In fact, California State has added additional material to its template for College Portrait to highlight information that would be of particular interest to legislators. This "public good" section will include data on the numbers of graduates in key fields, such as nursing, engineering and education. In addition, data will be provided on degrees awarded to minority students.

McPherson and others said that NASULGC had been thinking about these issues before the Spellings Commission, but many at the meeting viewed the College Portrait through the prism of the commission. Reed said he believes that College Portrait will be "the biggest idea to come out of the Spellings Commission."