Rifts Over Global Test of Learning

PARIS -- Most of the discussion at the biennial conference of the Organization for Economic Cooperation and Development here this week was, per the group's name, about potential collaboration in solving the common issues and problems that many countries face in their higher education sectors: shrinking government support, growing dependence on tuition dollars and rising student demand, to name several. They didn't quite break into a chorus of Kumbaya, but darn close.

Behind the scenes, however, a potential conflict was gathering momentum over one of the OECD's more-controversial initiatives: its Assessment of Higher Education Learning Outcomes (AHELO), an international assessment of student skills and knowledge that is several years in the making.

Scores of colleges from 17 countries, including the United States, have been participating in a nearly completed feasibility study of the AHELO test, and at a session at this biennial meeting Tuesday, officials from Russia and Mexico joined OECD staff members in describing their work so far and expressing enthusiasm about the results they'd seen so far.

"We will carefully review the feasibility study to see whether [the test] is actually possible and scientifically sound," said Karine Tremblay, a senior analyst at OECD. "So far the results are encouraging."

But agitation about the test's potential uses (and misuse) is rising among some of the OECD's largest and most powerful constituents.

In a letter sent in late July to the leader of the organization's higher education program, Deborah Roseveare, the heads of the major associations of colleges and universities in the United States, Canada and Europe expressed their concerns about how OECD envisions the test being used -- and specifically that the organization "is attempting to create a transnational 'test' of learning outcomes, without clarity of purpose or consultation with institutions."

The letter, from the American Council on Education, the Association of Universities and Colleges of Canada, and the European University Association, was designed to reinforce unhappiness that a group of college presidents from the represented countries expressed in person to Roseveare at a meeting they held in Salzburg, Austria, in early July. There, the presidents expressed concern that OECD appeared to be heading toward creating a test without having had a thorough debate among its members about its purposes.

AHELO -- which has been portrayed as a higher education version of OECD's Program for International Student Assessment (PISA), a worldwide comparative measure of what 15-year-olds know and can do academically -- has been an intriguing and controversial prospect from its very beginnings. The administration of President George W. Bush was uninterested in the test, but the Obama administration has largely embraced it.

From the start, OECD officials have cited many possible uses for AHELO -- helping universities better assess and improve their instruction, students make better choices about where they belong, policy makers hold higher education institutions accountable, and employers gauge the skills of the graduates they hire.

But while some college leaders (and officials in some countries, such as Finland, that might expect to fare well if the tests became fodder for rankings) have welcomed the prospect of the release of institutional and country-level data that would make comparisons possible, college leaders in many of the largest and most diverse countries, like the United States, dread the idea of introducing yet more data that might be used, in an oversimplified way, to rate and rank countries with drastically different populations and approaches to postsecondary education.

"We hold strongly the view that it is possible to measure outcomes and enhance quality without promoting standardization of educational programs and homogenization of institutional missions," the U.S., Canadian and European campus groups wrote in their letter. "We hope that the OECD joins us in this view."

Roseveare said at this week's session on AHELO that OECD's higher education officials had "a lot of material to digest" from the feasibility study and that a "period of reflection" would help OECD's education policy committee decide whether to go ahead with a "full-scale AHELO" and, if so, how the test should be used. The issue is to be discussed at a meeting of the IMHE governing board -- on which the letter writers sit, but in the minority -- in March in Utrecht, the Netherlands. (Note: This article has been updated from an earlier version to correct information about who will make decisions about AHELO's future.)

Roseveare and Tremblay noted that the version of AHELO explored in the feasibility study had been focused at the institutional level, so the data "cannot be aggregated to any country level value," Tremblay said, making impossible the national comparisons about which the Western groups expressed concern in their letter. Asked in an interview if that meant a full-scale AHELO might also be crafted to avoid national comparisons, Roseveare said: "We haven't had that debate yet."

In an update about AHELO published in July, OECD officials conceded that the feasibility study had "brought to the surface a number of important broader questions about measuring learning outcomes that must be addressed before a full AHELO survey could be launched. Among them: "Is the purpose to measure the level of competency achieved or to measure the value added from attending a specific higher education institution?" "Who or what is really being assessed -- the students or their higher education institutions?"

Those questions are so "fundamental," said David Sweeney, director for research, innovation and skills at the Higher Education Funding Council for England, that a discussion informed by the results of a feasibility study that focused largely on methodology seems unlikely to answer those hard questions. "What," he asked Roseveare, "is the forum for discussing what the project is really for?"

Those issues and more will be debated at the conference in Utrecht, she said. "This is an interactive process."