A Dent in the Data

For anyone looking for signs of the decline of American higher education, the annual statistics published by the Organisation for Economic Co-operation and Development have become a standard building block to make that case. The data -- particularly those showing the comparative rate of educational attainment by 25- to 34-year-olds in the U.S and elsewhere -- are regularly cited to show the nation's failure to keep pace as other developed nations that long trailed the U.S. ramp up their higher ed systems.

As the OECD numbers have gained currency, some researchers and statisticians have warned against overdependence on them, citing questions about the international agency's methodology. At the same time, American college leaders, recognizing that the data are being used widely, have been digging into the numbers to try to understand them better and, where possible, look for ways to learn from what they say about the United States and its peers/competitors.

It was in one such quest that researchers at the American Council on Education, higher ed's chief coordinating group, discovered that the OECD's latest data contain a significant error in the U.S. attainment numbers, and perhaps those of other countries as well. OECD officials have acknowledged the problem in the American statistics and expressed concern about others as well.

And while the errors don't radically alter the picture of potential problems in the educational attainment among younger U.S. citizens, they do point to the dangers of overdependence on any single source of data to make sweeping, and potentially oversimplified, generalizations about public policy matters.

The numbers in question, which examine the proportion of citizens of various countries that have received a "tertiary" (or postsecondary, in U.S. parlance) credential, have been widely cited in the increasingly common critiques of the United States' ability to produce enough educated citizens to compete in the global economy. Margaret Spellings' Commission on the Future of Higher Education leaned heavily on the OECD data, as have more recent reports like “Measuring Up 2008” and the College Board’s “Coming to Our Senses.”

The data most commonly cited to show the decline of the U.S. higher education system examine the rates at which citizens of different age groups in different countries attain a postsecondary degree of some kind. The United States has long been among the world's leaders in ensuring access to higher education for its people, and the country still fares well in rankings for older citizens. But the OECD's 2008 data (which cover outcomes from 2006) for the proportion of 25- to 34-year-olds who have a postsecondary (or tertiary) degree show the U.S. falling to 10th among the 30 OECD countries, with 39 percent of citizens that age having some kind of degree.

Jacqueline E. King, assistant vice president at the American Council on Education's Center for Policy Analysis, is among the higher education researchers who have been digging into the OECD numbers to try to understand them better; among other things, she noted, ACE is preparing to undertake a study looking at how Canada has managed to give such a large proportion of its citizens "sub-baccalaureate" degrees, a category in which it ranks tops in the OECD statistics.

It was in analyzing the OECD numbers that King and her staff noticed that they showed the U.S. as having dropped to 19th, from 10th in 2005, in the proportion of citizens with a sub-baccalaureate (or, in the United States' case, associate) degrees. The steep decline, from 9 percent to 5 percent of citizens, was accompanied in the OECD data by an equally sharp (and equally unlikely) rise in the proportion of Americans with a baccalaureate degree, to 35 from 30 percent, and to 2nd place from 6th place. (A spreadsheet comparing the 2006 and 2005 numbers in various categories can be found here.)

After ACE officials brought the issue to the attention of OECD, the international group admitted that its researchers had "mistakenly categorized those academic associate degrees" as baccalaureate degrees, King said. In an interview Friday, Michael Davidson, a senior analyst in OECD's education directorate, acknowledged that a "mistake was made" in the mapping of U.S. programs to international classifications, and said that OECD was at work correcting the data.

As ACE officials dug further into the numbers, they noticed several other apparent discrepancies, as well. The OECD data for 2008 show New Zealand, for example, increasing the proportion of its 25- to 34-year-old citizens with sub-baccalaureate degrees to 14 from 5 percent, and the proportion of people in that age group with baccalaureate degrees to 30 from 26 percent. Over all, New Zealand shot up to 4th in postsecondary attainment in that age group in 2006, from 18th in 2005.

Several other countries showed multiple point changes between 2005 and 2006, which Davidson said in the interview Friday would be surprising under normal circumstances, given how difficult it is to achieve significant movement quickly in most countries. He said the OECD would review the data for other countries to ensure that the "glitch" that occurred in the United States numbers was not replicated for other nations.

King, of ACE, said her organization was not questioning the OECD numbers to try to pretend that there are not serious issues in postsecondary attainment in the United States, or to soft-pedal the extent of the problem. But these data "have gained a lot of currency," she said, and "you need to unpack [them] if you're really going to understand U.S. performance compared to other countries. We need to make sure the data are right so we in the analytical community can figure out what the differences really are, and work on how to improve what we're doing."