The Great Mismatch

Doctoral education in the United States has changed rapidly over the last 30 years, with increasing specialization and the emergence of new sub-fields for graduate study. Depending on the nature and size of a university, some of these fields and sub-disciplines fit into traditional academic departments, while others demand their own departments or even colleges. Examples of the former abound, such as post-colonial studies, which may often find a comfortable home in an English or comparative literature department. In the latter category are fields like criminal justice, public policy, social work and nano-scale science and engineering -- highly-developed fields that attract increasingly large numbers of students and significant government and foundation funding.

We live within an academic marketplace of ideas, and the best institutions respond to the emergence of new areas of inquiry with vigor. Indeed, research universities can be judged by their ability to recognize and institutionalize new areas and disciplines, supporting excellence within them and nurturing their growth. Scholars typically lead administrators in these efforts, writing books that outline possible boundaries of a new field, establishing journals to define the area, or gathering colleagues for forward-looking conferences intended to advance cutting-edge approaches and methods.

From our standpoint, the more fields and defined areas of doctoral study, the better:  Formal establishment of these areas is typically the result of tremendously high student and scholarly demand. New areas of study are also the result of significant investments by universities and, in the case of public institutions, taxpayer dollars. Not surprisingly state officials and the public are eager for an accounting of how their investments stack up against others. This issue becomes all the more important when, as is true with the National Research Council ratings project, participating public as well as private institutions must pay to be part of the study.

Given that pushing the research envelope is one of the central tenets of any great university, it seems ironic that a survey designed to evaluate the quality and breadth of research would leave so much of our nation’s research untouched. Despite the imagination, interdisciplinarity, and fluidity one finds across the academy in recognizing emerging fields, our most prominent rating system -- the NRC Assessment of Research-Doctorate Programs -- has not responded, and in fact has resisted the change we see around us.

This fall the NRC released its new taxonomy, listing the fields that would be assessed and those that would not be studied. The new taxonomy reflects our worst fears for the assessment of Ph.D. programs: It fails to recognize a large number of thriving and vitally important fields where some of the most talented researchers in the world can be found. Among these fields are criminal justice, public administration and policy, social work, information science, gender studies, education, and public health. We have expressed our strong objections about these exclusions to Ralph J. Cicerone, president of the National Academy of Sciences, and to Charlotte Kuh, study director for the NRC Assessment. We have received no response, and other academic leaders have been treated with the same disregard when they have challenged plans for the new assessment. Such behavior seems especially problematic given the importance of the NRC study for institutions and researchers.

Placing fields like gender studies and information studies in the new, nebulous "emerging fields" category -- fields that will not be rated -- does not solve the problem in the least, but simply steers important scholarly endeavors in a giant black box. The justification of the taxonomy boldly notes that "emerging areas of study may be transitory," hence it is risky to evaluate them with the same rigor used for other fields. From what we can discern at least, information science, the study of race, of ethnicity, of sexuality, and gender have already emerged, and have profoundly changed the academy for the better. We imagine that scholars in these fields are not transitory in the least: A large number of them hold endowed chairs, run centers, manage departments, edit journals, lead foundations and run major institutions.

To make matters even more painful for us, for our faculty, and for colleagues around the nation, the National Academies recently asked for our financial support for the project -- a contribution of $20,000 for larger research universities like our own. We felt compelled to pay the price, but we did so reluctantly and over the strong objections of leading scholars on our campus.

One can critique numerous aspects of the NRC rating system, and a variety of leaders in higher education have done so quite eloquently for more than a decade since the last report. The data collection takes years to compile, and these data quickly become outdated as faculty members move and institutions change. We understand that the new system will involve online questionnaires and include a database that can be updated annually, and we appreciate the National Academies’ efforts in this regard. Another difficulty with earlier NRC studies has been the inclusion of reputational surveys. The forthcoming NRC study has promised to eliminate the reputational rankings from its rating system, and this, too, is an improvement. Among the worst offences of the system has been the bias toward large programs; much of the variance in previous ratings can be explained elegantly by department size (The 600-pound gorilla of a department, even with many unproductive scholars, will come out ahead of the smaller and higher quality programs). Perhaps the questionnaires planned for institutions and admitted-to-candidacy doctoral students in selected fields will help add a new dimension to program quality that will compensate in some programs for differences in size.

But these revisions, while potentially significant, only make the intentional and unexplained omission of major fields of knowledge, critical to the development of the academy, more inexplicable. Methodological change is not much of an advance if one is not measuring the right population of fields and disciplines. A social science parallel, from public opinion research, is relevant here:  You can refine a survey instrument all you like, sweating over question wording, order effects, and non-response to the survey. But if you are asking respondents about banal issues of little political import, why bother?

There is an even more troubling irony in the current effort. The NRC has chosen not to include “those fields for which much research is directed toward the improvement of practice,” such as Ph.D. programs in “social work, public policy, nursing, public health, business, architecture, criminology, kinesiology, and education.” This approach, of course, flies in the face of a recently-released report on “The Responsive Ph.D.” by the Woodrow Wilson National Fellowship Foundation. This report identifies the principle of “a cosmopolitan doctorate” as central to the future of the Ph.D. The report emphasizes that such a doctorate “will benefit enormously by a continuing interchange with the worlds beyond academia” and calls upon doctoral education to “open to the world and engage social challenges more generously.” The NRC assessment, by excluding so many well-established Ph.D. programs, will simply have the effect of reifying the status quo at research universities, instead of helping us respond boldly to the loud and chronic public call for an open and responsive academy.

The Taxonomy Committee argues that the task of evaluating research in these fields lies “beyond the capacity of the current or proposed methodology.” We do not accept this argument as valid, particularly given the proposed scope and expense of the projected NRC study. Further, the taxonomy displays no systematic logic with regard to which applied and interdisciplinary programs are included and which are excluded. Why include nutrition or pharmacology, clearly applied fields, but not criminal justice? Why is the study of sexuality not included while German linguistics and Latin American Literature both are? There is no decision rule in sight, and the taxonomy does not even come close to matching the current landscape of the academy. Perhaps if the NRC had retained the reputational measures, they might have been able to mount an argument about excluding particular fields. But, ironically, the new approach makes the taxonomy more distant from reality. It is removed from the marketplace of ideas, and excludes the voice of the scholarly community.

Apparently the NRC is not open to arguments like the ones above, and as a result, the ratings they will eventually produce will not reflect a great deal of the most important scholarship in higher education today. Not only will the final report have gaping holes, ignoring the work of thousands of scholars, but the NRC will also fail to recognize that interdisciplinary research with practical application matters immensely.

We predict that this next round of results will be received -- whenever it is complete -- as a dinosaur, an artifact of uneven logic and old-fashioned thinking about what constitutes true scholarly discovery. We are grateful that other assessment systems are appearing and regret that the NRC will spend over $5 million on a quickly outdated effort to assess graduate education. Thankfully, such short-sightedness will not stop our best scholars from developing new approaches, forging innovative fields, training hungry students, and changing the world for the better through their work. We call on the National Academies  – yet again -- to reconsider their taxonomy, so that leaders in higher education can demonstrate to our public officials that we are capable of evaluating the very research enterprise with which we have been entrusted.