Joining the Law School Rankings Game

In the highly competitive worlds of law school admissions and faculty recruitment, it often seems as if the Lake Wobegon effect is in full force. On their Web sites and in the other marketing materials that law schools distribute to raise their profiles -- sometimes derided as "law porn" -- virtually every law school boasts of having a faculty made up of stellar scholars, brilliant teachers and selfless public servants. "We continue to add depth to our already diverse and multifaceted faculty -- excellent teachers whose high-quality research impacts leading academic and public policy issues," reads the Web site of Northwestern University's law school.

"Columbia Law School is justifiably world renowned as a leader in scholarly research and a trailblazer in the development and application of legal theories and principles," Columbia University says on its law school's faculty page. "In both traditional and emerging fields of law, Columbia professors are at the forefront of developing and interpreting legal issues and precedents of great consequence to society. But the Law School's overriding commitment continues to be as a teaching institution."

But how are applicants -- for admission and/or jobs -- to know whether the schools are living up to their promises on faculty quality, that all-important indicator of the institutions' overall quality? asks the Green Bag, which describes itself as "an entertaining journal of law." Consider some potential sources of such information. The Association of American Law Schools and the American Bar Association, both of which have law schools as their members, "appear to be committed to obfuscation" and avoid qualitative assessment of law schools at all costs, the Green Bag argues. And while the "void has been filled in part" by U.S. News & World Report, the only national journalistic publication that now ranks law schools, its ranking virtually ignores questions of faculty quality in its criteria, members, focusing instead on student-faculty ratio, spending on staff (including faculty) and peer assessments by other law school officials.

The Green Bag plans to step into that breach, the journal announces in an editorial in its forthcoming issue. Starting this spring, it will begin work on the "Deadwood Report," which it envisions being an annual assessment of "whether faculty members do the work that the law schools say they do." The journal acknowledges that the ranking will provide "rough and admittedly partial" measures of law school faculty quality, but posits that by being transparent (it will disclose the sources of its data and how it derives its numbers and rankings from those data), and by bringing more information into public view, "it will help law school applicants make better decisions about where to study or work.... We are trying to do some good here." (The editors have an ulterior motive, too: compelling law schools to make public better information about their operations -- more on that later.)

What exactly will the Deadwood Report measure? Law schools, the editors write, "generally hold themselves out as institutions led by faculties whose members are committed to teaching, scholarship, and service." They argue that the best teachers tend to be active scholars and vice versa, "and all the best lawyers of every stripe engage in service for the public good.... Evidence of the law schools' commitment to this view is reflected in the practically universal requirement of high achievement in all three areas for tenure. And so we should be able to say with some confidence that a good law school will have a faculty consisting of hard-working teacher-scholar-humanitarians," the Green Bag editorial says.

"The Deadwood Report will simply test the accuracy of that picture," the journal's editors write. "Our focus will be on the most dully objective of measures: whether the work is being done -- whether each law school faculty member is teaching courses, publishing scholarly works, and performing pro bono service." (The journal plans to start with teaching and research, turning only eventually to service, and notes that it does not plan -- "at least not yet" -- to answer what it calls the "trickier and more entertaining subjective questions: whether the teaching is effective, whether the scholarship is sound, whether the service is in the public interest.")

The Green Bag's method for answering those questions sounds like it will be painstaking. Its staff (editor Ross E. Davies and research assistants) plan to:

• Download a law school's faculty Web pages, course catalogs, and publications lists.
• Compile the data, with an emphasis on recent scholarship and recent teaching ("A school whose faculty is heavy with people who used to be active might do well in a citation or reputation study, but it will do poorly in the Deadwood Report").
• Analyze the data, using a still-to-be-finalized process of "sorting and weighting." Basic principles: "We are interested in well-rounded, activie faculty members, and so we will give more weight to the moderately active teacher-writer than to the hyper-writer who neglects teaching or to the hyper-teacher who neglects writing." Schools will also be rewarded for having well-rounded faculties, rather than a handful who are big-time scholars and a bunch of others who aren't.
• Send each school's dean his or her institution's preliminary results for a chance to correct inaccuracies.
• Correct the errors and publish the results.

The journal's editors offer some advice to law school deans, which offers additional evidence about their motives in joining the rankings game. Keep your Web sites up to date, since that is where all of the rankings' information will come from. "This seems reasonable to us because your Web site is surely where most applicants and other inquisitive people go for information about your law school. If a school cannot be bothered to provide accurate information about the teaching, scholarship and service of its own faculty on its own Web site, it deserves to be haunted by any inaccuracies."

"Puffery is double-edged," the Green Bag warns. If a law school's "faculty" page offers a long list of names, the journal's editors will include and assess them all in the school's "deadwood" numbers. "Inflated denominators will not be helpful to you," the journal's editorial says. "If you have employees who are employed to teach but not to write, or to write but not to teach," or who once did one or both but no longer do, or who are on leave, "you might be well-served -- and people visiting your Web site would certainly be better-informed -- if you moved those folks off your list of "Faculty" and onto lists labeled, perhaps, "Instructors" and "Researchers" and "Emeriti" and "Administrators" and "On Leave." (Visiting instructors will be treated differently.)

While they generally swear off assessments of quality, the editors make clear that they will make some judgments. "Teaching Property or Torts or Individual Tax to an auditorium filled with students is not the same as co-teaching a half-semester seminar on a highly specialized topic with three colleagues, a weekly guest speaker, and enrollment limited to 12. Do not expect us to give them the same weight." And "we will be taking account of scholarly books and articles in scholarly journals. Not novels. Not editorials, even if they appear in The New York Times or The Wall Street Journal," unless "we find on your Web site official regulations indicating that for tenure purposes your law school equates works of fiction, letters to the editor, and the like with conventional works of scholarship, and if further inquiry reveals that your school has actually awarded tenure on the basis of such publications..."

And, potentially controversially: "Works appearing in organs published by your school or your students, or on which a member of your faculty serves as an editor or in some similar capacity, do not count. The pressure to make publication decisions on grounds other than scholarly merit is too great, especially when relationships between students and teachers ... are in play." ("In that same spirit," the editors note, "we will not count anything published by the Green Bag, not because we do not publish scholarly works, but because we are wed to Caesar for this project.") This criterion would presumably restrict professors from getting credit in the rankings for publications even in top journals that are published by their own institutions.

Reaction From the Field

Many college officials might -- and do -- argue that the last thing the world needs is another method of ranking higher education institutions. At the same time, all of higher education is under pressure, from politicians and policy makers asserting that they are speaking on behalf of students and families, to be more transparent about their operations. Given those trends, it's not surprising, perhaps, that the Green Bag's planned entrance into the rankings field drew a wide range of reactions from the rankers and those who would be ranked.

Robert Morse, who runs U.S. News's law school and other college rankings operations, said the magazine "welcomes other people into the field to do law school rankings or assessments of any kind." He questioned the Green Bag's focus on measuring law schools by the productivity of their faculty, since "it's not clear that that correlates to being good for law school students." He added: "We obviously use much broader criteria. We think an institution is more than the sum of its faculty."

Brian Leiter, a law professor at the University of Texas at Austin whose own rankings of law school quality began as an alternative to U.S. News , said he generally agreed with the Green Bag's criticism of the current sources of information about law school performance and "the value of alternative sources of comparative assessments." He said he believed the journal's definition of "service" -- doing work in the community or that serves society -- is "out of whack" with the type of service typically rewarded by law schools, and suggested that the Green Bag bag service altogether as a criterion.

"I also think the editors are setting themselves up for a world of grief from the faculties deemed to have lots of 'deadwood' and the individual faculty so classified," Leiter added via e-mail. "I would have recommended a more delicate name for the undertaking! On the other hand, assuming the editors are prepared for the backlash -- most people do hate to be evaluated, especially in public -- the report may perform a real service for deans trying to change institutional cultures."

Initial reactions from law deans was mixed. Carl C. Monk, executive director of the Association of American Law Schools, said the group supported any effort designed to "provide even more consumer information" to the public. "The more factual information there is, the better, so to the extent that they are revealing factual information that is not currently revealed, then that will be benefical," he said. "Unfortunately," Monk said, the Green Bag plans to go well beyond merely providing more information to the public, and to weight the data using "their own subjective measures," such as by excluding papers published in a law school's own journals, among other factors. "To automatically exclude such a piece immediately introduces an element of subjectivity," he said. "I understand their reasoning, but it does not apply across the board in the way they suggest, and it means that they're placing their own value on the quantity and quality of the publication in which something is published."

For that and other reasons, Monk said, "at at this stage I am very skeptical about anything that they will publish in the way of rankings."

As a law school dean who generally favors rankings, David Van Zandt, dean of Northwestern's law school, acknowledges that he is a bit out of step with most of his colleagues. "But I believe we're in a business that has consumers paying an awful lot of money for what we provide them," and that schools have a duty to show their value to the public. "Rankings are one way to do that," Van Zandt said, and they encourage organizations to be better. Business schools are much more prone to be ranked than are law schools, in different ways and through different lenses, and having more independent organizations produce their own law school rankings would encourage law schools to compete and innovate in different ways.

In that way, he generally welcomed the Green Bag's effort. He, too, took issue with certain of the journal's emphases, notably its premise that all faculty should be the same, scholars and teachers and providers of service, which Van Zandt said would "reinforce a status quo" that could impede law schools from encouraging some faculty to excel in teaching and others in research, a direction in which he said Northwestern has moved. To excel, he said, "we think you have to have a group of specialized people who are specialized in different things."

Davies, the Green Bag's editor and a law professor at George Mason University, said that he appreciated the critiques of Van Zandt and the others, and that the journal, rather than reinforcing the status quo, was instead trying to gauge reality as most law schools like to portray it. "The message out of the law schools is generally the same. All our faculty are the same: Each one is an active scholar, an active teacher, and committed to law in the service of the public. "All the Green Bag is tring to do is measure the accuracy of that portrayal -- examine the real world, add one more little slice of data to these decisions people have to make. We have 180-190 institutions saying, We are bastions of teaching, scholarship and service. The Deadwood Report will say, Okay, Professor A sort of fits that. Professor B sort of fits that. Professor C really fits that. Professor D only fits one third of that. Professor E is AWOL.

"We just hope to open things up."

As for the title, which the journal's editorial calls "only nominally pejorative," Davies insists that it is "not supposed to be mean." "It is supposed to be constructive and informative, and if somebody can come up with a word other than 'deadwood' that gets at the concerns that this study is supposed to be probing, we’d love to hear it.

"But when you say 'deadwood,' people know what you mean."