What Is Merit?

LOS ANGELES -- After a morning here in which admissions leaders and legal experts discussed strategies for colleges to look beyond the grades and test scores of applicants, Art Coleman said that it was time to acknowledge the "proverbial elephant in the room." That's the issue of merit.

Coleman is a lawyer who has worked with numerous colleges and higher education groups to craft admissions policies that promote diversity and can also survive legal challenges. And he is sympathetic to the strategies discussed here, and to the idea that diversity is important to higher education, and that colleges have good reason to look beyond a formula of test scores and grades. But he said that if colleges fail to talk about merit and what it means, they are likely to lose the battle (in courts and public opinion) for the way they seek to diversify their classes.

He noted that the University of Michigan won the last Supreme Court battle over affirmative action in higher education, in 2003, only to have that win "wiped out" in 2006, when Michigan voters barred the consideration of race in admissions -- the very policy for which the university built a broad coalition to back its legal case. Coleman said that the only counties that voted to preserve the consideration of race in higher education that year were those had colleges in them. "There is a fundamental disconnect between the ivory tower and Main Street on these issues," he said.

The conference here -- "Attributes That Matter: Beyond the Usual in College Admission and Success" -- was organized by the University of Southern California Center for Enrollment Research, Policy and Practice. Presenters and those in the audience include admissions directors from institutions nationwide, along with counselors from some high schools, and the looming Supreme Court decision on the admissions policies of the University of Texas at Austin was on everyone's mind. Many fear that a decision against Texas could limit the way colleges consider race and ethnicity in admissions, and so interest was high in new strategies colleges could use to promote diversity.

Among the themes today:

• Numerous approaches exist to consider "non-cognitive attributes" that may improve the quality and diversity of the student body.
• More colleges are embracing those approaches.
• Colleges that have done so are reporting mixed experiences. And one of the examples most often cited -- Oregon State University -- is regrouping after failing to find some of the successes it was hoping for with its plan.
• The major players in standardized testing are voicing support (to a limit) for discussions of these alternative approaches, while some are also noting limitations of the approach. (The College Board was among the conference sponsors.)
• It may be easier to agree that merit is central to the debate than on what constitutes merit.

'We Need to Look at Something Else'

Thursday's sessions opened with William E. Sedlacek, a professor emeritus of education at the University of Maryland at College Park who is considered one of the leaders in promoting the idea that "we need to look for something else" besides grades and test scores. He described his work demonstrating that colleges can evaluate students based on qualities such as leadership, creativity, a realistic self-appraisal of skills, a sense of how to work the system, the ability to overcome adversity and others.

The audience here seemed to be in philosophical agreement with Sedlacek; several institutions here have hired him to help with admissions strategies. But there was one point of hesitation. Sedlacek described how colleges could mix and match, taking some of his ideas for testing non-cognitive attributes but not others, using his ideas with or without standard measures and so forth -- and without having to pay him a penny. He has intentionally not sought to copyright his strategies or to have colleges pay to license them.

But he received questions about those very characteristics: Why hadn't he created a national system for evaluating students? Privately between sessions, several admissions directors who said that they agreed with Sedlacek also said that they would have a tough time adopting his ideas -- at least if they meant admitting some students with lower grades and test scores than those currently admitted. Some said they needed nationally normed systems so that they wouldn't be hurt in rankings (even if they dislike the rankings).

Many of the questions for Sedlacek were on how to "operationalize" the idea of non-cognitive measurements.

Oregon State University has been considered a leader in the field, and officials from the university have periodically credited its "Insight Resume" Oregon State doesn't use the accents we do in resume -sj-- in which students respond to short questions about leadership, creativity, dealing with adversity, community service and other topics -- with attracting diverse students. For each category, an essay is read and receives a numeric score of 1 to 3 (3 being highest). The scores are assigned without regard to an applicant's other materials, so the evaluator doesn't know which applicants have great grades or are from a minority group or anything else about them.

Noah Buckley, director of admissions, described the system and also why it is no longer used as a major portion of evaluating all applicants.

A study of the first class on which the system was used (those who entered in 2004) showed great promise. with students scoring high on the alternative measures showing a greater chance of being retained and graduating. But that evidence didn't last when subsequent classes were identified. Officials could not document similar impacts on retention or graduation. So Oregon State has scaled back the way it uses these tools.

The university remains "very invested" in the idea, Buckley said. But high school grades remain the most predictive admissions criteria for all students, he said.

For students with high school grade-point averages of below 3.0 (normally a group that would be rejected), Oregon State uses the Insight Resume to identify "diamonds in the rough." For those with G.P.A.s over 3.75, the Insight Resume is a factor in deciding to whom to offer scholarships. For those in between, the system is used in large part to look for "red flags." So Buckley said some applicants' answers may indicate that they pose a safety threat. A few have indicated that they don't want to attend a diverse institution, and Buckley said they are rejected because the university's diversity suggests that they would not be a good match.

Unexpected issues have also come up. Buckley said that, over the years, some students have described overcoming child abuse. This year, the university just received such an application, but Oregon recently enacted a law that requires any state employee to report any child abuse they hear about if they come into contact with an abuse victim. Buckley said that he is not sure if an admissions application was intended to constitute "contact," or if this applicant considered the application a way to report abuse, but he said this applicant's essay may now be covered by the law.

"We're now at a fork in the road" in deciding how to use the system, he said.

Jon Boeckenstedt, who leads undergraduate admissions at DePaul University, said that institution has used a similar system, but only for those who opt not to submit the SAT or ACT. He said DePaul has numerous groups it wants to see more of at the university -- minority students to be sure, but also low-income students, first generation students and graduates of Chicago public schools.

He said that students with higher scores in DePaul's system (known as DIAMOND for Developing Insight for Admission Through the Mining of Non-traditional Data) were, on average, more likely to be retained that similar students with lower scores. And he said that the boost was evident across socioeconomic groups. So Boeckenstedt said he believes this approach does help identify diverse talent that might otherwise be missed.

But he added that the use of DIAMOND or similar measures "is not the be all and end all," and that he too believes high school grades are the best way to predict college success.

Boeckenstedt also referred to the good fortune he has to work at DePaul, where he said he has been supported (and encouraged) for using this system, even if it means some applicants are admitted with lower ACT scores than others. Unlike many admissions deans, he said, "I have never in 10 years heard a single mention from anyone about raising the test score average.” But, he added, "I get lots of questions about how many freshmen are on Pell."

Calls for Consistency

Officials who were here from testing companies didn't dispute the ideas behind non-cognitive admissions approaches, but didn't embrace the trend either. Steve Kappler, assistant vice president of the ACT, said his organization's research does see patterns in student "academic behavioral readiness" in junior high school and high school, and later academic success. He said that there is evidence that students who feel connected to school, whose families are involved with school, who participate in extracurricular activities, and who see academics as a key to advancement do, on average, perform better over time.

But he said that a direct relationship can be seen between succeeding in college preparatory courses in high school, ACT scores, and freshman grades in college. When someone in the audience asked about the impact of test prep, he said, "take the core curriculum." That is the way to get higher ACT scores, he said.

Kappler also spoke about grade inflation, saying that it is so widespread in high school that high school grades are much more predictive if combined with an ACT score than examined alone.

Jim Montoya, vice president of the College Board, watched the various sessions. In an interview, he said he wasn't bothered in the least by the interest in tools that go beyond tests such as the board's SAT.

Montoya stressed that the College Board has always argued that the SAT should not be used alone, but as part of a broader admissions review. So he said it was natural for colleges to explore ways to identify talent that might not be evident from test scores or grades. He said that he heard in the questions here a desire for "a more systematic way" and for greater "consistency" in how colleges try to evaluate non-cognitive qualities.

The Law School Admissions Test was subjected to a lengthy, critical analysis by Sheldon Zedeck, a professor of psychology at the University of California at Berkeley. He described how Berkeley's law school -- concerned that the LSAT primarily identifies non-minority talent -- has been exploring qualities associated with good lawyers and ways to measure those qualities, such as analysis and reasoning, problem solving, listening, communicating, engagement and integrity.

The results so far suggest, Zedeck said, that there is a negative correlation between getting a high LSAT score and eventually having a majority of the qualities that make one a good lawyer.

Via e-mail, Wendy Margolis, a spokeswoman for the Law School Admission Council, who was not here today, said that the council has not seen "anything that suggests a negative correlation." But she added that "it has not been the LSAT’s job to predict success in the profession," but rather to predict academic success in law school. She also said that it would be "very surprising if high-level reading and verbal reasoning skills (which is what the LSAT measures) don’t correlate with success as a lawyer (however that’s defined)."

Defining Merit

What much of the discussion came down to was, as Coleman argued, what constitutes merit.

He said that when colleges fail to define merit in ways that can be understood and demonstrated, they lose court cases and the support of the public. For instance, he cited the University of California v. Bakke case, in which the Supreme Court in 1978 upheld the consideration of race and ethnicity, but shot down programs that set aside places for applicants from minority groups.

The University of California at Davis medical school -- whose admissions policies were at dispute -- had a rule at the time that it would not interview applicants whose college G.P.A.s were below 2.5. Yet evidence was presented in the case that Davis filled the minority slots, at times, with students with G.P.A.s as low as 2.11.

Coleman noted that those minority applicants with the low grades actually went on to become successful physicians. And he said that their success raises questions about whether the 2.5 limit was a good one. But, he said, a university can't declare as official policy that it believes that, to have merit, you must have a G.P.A. higher than 2.5 and then defend the admission of students who don't meet that criteria.

When colleges don't think through all the messages of their admissions policies, he said, "they deserve to lose" their cases -- even if their intentions are good ones. "Live by the score, die by the score," he said.

The argument put forth in just about every legal challenge to affirmative action, he said, is that merit is defined by grades and test scores. Colleges need a public discussion about why that's not the case, he said. Colleges that want to use nontraditional measures need to show that they have thought them through, he said, that they are only admitting students who can succeed, and that they have evidence linking their policies to patterns of enrollment and graduation.

College leaders need to be willing to "go out of their comfort zones" and talk not only about "the process" but about the ideals behind their quest for diversity, he said. Medical schools, he said, have done a good job lately of reminding the public that they are admitting not just students but "future physicians," so knowing whether some may be more likely than others to embrace community medicine, or have good bedside manner, or be able to communicate to a diverse group of patients makes sense, he said.

Richard Sander, a law professor at the University of California at Los Angeles, challenged Coleman, saying that colleges and law schools that say they are using nontraditional criteria are really just admitting black and Latino applicants with lower test scores than white or Asian applicants have. He said that he doesn't oppose the consideration of nontraditional criteria, but that colleges need to apply such measures equally, not just to admit minority applicants.

But Coleman said that colleges are considering other factors, and have primarily failed to explain why. He said that if colleges had open discussions about why diversity matters, they might just gain support for considering factors beyond grades and test scores. Coleman gave the example of an orchestra. A conductor holds auditions for four new musicians to join the orchestra and 30 musicians try out. The top four -- based purely on performance -- all are oboists. Would anyone, Coleman said, expect the conductor to just take on those four?