Missing the Expected Impact

Many in higher education believe that test-optional admissions policies –- when colleges and universities do not require applicants to submit standardized test scores -- boost enrollment of low-income students and underrepresented minorities (African Americans, Latinos and Native Americans). A new study, however, challenges this conventional wisdom.

An article published last week in Educational Evaluation and Policy Analysis argues that a college’s adoption of test-optional policies does not increase the proportion of low-income and African-American, Latino and Native American students who enroll. Instead, such policies benefit the institution.

The study examined 180 selective liberal arts colleges, 32 of which had adopted test-optional policies between 1992 and 2010. It compared colleges with test-optional policies against colleges that required test scores.

The report found that between 1992 and 2010, test-optional institutions did not see any more gains in low-income enrollment (measured by the proportion of Pell Grant recipients) than test-requiring institutions did. Nor did test-optional institutions see larger gains, on average, in the proportion of black, Latino and Native American students they enrolled.

Andrew Belasco, one of the study’s authors, said the findings surprised him and his colleagues.

“We went into the study thinking that we would absolutely find that some of these policies would make a difference in terms of bringing underrepresented students to campus,” Belasco said. “These policies, at the end of the day, do little to promote greater access.”

The report speculates that test-optional policies may in fact “perpetuate stratification within the postsecondary sector,” because test-optional colleges, instead of relying on SAT scores, might assign more weight to advanced courses and extracurricular activities –- opportunities that are unequally distributed across socioeconomic groups.

Colleges, not students, are benefiting from test-optional policies, the study argues. Test-optional policies result in higher average reported test scores, presumably because students with scores below a college's average have no incentive to report them. An institution’s average SAT and ACT scores factor into college rankings systems, such as the U.S. News & World Report’s annual list.

Institutions also attract more applications after adopting a test-optional policy. This rise in applications can lead to a perceived increase in selectivity, because a college will admit a smaller share of applicants.

Test-optional policies are more popular at selective liberal arts colleges than at any other type of institution. Eleven have introduced such policies in the last two years. Hampshire College, which had been test-optional since it opened in 1970, announced Wednesday that it would become “test-blind.”

Belasco’s article contradicts a study published in February by William C. Hiss, a former dean of admissions at Bates College. (Bates has been test-optional since 1984.)

Hiss’s report analyzed data from 123,000 students at 33 test-optional institutions. The study included 20 private colleges and universities, six public institutions, five minority-serving institutions and two art institutes. He found that at the private and public institutions, enrolled students who had decided not to submit test scores were more likely to be minorities, women, Pell Grant recipients or first-generation students.

Hiss concluded: “The optional testing policies seem to be supporting increases in underrepresented minority students” at these colleges and universities.

Belasco noted that underrepresented minorities may indeed be more likely to withhold test scores, as Hiss’s study showed. But this correlation doesn’t establish a link between test-optional policies and minority enrollment.

“It could simply mean that minority/low-income students are applying (and enrolling) in comparable numbers, but submitting the standardized test scores at lower rates,” Belasco said in an email.

Bob Schaeffer, public education director of the National Center for Fair & Open Testing, a group that supports test-optional policies, said test-optional policies resulted in more diverse applicant pools. "That presumably makes it easier to choose a diverse class in all ways," he said.

Belasco said, in reference to Hiss's study, that "you can't equate applications with enrollment" -- especially given the financial hurdles that prevent some low-income applicants from attending college even if admitted.

Schaeffer said he expects to continue to see more institutions go test-optional despite Belasco’s findings. “Schools that go test-optional very rarely switch back because they see in their own data and the kids that apply that they are fulfilling their goals, which may include a range of diversities,” he added.

Jack Buckley, senior vice president for research for the College Board, said the study did not disprove individual institutions that think test-optional policies are improving diversity. The article instead shows that, on average, these policies have little effect on low-income and minority enrollment.

"In some cases an institution might think this policy is the cause of an increase" in minority enrollment, Buckley said. "It could be true that they’ve attained the objective they’re after, but this wasn’t the cause."

Buckley said the College Board had found similar results internally but hesitated to publish them for fear of appearing self-serving.

"It’s really great to see other folks doing research on this question,” Buckley said. “Every time we try to make a similar statement people question our motivations.”

Joseph Soares, a professor of sociology at Wake Forest who has written extensively in favor of test-optional admissions, said Belasco’s model would have been “cleaner” if the researchers had limited their sample to the top two tiers of selectivity. The study includes all liberal arts colleges identified by Barron’s Profile of American Colleges as “competitive,” “very competitive,” “highly competitive,” or “most competitive.”

Belasco contends, however, that if he had focused on the most selective institutions he would have neglected the bulk of colleges adopting test-optional policies. 

"Aside from Bates, Bowdoin and a handful of other 'most competitive' institutions, these policies are primarily adopted by colleges that are (slightly) less selective, which don't always attract a sufficient number of college applicants and which have a lot more to gain from moving up the college rankings (via more applications, lower acceptance rates and higher test scores)," he said in an email.

Soares also said the study’s coefficients of determination – the r-squared figures -- were unusually high. Coefficients of determination provide a measure of how much variation a study’s model explains. For example, Belasco’s r-squared figure of 0.917 says that 91.7 percent of variance in SAT scores among institutions can be explained by the variables incorporated into his model, such as admission rates and whether the institution has a test-optional policy.

The lowest r-squared figure the study reports is 0.846. Most coefficients of determination in educational research are in the 0.2 to 0.3 range, Soares said.

A high r-squared figure could suggest that a researcher is not controlling for all the right variables.

Belasco is the president of College Transitions, a company that guides students through the college-application process. He said his company’s involvement with standardized testing is limited to advising students on which tests to take to bolster admissions prospects. The company also advises students on test-optional institutions. did he answer Soares' criticism? -sj -- unfortunately I talked to Soares after I talked to Belasco -- I sent Belasco an email and am waiting to hear back