May 22

Non-Cognitive Qualities Join the GRE

A major criticism of standardized testing is that it typically fails to measure non-cognitive qualities. The Educational Testing Service, one of the major producers of standardized tests, is acknowledging this concern — and has decided after years of pilot projects and research to add a new, non-cognitive portion to the Graduate Record Exam, which is taken by 600,000 students a year, including most future Ph.D.’s.

Looking for a job?

See all 312 new postings

Browse all job listings:

Faculty: 2,352
Administrative: 2,117
Executive: 155
FEATURED EMPLOYERS

The GRE board and ETS have now agreed to start using the “Personal Potential Index,” and it will be a part of every general GRE starting in July 2009. While applicants to graduate programs that do not require the GRE may purchase the index separately, it will be bundled with the GRE such that everyone who needs to take that test — which is fairly standard in graduate admissions — will have access to the index. The addition may not only point to talents that applicants may be unable to demonstrate through GRE scores or college grades, it may encourage changes in the way faculty members write and consider evaluations.

In the index, three or four professors or supervisors — generally those who will also be writing letters of recommendation — will answer a series of questions about candidates’ non-cognitive skills in various areas, as well as a more general set of questions. Applicants will be rated on a scale of 1-5 on questions about their abilities in these six areas: knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity. Those filling out the forms would also be able to provide narrative answers on each of those areas.

While ETS and GRE officials have been talking about the idea of moving ahead with the index for several years, the formal decision to add the index is a sensitive one. Many of the strongest proponents of adding non-cognitive measures to the admissions process are strong critics of standardized testing in general. So in some sense, ETS is embracing ideas advocated by those who spend considerable time bashing ETS. Further, some faculty members are sensitive about the idea that their letters of recommendation might not already be the best tool for evaluating applicants’ personal qualities. ETS officials have been careful not to offend or burden professors, stressing that it would take about 15 minutes to complete an index, and changing the name of the index from its original moniker, the “Standardized Letter of Recommendation.”

ETS and the GRE have also made one significant change, based on pilot testing of the index. The original plans also called for those professors filling out the index to indicate whether they thought the applicants were suitable for the program they were applying to enter. David G. Payne, associate vice president of ETS for college and graduate programs, said that question was dropped because many of those doing the index may not know enough about the programs students are applying to, and out of concern that the question might hurt disadvantaged applicants who may in fact be better fits in graduate programs than people realize.

That is, of course, the premise behind the index. Payne said that once graduate schools use the index for a few years, and watch the performance of students who are admitted, long-held assumptions might change. For instance, graduate admissions committees that previously assumed that applicants needed a certain GRE score or a certain college grade-point average might find that students with lesser numerical scores but high ratings for resilience or creativity perform better in graduate school. If such findings are made, Payne said, graduate schools might find themselves more comfortable offering admission to a broader range of applicants.

That, Payne said, would benefit both graduate schools and those who want to enroll. “The board feels so strongly that including non-cognitive measures in graduate school admissions is needed,” he said. “We hope schools will see value in it and applicants will”

The PPI, as the index is known, may also appear on other tests. Payne said that ETS is in discussions with groups he identified only as “higher education organizations that provide information regarding student abilities” about using the PPI or the concept in conjunction with other tests.

ETS will not require the PPI to be used and applicants will decide — if the graduate programs to which they are applying don’t require it — whether to have professors fill it out. Payne said that in the first year it is used, ETS would be happy if 30-40 percent of GRE test-takers use the PPI. But Payne said that ETS is confident, based on extensive discussions with graduate officials, that the percentage will go up quickly, and could hit 90 percent.

Carol Lynch, a senior scholar at the Council of Graduate Schools and a member of the GRE board, said interest in PPI is high in part because of heightened activity by the graduate school group and others to identify the factors that help students finish doctoral or other graduate programs. In many cases, she said, graduate deans are finding that “the things that make for success are not just raw academic credentials.”

Payne said that the price of the GRE would increase “modestly” to cover expenses related to the PPI. Currently, the GRE costs $140 in the United States and $170 elsewhere. While final pricing has not been set, Payne said it would probably amount to an increase in the range of $10-15.

Historically, when major admissions tests add significant portions (such as the writing test for the SAT and the ACT), many applicants try to take the exam before the changes — or spend a lot of time on strategy for the revised exam. Payne said that ETS planned significant outreach to prospective graduate students and deans about the changes, in an effort to minimize anxiety. He stressed that the people who would fill out the PPI are the same professors students would already be approaching for letters of recommendation.

Michael Sullivan, director of Project 1000, a national program based at Arizona State University to help underrepresented minority students be admitted to graduate school, said he sees anecdotal evidence that PPI could “revolutionize” the way professors provide information to admissions committees. ETS has offered the PPI to Project 1000 participants.

Sullivan said many of these students are “very smart and very creative” and will succeed in graduate school, despite GRE scores that are below the averages for the programs to which they are applying. Frequently, he said, this is because English is not the students’ first language. Project 1000 offers assistance to these grad school applicants in many ways, so Sullivan said he couldn’t be sure of the impact of the PPI alone, but said that participants are doing “very well in admissions,” in part because of efforts like PPI that make the applicants “more than a GRE score.”

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, a frequent critic of ETS, said that with the testing service’s new initiatives, “it is difficult to differentiate between genuine attempts to improve the admissions process and calculated efforts to sell more products.” But he said that the PPI may have “elements of both motives.” He noted that ETS is trying to gain testing business and that any new feature — particularly one that answers critics of testing — could be helpful. ETS officials acknowledge that the PPI is part of their pitch to attract more business schools to the GRE — a major goal of ETS, and one on which it appears to be making some headway, since it lost the contract for the Graduate Management Admission Test.

But whatever the motives, Schaeffer said that adding the PPI was a real shift. “Unquestionably, it adds a non-test component to the graduate admissions process,” he said.

— Scott Jaschik

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

numbers

Even if all the claims about the important of non-cognitive abilities were true (and I doubt that being a team player has much to do with intellectual brilliance)—what’s the evidence that professors are in a position to evaluate a student’s non-cognitive abilities?

Marlena Corcoran, at 6:15 am EDT on May 22, 2008

rating on noncognitive qualities

Bravo to ETS! My wife and I have had, over 40 years, many graduate students who shone on GPA and GRE measures but were complete flops that never measured up to the potential we thought they should have, and many graduate students whose scores were mediocre but who turned out to be supberb students. The most poignant case was her department’s decision to give a fellowship to one applicant with relatively high GRE scores over another with relatively modest ones. The student with the fellowship couldn’t write grammatically, couldn’t finish experiments, was constantly distracted by outside interests, and never completed his PhD. The one with modest scores turned out to be the most knowledgeable, creative, brilliant student she ever had, and is now enjoying a postdoctoral fellowship at Harvard.

Richard Hull, at 6:45 am EDT on May 22, 2008

Will the Professors Be Paid?

As someone who writes a lot of recommendations for graduate schools (and reads them for graduate admissions), I’d like to know if ETS plans to pay the professors for filling out an instrument that THEY PLAN TO MAKE MONEY OFF OF. If not, this scheme is not only stupid, it’s outrageous. And have you ever read the checklist part of most graduate admission- letter of recommendation forms? They are completely non-discriminating and nearly worthless.

Brian A. Bremen, Associate Professor at The University of Texas at Austin, at 6:45 am EDT on May 22, 2008

Idiotic

This is a harebrained solution to a problem that may not exist. As anyone who has ever reviewed applications or written a letter of recommendation knows, the “ranking” questions used by some institutions’ recommendation forms are nearly useless. One professor’s 5 is another’s 3. I’m not a statistician, but I do know that in a small survey sample, there is no way to control for such variations. Besides, many professors refuse to answer those questions or just give high scores across the board, showing resistance to the whole concept of crude rankings of abilities. Students already submit faculty letters that give narratives of their cognitive and non-cognitive skills. I don’t see any way ETS’s numbers will improve the evaluation process. Just read the letters if you want to get to know an applicant’s personality.

J. Staines, Assistant Professor at CUNY, at 7:40 am EDT on May 22, 2008

“knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity”

In what sense is any of this “noncognitive?”

JBM, at 7:50 am EDT on May 22, 2008

Rate My Student

Oh, goody, finally: a chance for professors to fill out bubble sheets on our students, en masse, rating them on unquantifiable things about which we are unprepared and perhaps unqualified to judge them and which may or may not predict their long-term success as learners, such as their potential, their attitude, their joyousness in learning, and whether they made us feel professorial and beloved. All’s right with the world, at last.

Is there anyone here who hasn’t ever looked at one of those recommendation-sheet grids — “rate this student as Top 1%, Top 5%, Top 10%, or Really Pathetic in the following categories” — and found themselves sorely tempted to inflate the score for a student who’s merely *likeable* or one who has “struggled” in some way, whether or not we think he or she has the intellectual prowess being measured? Anyone who hasn’t looked at such a category — Personal Devotedness, say, or Experiential Acumen — and thought, “Geez, I only read a couple of her exams, how the hay am I supposed to judge her on this?”

If the student is likely to be better than her scores predict, she deserves a full narrative testifying to that fact; if a graduate program wants to find the best candidates rather than the best test-takers, they should demand and attend to evidence beyond GRE scores. Any attempt to take a short-cut will bring short-cut returns, and the test-makers will be the only ones who come out ahead.

ezry, at 7:55 am EDT on May 22, 2008

This is evil

This totally kills the point of the GRE. You want to get something anonymous that shows how people compare on a completely objective — even if flawed — measure. Now we’re going to see all cultural biases get doubled. Look, professors already write qualitative recommendations. They already give qualitative grades. This is makes it look scientific, but it’s not.

Predictions: Good looking students get higher scores.

Uncreative suckups will get higher scores than independent creative minded students.

Students of different ethnicity, gender, sexuality or faith than recommenders get lower scores.

Sexual and financial scandals will occur with professors receiving gifts in exchange for scores.

Professor grades will rapidly converge to say that every student is perfect.

urbanprof, at 8:35 am EDT on May 22, 2008

Wow...

This is perhaps the silliest idea that I’ve heard in a long time. Isn’t this exactly what letters of recommendation are for, without charging students more money?

Robert, PhD Student, at 9:40 am EDT on May 22, 2008

Letters of Recommendation

So let me see...

The ETS is adding a component to the GRE that takes about 15 minutes of a professor’s time to complete, for each student. ETS will then charge students for this service, presumably making a profit.

And why should the professors provide this free service to ETS, given that regular letters of recommendation will probably continue to be required in the foreseeable future?

I am already thinking of declining any requests from students to do this, adding, as it will, to the burden of preparing the letters of recommendation I regularly compose.

Nightmare scenario — unscrupulous faculty charging students to provide glowing appraisals to ETS.

The commercialization of providing references about a student’s preparedness for graduate study is fraught with potential problems.

Bruce Thyer, Professor at Florida State University, at 9:40 am EDT on May 22, 2008

This is a ridiculous idea. Though most of the professors would be objective when giving ratings and writing letters of recommendation, how one professor’s rating is comparable to another professor’s? I think the letters of recommendation already do the qualitative job, so there is no need to add this ridiculous index to the GRE test. Please save students’ money and professors’ time!

Equal, at 10:45 am EDT on May 22, 2008

While this new addition is open to a lot of subjectivity and requires students making a good impression on faculty in order to score well, it is no more flawed than the ridiculous standardized testing methods currently used to admit students into graduate programs. Right now students who are able to memorize and regurgitate knowledge are given high tests scores and admitted into graduate programs based on these useless skills.......meaning we end up with graduate programs that have great memorizers, but students who are poor critical thinkers and don’t have other valuable skills need for good research and leadership. Yes this addition would not make evaluation of students a perfectly tuned tool, but it would be better than the current disaster used to assess prospective graduate students.

PhD Student, at 10:50 am EDT on May 22, 2008

Were I still in academe, I’d flatly refuse to perform this illogical task. IMHO, the AAUP should lobby against this measure.

Bemused Former Prof, at 11:00 am EDT on May 22, 2008

And this differs from recommendation letters how....?

Perhaps some of this was addressed when they were testing it, but I’m having a hard time seeing how this is better than recommendation letters. I can see how some checklist might (or might not) make for a suitable summary of a letter, but the narrative gets to the meat of it. When I recommend a student for grad school, I talk about the work that the student did with me and offer examples to support my rating of the student. That’s more valuable than just “Initiative: 5. Creativity: 4.”

Might as well add scores for STR, DEX, CON, CHR, WIS, and INT while we’re at it...

Alex, Asst. Prof, at 11:10 am EDT on May 22, 2008

Keepers of The Gate/Keepers of The Flame

Now if you fellers could just get yourself a secret handshake, you’d be every bit as relevant as the Mystic Knights of The Sea the general public thinks you are already."Non-Cognitive Qualities,” indeed. Prepare to prove your credentials on that—as class action defendants. And good luck in your future, post-cognitive endeavors!

comatus, at 12:05 pm EDT on May 22, 2008

A good if slow start; not quite the right list

ETS’s largest problem in making this work will be to calibrate the inputs of the raters. As a group, instructors are not good at this and they don’t take guidance well. Analogous but less severe problems exist with supervisor ratings.That said, I can recall 1983 conversations with ETS’s Richard Peterson (a good friend for many years) in which I argued from what, even then, was considerable empirical evidence that at least half of the variance in the success outcome is accounted for by non-cognitive proficiencies (’qualities’ is the wrong construct and there are still strong cognitive components with affective dispositions dominating). These proficiencies can be learned and improved, primarily through horizontal learning processes that can be embedded into the activities of the learning objects. We at InterEd believe we know what these proficiencies are and, 25 years later, ETS has captured at least some of the ones that we have embedded as strands in learning objects since the 1980’s, and added a non-performer.

Would anyone care to offer their judgment as to which of the “ETS Six” is wrong (i.e., will not validate as a construct with predictive strength) and which are missing (at least two, we believe).Nonetheless, this is good, if painfully slow progress and should lead to better judgments.

Robert Tucker, President at InterEd, Inc., at 12:05 pm EDT on May 22, 2008

Well, this sounds like it will be about as useful as the GRE itself. That is, not at all. I second the comments of PhD Student.

Amy, Graduate Student, at 12:30 pm EDT on May 22, 2008

Step in the Right Direction

I, for one, am happy to see this development. We all know that IQ is only part of the equation when it comes to student success. Just pick up a book on emotional intelligence and see how the concept has been embraced by the business community for pragmatic reasons.

For those who balk at the concept of “non-cognitive qualities,” I would suggest you peruse Sedlacek’s Beyond the Big Test (which considers alternatives to the SAT/ACT) or other works on the topic. The gist is that qualities labeled as non-cognitive (as compared to cognitive qualities now assumed to be measured with standardized tests) are much better predictors of success. Qualities such as persistence, self-concept, and leadership determine how one uses his/her cognitive skills in the pursuit of an education (and life).

Nonetheless, it would appear that there are significant validity concerns to professor ratings in the proposed ETS system as raised by many others’ comments. It would seem a better idea for students to submit their own free-text responses to questions/situations and have professional evaluators rate student responses for each factor.

A number of undergraduate institutions are using such a system with great success. Seems reasonable to expect that the same kind of a system would work for graduate education. If nothing else, students applying to graduate school would have more robust experiences from which to draw promoting response variety and offering more opportunity to determine assessment validity.

Reid Kisling, Ph.D., Enrollment Services Professional, at 2:35 pm EDT on May 22, 2008

Worthless.

But I have to say I appreciated the reference Alex, in fact I volunteer to assign stats randomly to students. “He looks he has street smarts, I will give him Wisdom 12.” Then after we have the whole stats profile for a student, we put them in a cohort, or “party,” with other students. Then the Departmental Graduate Admissions Advisor will be profiled for their stats, and whichever party defeats the advisor (after much dice rolling as ensued,) will be the cohort admitted the following year. And I think we will charge students $10 for stat assignment, and $5 dice rolling fee (The students can save money if they fly to the department and roll their own dice).

I think this process may be as logical or more logical than the current GRE. And heck, we saved them $140!

I would suggest moving past the GRE and finding a better way to evaluate grad students. Although I do believe that a standardized test in the sciences may be somewhat a good idea, it is absolutely worthless in the Social Sciences and Humanities.

Barbarian Pete, at 3:00 pm EDT on May 22, 2008

Shameful

It is not shameful that the members of and aspirants to the professoriate make their living disseminating scientific knowledge but are, themselves, largely ignorant – and proudly so – of modern measurement science? So many of you teach and construct (largely invalid) assessments much as your great grand professors did. You go about your business, blissfully ignorant of the last 50 years advances in measurement and assessment. Most shameful is the fact that you whine like children at the prospects of having to do anything different rather than embracing new ideas and contributing to their refinement. I have no connection to ETS but I know them well. I can assure you that many exceptionally bright and deeply competent individuals have been involved in this project. Without doubt, the initial framing questions of ETS began substantially above the level of the comments expressed herein. I am not offering unquestioned support for this project. I am suggest that it is a step in the right direction and deserves positive engagement rather than the all-too-common remarks I see here.

Senior Professor, at 4:05 pm EDT on May 22, 2008

Senior Professor-

I get what you’re saying about doing things systematically and quantitatively and responding to research. However, what ETS is offering sounds suspiciously like a form for recommendation letters. Would it be shocking if a large company were trying to make a buck by rebranding something that already exists?

Yes, we should all be open to adopting innovations, but we should also be suspicious of people trying to make a buck off a gimmicky version of an established methodology (i.e. recommendation letters).

Alex, Asst. Prof., at 5:00 pm EDT on May 22, 2008

“The gist is that qualities labeled as non-cognitive (as compared to cognitive qualities now assumed to be measured with standardized tests) are much better predictors of success. Qualities such as persistence, self-concept, and leadership determine how one uses his/her cognitive skills in the pursuit of an education (and life).”

The label is irrelevant. The question is whether persistence, “self-concept,” and leadership are based in cognition. They clearly are.

JBM, at 5:35 pm EDT on May 22, 2008

Alex

Agreed. I don’t know whether this initiative is of ETS is dominated by their sales or product development functions, or if it is a healthy blend across the organization. We can, however, keep our focus on defining and tracking the metrics and see how well this new tool adds discriminating power of the kinds we are interested in.

ETS has a history of fairly compensating those who do work for them so I assume that will be worked out. Either way, I find it uninspiring that this is a strident first line of concern of the self-serving Mandarins. “Call the unions. . . alert the media. . . we overworked professors are being abused. . .” Give me a break! I was a professor for 20 years and the only difficult part was putting up with the whining of my least productive colleagues. Many of these people can’t assess anything much above chance level and, all too often, their assessments correlate negatively with the larger success indicators.

I believe there is no excuse for a member of the 2008 professoriate being so ignorant of measurement science. Would any of these complainers take there surgical needs to a physician who said, “Measurements. . . I don’t need no stinkin’ measurements beyond what I can calculate in my head!”

Senior Professor, at 6:55 pm EDT on May 22, 2008

This has gotta be some kind of sick joke... is it April 1?

Urbanprof has it exactly right. This is a moronic idea: it’ll be subject to exactly the same kinds of pressures as recommendation letters, plus ten times the cultural bias. This smacks of the “social” discrimination against Jews that the ivy league schools did (cf. that big New Yorker piece) and that people are starting to allege at lots of places w/r/t Asian-Americans.

Paul Gowder, at 6:55 pm EDT on May 22, 2008

All well and good for the “traditonal” grad student population... But what’s an “old fart” who wants to return to academics to do to get “top score” on this “new fangled” section of a “gotta take” test?

OLD grad student, at 9:20 pm EDT on May 22, 2008

In defense of GREs

In my experience GRE scores do a pretty good job of predicting whether or not a graduate student will be at least decent. Below a certain threshold you will find students doing pitiful work as TAs and as fledgling scholars. I wish my department paid more attention to GREs when they admitted people into the program.

I can believe GREs are not so good at explaining gradations of academic success, but I have to say they do a pretty good job of predicting academic failure.

Having said that, I agree this new assessment of noncognitive abilities seems like a pretty dumb idea for the same reasons that other commenters have given.

JoshTK, at 10:10 pm EDT on May 22, 2008

I agree with many of the concerns that have been raised on how these rating scales may be used differently by different raters and the potential for bias in the ratings.

Do they really expect faculty to be able to provide a rating of students’ “resilience” and “ethics and integrity"? What exactly does a resilient (or a non-resilient)student look like? It is also troubling to me that they are asking for ratings of a student’s ethics and integrity. How are faculty really supposed to know that for the majority of students they write recommendations for? Also, what are they really trying to measure with “ethics and integrity"? That the student didn’t cheat on any exams?

I don’t see what this will add to what already exists in recommendation letters. Why the need to quantify everything into scales and scores? Qualitative data can be a much richer source of information in this case.

Kim, Research Analyst, at 12:50 pm EDT on May 23, 2008

Welcome the new Non cognitive Qualities

Hi,

I’m very happy to know about this new section in the GRE. I’m an international student and I’m working very hard in order to meet the TOEFL score. As international student, the new section will help me to demonstrate more about my professional skills and qualities in order to demonstrate why am I a good candidate.

Thank you and well done for the new GRE,

Jazmin Nuno

Jazmin Nuno, at 2:00 pm EDT on May 23, 2008

If the need were truly serious...

My applause to all the doubters above. If the need and the risk of allowing bad apples into grad programs were truly serious enough to consider this pay-ETS-for-our-labor scheme, then grad programs would do better to adopt what medical schools did in the past 20 years: conduct live interviews in the admissions process. And even that cannot promise 100% success. There is no magic formula.

David, U Florida PhD 2003, at 2:15 pm EDT on May 23, 2008

GRE

This is one of the most asinine things I have heard in a long time. Is academia actually trying to doom itself to irrelevance? If institutions want that kind of input, require letters from those professors.

Jeff Pyle, at 5:15 pm EDT on May 23, 2008

The (ir)relevance of the GRE

In all seriousness, does anyone admit grad students based solely on the GRE? Or even primarily on the GRE? For me, and everyone I’ve talked to about it, the GRE is merely one (unimportant) measure in determining whether a student should be admitted. Letters of recommendation are another (also not particularly important). So are personal statements, research statements, CV, a sample of previous work, and so on. Personally, I always turn to the work sample first.

So, ETS can add whatever they want to the GRE, it will still rank last (usually not at all) in how I judge applicants.

The importance of the GRE, at 9:30 pm EDT on May 27, 2008

Jobs Related to Non-Cognitive Qualities Join the GRE

or search for jobs directly.

Assistant Director, TRIUM Program
New York University

The TRIUM Executive MBA program is the groundbreaking global MBA forged from an alliance of three of the world’s top ... see job

Associate Director, Doctoral Programs (112008)
Northeastern University

Northeastern University, founded in 1898 and located in Boston, is a private research university that is a leader in ... see job

Assistant Director — Graduate School of Arts and Science
New York University

New York University has an exceptional opportunity for an Assistant Director to manage the Graduate School of Arts and ... see job

Nonprofit Managment and Leadership Expertise
Portland State University

Position Announcement Nonprofit Management and Leadership Specialization Public Administration Division, Mark O. Hatfield ... see job

Assistant Director of Executive MBA Programs
Loyola College in Maryland

1. Student Recruitment Assist Director with EMBA and MBA Fellows prospective student lead generation, admission recruitment ... see job

Assistant Director of Experiential Programs
University of Colorado at Denver and Health Sciences Center-Downtown Denver

Posting Description: The University of Colorado Denver, School of Pharmacy is entering a major growth and ... see job

Director of Graduate and Continuing Education Admission
North Central College

GENERAL RESPONSIBILITIES • Set a strategic vision for graduate and continuing education admission in collaboration with the ... see job

Career/Student Affairs Advisor
Willamette University

Willamette University’s Atkinson Graduate School was named as one of the Princeton Review’s “Best Business Schools” ... see job

Asst Dean for Graduate Programs
University of Pennsylvania

The nation’s first university, Penn is a world-renowned leader in education, research, and innovation. Situated on a ... see job

Temporary Graduate Social Work Instructors, Fall 2008
West Chester University of Pennsylvania

Join a vibrant campus community whose excellence is reflected in its diversity and student success. West Chester University ... see job