Measuring Motivation

Like most colleges, the University of New Mexico is trying to help more of its at-risk students get to graduation. And officials at the university know academic preparation isn’t the only factor to watch.

There is a broad consensus that so-called noncognitive skills -- motivation, stress management, organization and even the ability to cope with financial pressure -- influence whether a student succeeds in college. So the university recently started using a test from the Educational Testing Service, dubbed SuccessNavigator, to measure those attributes.

New Mexico is a flagship research university. But it serves a relatively poor state, and enrolls thousands of low-income students, who are often the first in their families to attend college.

“We have the challenge of many students arriving at our university who are not prepared for college-level work,” said Renee Delgado-Riley, the university’s director of student affairs and research.

The university offers these students extra academic supports, such as regular meetings with tutors or academic advisers. To identify students who are at risk, the university looks at high school grade point averages and scores on standardized tests like the ACT or SAT.

But those indicators don’t measure students’ drive or whether they are likely to seek help, said Delgado-Riley. The university can be more optimistic about students who score high with those attributes, she said, and can track down those on the other end of that scale to give them a nudge. That’s where SuccessNavigator came in.

This year, 1,500 students took the 30-minute test during New Mexico’s new student orientation. Those who were required to take the test included scholarship recipients, first-generation college students, those enrolling in STEM programs -- and all of the athletes who play for the university’s intercollegiate teams, which participate in Division I of the NCAA.

The university decided to include its athletes because of the high stakes they face, according to Delgado-Riley. The students have tight schedules, and can lose their scholarships if they fall behind.

“It’s an early warning indicator,” she said, for both athletes and students who are more at risk of dropping out.

Sometimes the university uses the test’s results in a one-off way, such as an adviser recommending a specific course for a student.

The test can also be helpful in developing and managing the success plans some students are required to create with their advisers, said Delgado-Riley. Those plans, which are individualized and map a path through courses and semesters, come with resources and referrals. For example, some students must meet with advisers once a month to review their plans. And some do grade checks with their tutors.

Delgado-Riley said SuccessNavigator has become a useful tool in the university’s holistic approach to student advising.

“It gave us a better standardized metric for understanding our students in a different way,” Delgado-Riley said.

Remedial Placement

ETS introduced SuccessNavigator two years ago. The large nonprofit testing firm spent seven years developing it.

The test is one of several noncognitive assessments on the market. Others include the Learning and Study Strategies Inventory (LASSI) and ACT Engage. The Grit Scale from the University of Pennsylvania maps out similar territory, but is not a commercial product. Researchers also have studied the effect of hope on student success.

So far nearly 200 colleges and universities have used SuccessNavigator, said Ross Markle, senior research and assessment adviser for ETS’s higher education division. The test has been taken more than 41,000 times -- 19,000 times this year. Colleges typically pay $6 for a student to take the test.

SuccessNavigator is designed to help at-risk students stay on track during their first year of college. But it can also be used as they move farther along toward a degree. The test measures four major psychosocial general skills: tools and strategies for academic success, commitment, self-management, and social support.

Underneath those categories, the assessment looks at 10 subskills ETS says are directly related to student success and persistence. Those include the likelihood of completing assignments on time, the belief that the benefit of a college education outweighs the cost, test anxiety, feeling connected to peers, the likelihood of asking instructors for help and whether family pressure makes it hard for a student to commit to college. (See below for more.)

Results from SuccessNavigator generate a score report and action plan, which go to advisers. They include a student’s projected GPA range in their first year, their probability of returning the following year, individual scores for general skills and subskills, and recommended next steps.

The test seeks to move the focus from a student’s background to their behaviors, said Markle, what he calls the why of success.

By helping to better determine whether a student can stick with it, colleges can use the test to structure feedback -- like New Mexico is with advising and interventions. Others, however, use the test to attempt to better determine which students can skip remedial math and English, moving directly to credit-bearing, college-level courses.

Iowa Western Community College factors in SuccessNavigator while making decisions about developmental course placement.

Fully 80 percent of Iowa Western’s students test into remedial math, said Tori Christie, vice president of student services at the college. And the two-year institution lacks the money to give all those developmental students the support they need, such as interventions with advisers.

Influential studies have found that too many students are being funneled into remedial courses. The research showed that popular placement tests misplace substantial numbers of students who would have been able to pass college-level courses.

SuccessNavigator can be a complement to placement tests and high school GPA, Markle said, by helping to “accelerate” students who have the motivation and other skills to bypass remediation, despite their low placement test scores.

“It’s an additional piece of information,” he said, to help students “bump on to that next level.”

ETS conducted a study to see how students who made that leap fared in credit-bearing courses. The study -- which looked at results for 3,647 students who attended a large, urban community college -- found nearly identical passage rates for students who were accelerated into college-level courses based on their noncognitive scores compared to students who were placed into the course through conventional means.

Iowa Western began using the test for course placement this semester, for all incoming students. They take it before a mandatory new student orientation, Christie said, and in a student-success course.

The college also is planning on working the results of SuccessNavigator into its parts of its curriculum. For example, Christie said Iowa Western plans to train faculty members in its nursing program to use the results of the nonacademic skills test in delivering their courses.

In addition to advisers and instructors, students can receive their test results from SuccessNavigator. The student reports are different, however. They include skill scores and tailored recommendations, but not predicted GPA, which could be discouraging to see.

Even though Christie said ETS appeared to have been careful about how it words student reports, she still worried that the test results could be off-putting to some students.

The opposite happened, however. She said students overwhelmingly seem to be encouraged by the results, and have pushed their advisers and faculty members to use them.

“They are really receptive to it,” said Christie. “They want to be successful in college.”