Getting Smarter on Skills Testing

The big enchilada of potential disruptions to higher education is if employers go outside of the academy to size up job seekers. While that prospect remains fanciful, for now, new approaches to skills assessment show what the future could look like.

Take Smarterer, a Boston-based start-up that offers 800 free online tests for people to prove their chops in areas ranging from C++ programming to speaking English for business or understanding Gothic architecture. And not all the assessments are about getting a job -- there are quizzes on punk rock history and how to use Twitter.

Smarterer's tests are crowdsourced, Wikipedia-style, and users can get a meaningful score by answering fewer than 20 questions, company officials said. That means they can tout a skill on their Smarterer profile after spending as little as one minute taking a test.

Jennifer Fremont-Smith, Smarterer’s CEO, describes the company as a “third-party, super powerful assessment and credentialing tool.” Its goal is not to replace the college degree, which Fremont-Smith acknowledges is currently the gold standard of credentials, but to give employers an additional way to sort through job applicants.

“Our big vision is to be the global standard of skills measurement,” she said.

The company does not currently have a plan for making money, with a build-it-and-they-will-come approach that is fairly common among tech start-ups. However, Fremont-Smith said more than 400 employers have used the service to help evaluate job candidates by testing skillsets

Smarterer joins several other relatively new attempts to offer job seekers a way to demonstrate what they can do through testing. Rival companies like Skills.to and Degreed attempt to assess skills and learning. And the ACT's National Career Readiness Certificate measures employability with tests on applied mathematics, locating information and reading for information. The certificate is geared for entry-level jobs, even for applicants who lack a college credential.

On the other side of the spectrum, Bloomberg in 2010 introduced an assessment aimed at students who want to bulk up their C.V.s to land jobs in finance. The test covers 11 fairly narrow categories, like investment banking and analyzing financial statements.

This method of signaling qualifications based on knowledge and skills, rather than college-issued credentials, shares similarities with competency-based education and prior-learning assessment – also hot ideas in higher education. Those two approaches lean heavily on assessments for the issuing of college credit based on what students know, rather than what they learned in an academic setting.

However, colleges are still in the driver’s seat in competency-based education and prior learning. They typically issue the tests, such as Excelsior College’s assessments or College Level Examination Program (CLEP) exams, to measure learning. And even in prior learning, where colleges grant credits for learning outside of the classroom, students generally enroll and pay for those credits.

Another related idea is badging, most notably Mozilla’s Open Badges project. That free platform allows anyone to issue, earn or display badges that display their earners’ skills or achievements. Mozilla, however, does not conduct testing, which is up to issuers of badges.

There is an “emerging market” for skills-based assessment, said Louis Soares, an expert on higher education and a senior fellow at the Center for American Progress. But he said it’s too early to predict whether services like Smarterer could break higher-education’s stranglehold on credentialing. And for now, they remain add-ons to the conventional resume.

One challenge is that ditching the degree would cause legal problems. For example, Soares asks, if an employer passes up an applicant with a relevant master’s degree to hire someone with an associate degree and plenty of assessed skills, would that open up the employer to a lawsuit?

Smarterer and other similar services cater to professionals, most of whom probably hold bachelor’s degrees, at least. If skills assessments spread further down the education pipeline, perhaps to certificate holders, then colleges might need to start worrying, Soares said. And he said the broader adoption of skills-based credentialing will happen at some point, whether by colleges or outside players.

“I believe that it is coming,” said Soares. “Companies are looking for ways to assess more modular skills.”

Crowdsourcing 50,000 Tests

Smarterer uses adaptive scoring to measure what people can do. The approach is based on the Glicko rating system, which chess players use to signify where they stack up.

Under the Glicko system, a chess player has a ranking between 0 and 2,400. Each game played affects the ranking, and weight is given to an opponent’s ranking. For example, an 800 player gets a lot of points for beating someone with a 1,400 ranking, and fewer for beating a 700 player. And a loss to someone with a lower ranking is a big hit.

Likewise, Smarterer’s assessments weigh questions based on a test-taker’s previous results. And people can return over time, continuously reassessing and benchmarking their skills as they learn more. Users get tags like "proficient" and "expert" along the way. Only 2.5 percent of test-takers in any topic make it to "master" level.

“With every answer we’re learning a little more about their skill level,” said Fremont-Smith.

The company also draws strength from the crowd, by adjusting questions based on feedback. And until recently, top scorers could get their names on “leader boards.”

For example, one of Smarterer’s most popular tests is on using PhotoShop, the photo editing program. A creator of the software popped up on the leader board for that test, Fremont-Smith said, and “added a few questions” to the assessment. (The company dropped leader boards a couple months ago.)

Fremont-Smith said crowdsourcing has helped the company create more, and better, questions than it could have through more conventional means. “We could never hire a bunch of Ph.D.s and put them into a room and write them all.”

So far, the website features about 50,000 questions. People have taken 1.7 million of the tests.

Smarterer is not a threat to traditional higher education, Fremont-Smith said. The tests don’t measure the same class of skills colleges do with grades for academic knowledge, with more of a practical, vocational focus. And Fremont-Smith said college career services offices could steer students toward Smarterer to help make them more appealing to employers. Some colleges have expressed interest in the site, she said.

But Smarterer gives a glimpse of how skills assessment could encroach on colleges’ turf.

Michael Staton is the founder of Inigral, a company that develops social software for student recruitment and retention. In a recently released research paper on disaggregating the components of a college degree, Staton discussed the potential for Smarterer and other tools to measure competency.

“It is easy to imagine a future where employers can identify prospective recruits or screen applicants through an easy to use assessment system,” Staton said. “If reported performance on these assessments becomes a credential of accepted value, then it may contribute to additional questions about the value of an expensive degree.”