The Problem Solvers

As a fledgling voice of reform in higher education, Salman Khan is an oddity. He cannot name any higher education accrediting agencies off the top of his head. He advocates for competency-based credentialing, but has never heard of Western Governors University. He is capable of talking on the phone for a full hour without using the word “disruptive” once. Until recently, he was an analyst for a hedge fund.

Here is what Khan does know: algebra, statistics, trigonometry, calculus, computer science, biology, chemistry, astronomy, physics, economics, and finance -- well enough, at least, to demonstrate the concepts via brief video tutorials on Khan Academy, his free learning website. What began in 2006 as an attempt to tutor his young cousin from afar has evolved into a 2,700-video library with millions of monthly visitors.

Many have lauded Khan’s natural skill as a teacher. Khan’s charmingly unpolished home recordings form the public face of the organization and provide a peg for media narratives about online learning and the YouTube-ification of the textbook in an era where the rising prices and demand for higher education has collided with the Internet’s culture of free.

But Khan and his 20-person team -- refugees from hedge funds, consulting firms, software companies, and tech start-ups -- say the videos are hardly the most innovative work they are doing. The real revolution at Khan Academy, they contend, is not being streamed on its vaunted website; it is happening in the back end of the platform, where Khan’s engineers are learning as much about the site’s 1.4 million registered users as those users are learning about math and science.

Khan Academy’s explicit goal is to teach people fundamental concepts. But in the process, it hopes to break new ground by changing how educators think about teaching, how psychologists think about learning, how employers think about credentialing, and how everybody thinks about the price of a good education.

“I think too much conversation about Khan Academy is about cute little videos," Khan said in an interview last week. “Most of our resources, almost two-thirds of [the staff], are engineers working on the exercises and analytics platform. That, I think, is what we’re most excited about.”

Mastering Solutions

Jace Kohlmeier is one of those engineers. Like Khan, Kohlmeier is late of the finance industry. Before joining Khan Academy on a volunteer basis last year, he trained computers to predict price changes in the stock market and make trades based on those predictions. Stock-trading and teaching might be different endeavors on the surface, but Kohlmeier says the work he does for Khan Academy is similar to the statistical modeling he did in finance.

In both cases, “You want to understand the process of why things happen the way they do, and find the explanatory variables,” he says. “And in certain situations you also want to predict what will happen in the future.”

Using math and computer science concepts decidedly more advanced than most of those in Khan’s video library, the Khan engineers have trained the website’s exercise platform how to predict, with startling accuracy, how likely it is that a student will correctly answer the next practice problem -- and whether that student will be able to solve the same type of problem a week, two weeks, and a month later.

They do this by accounting for hundreds of data points that describe, in numbers, the entire history of the relationship between a learner and a concept.

“If [a user is] logged in, then we have the entire history of every problem they’ve done, and how long it took them, and how they did,” says Ben Kamens, the lead developer at Khan Academy. “So whenever anybody does a problem, we see whether they got it right or wrong, how many tries it took them, what their guess was, what the problem was, how many hints they used, and how long they took between each hint.”

The Khan engineers are also working to tweak the exercise platform so it does not confuse genuine mastery with “pattern matching” -- a method of problem-solving wherein a student mechanically rehashes the steps necessary to solve that type of problem without necessarily grasping, conceptually, what those steps represent.

Pattern-matching is one of the human brain’s most basic learning tools, Kamens says. It is the sort of useful imitation that allows toddlers to learn how to use language without first learning how grammar works. But there is a difference between imitating problem-solving procedures and mastering the logic undergirding those procedures, Kamens says. Getting to that level of understanding, he says, is probably what determines whether students will remember how to solve a problem after the test is over, after a course is over, and -- most importantly, in Khan’s view -- once their formal schooling is over.

Khan has half-joked that his ideal assessment model is having professors ambush their students in the hallways with random questions, months after the student has passed the exam, and revise their score based on whether they've kept their chops. At Khan Academy, that half-joke is half-real. At a time when students are always within arm's reach of a computer and a wireless signal, “mechanic practice schedulers” can spring questions on students at intervals to gauge how well they remember how to do certain types of problems. This would allow Khan's team to collect data on how well students retain their command of different concepts, which in turn would enable them to look back at their original interactions with the concepts and try to spot variables that correlate with long-term retention.

“We have already built some internal models that incorporate memory/forgetting over time into the predictions,” says Kohlmeier. “We'll continue improving them.”

The Khan engineers think that randomizing the types of problems posed during these jump-outs might serve as a shibboleth to distinguish between people who have truly mastered concepts, and those who passed the tests by temporarily memorizing a series of rote steps.

This is crucial, Kohlmeier says. When students are doing practice problems in a textbook chapter on quadratic equations, they know each problem can be solved by factorization -- even if they cannot say why. But in the world, whether in finance, software or education, the right path to a solution is not always so obvious.

“A big part of real-life problem-solving,” Kohlmeier says, “is recognizing what kind of problem you’re dealing with.”

Defining Problems

While Khan Academy’s current institutional partnerships, and much of the content in the Khan Academy library, are oriented to the K-12 world, college and university leaders are paying attention. Khan may not know the names of all the big movers in higher education yet. But they know his.

At the annual Future of State Universities conference in October, Khan was featured on a bill with a who’s who of statesmen, Education Department officials, industry soothsayer Clayton Christensen, and a number of prominent executives from university systems and accrediting agencies. His presentation had the audience of college presidents and other traditional academic leaders rapt. He has since been added to the docket for the American Council on Education's annual meeting in March, penciled in to the same "reformer" lineup spot that Christensen filled at this year's meeting.

In October Khan told the Future of State Universities audience that the most pressing problem facing the education system is not so much the retention of students in academic programs, but the retention of specific academic concepts in the minds of those students. Completion means nothing, said the Bill & Melinda Gates Foundation grantee, without comprehension -- a command of crucial skills that stick around long after the test, and the course, are over.
 

College students do not graduate with a firm enough grasp of the skills -- particularly in STEM (science, technology, engineering, math) fields -- that they really need to land good jobs, he said. As a result, the credential colleges use to signal the competence of their graduates, the college degree, says very little about what its holder actually knows.

In a conversation with Inside Higher Ed last week, Khan expressed some ideas on how to improve the signaling quality of academic credentials. Under the current regime, a degree from a college amounts to something similar to an acceptance letter from that college, he says. And that is not ideal for employers.

“We find a lot of college grads with high GPAs that have been exposed to many things … but even in their purported majors, they have a pretty weak grasp of” essential concepts, Khan said. “It’s almost like you view them as a blank slate, and the most impressive thing about them is that they got in to University X.”

In other words, the current price of a college degree is not just the balance of four years’ tuition; one must also consider the cost, to students and employers, of the ambiguity hanging over what the degree actually means.

One root of the problem is the fact that the college degree is issued by the same institution that is in charge of setting, and enforcing, the standards of that credential, says Khan, who holds four degrees himself. This is tantamount to investment banks rating their own securities, he says. Meanwhile, the accrediting agencies that are in charge of making sure those “ratings” are legitimate do not currently focus on what students coming out of those institutions measurably know.

That is why, when an audience member at Khan’s Future of State Universities talk asked whether Khan Academy was interested in credentialing, its tutor-in-chief answered with an enthusiastic yes-but. Khan told Inside Higher Ed that he does not want to turn his free, online trove -- whose 2,700 videos could theoretically be organized into course-length sequences -- into a credential-granting institution. What he does want to do is advocate for the creation and mainstreaming of credential-granting institutions that exist wholly separate (“decoupled,” in Khan-speak) from the institutions (including his) that do the teaching.

In Khan’s ideal world, this would mean an independent third party that tests specific competencies and awards credentials corresponding to knowledge areas in which a student can demonstrate mastery -- like the MCAT or standardized tests like a bar exam for calculus, physics, or computer science. “It would be much more useful, speaking as employer, if they show they’re just at the top of the charts on a certain skill set that we really want,” he said.

Re-ordering Operations

Although he speaks about the value of college in the sort of sober, transactional terms that might read as hostile to liberal education, Khan insists that he is sympathetic to the idea of learning for reasons of pure curiosity or personal pleasure. Not for nothing has Khan researched and recorded tutorials on the Civil War, the Cuban Missile crisis, and the French Revolution. He recently hired two art history Ph.D.s -- Beth Harris and Steven Zucker -- to join him as the first two non-Khan faculty members at Khan Academy.

At the same time, Khan sharply criticizes the buffet approach to curriculum that leaves graduates with general impressions about many topics but few applicable skills. Teaching students “how to think” is not good enough, he says.

“If you can go deep in many things, awesome,” he told Inside Higher Ed. “That’s wonderful. But the reality is, right now, you’re forcing students to [obtain], and employers to hire students with, kind of a broad and very shallow experience base -- an expensive broad experience base. And it’s not clear that’s doing anyone any good.”

“Higher order” skills in critical thinking and creativity are useful only to the extent that graduates wind up in a position to apply them, Khan said. In the malaise of post-college unemployment, a graduate’s aptitude for analyzing themes in literature or conducting reliable research will languish. “If you don’t have that starting point of [graduates] being engaged and productive in society in some way, then the rest is just a waste of time,” said Khan.

Distribution requirements, the four-year model, and the buffet approach to curriculum all contribute to the “arbitrariness” that muddies the signaling function of college degrees and “have no relation to what makes you a more productive citizen or better for society or a more creative person,” Khan said.

“If you decouple [learning and credentialing], the arbitrariness is gone,” he added, and “it federates the options to adjust to what people’s needs are.”

Stability theories

Khan Academy’s efforts to describe the learning process with granular data are not necessarily new. Publishers and test-prep companies have developed software that creates detailed learning profiles for students working through problem sets and uses predictive modeling to gauge mastery.

The difference is that while those products can cost each student as much as a regular textbook (publishers often package the software with new textbooks, or else sell them separately for about the price of a used one),  Khan Academy offers its video tutorials, and unlimited use of its "smart" exercise platform, free. And its eponymous founder has no plans to bill students.

“While I’m around, Khan Academy will be free,” Khan says. He says the organization has attracted enough investment  ($16.5 million, according to the New York Times) to subsidize its $3 million annual operating budget. And although there are key members of Khan's team who could be making that much themselves each year in the corporate sector, Khan believes that mission can be just as powerful as money when attracting top talent, especially in the era of the Web and machine learning, when it only takes a handful of bright minds to ignite the potential of millions more worldwide.

"There’s a very strong ethic and philosophy at Khan Academy about openness," says Kohlmeier, who works for Khan Academy pro bono, having left a lucrative post as president and co-founder of a proprietary trading firm that employed 30 people. “That’s the M.O. that I intend to follow," he says. "I would not be at Khan Academy if Khan Academy were not a nonprofit.”

While Khan and his team shy away from confrontational statements about taking on commercial publishers, they are confident that their machine-learning program is sophisticated enough to apply some pressure to commercial products. “Is there a potential that our presence in those markets will change what the profit opportunity is?" says Kohlmeier. "I definitely think that’s a possibility.”

But Khan Academy's determination to stay on the cutting edge of teaching technology while remaining free is not entirely original, either. Carnegie Mellon University's Open Learning Initiative has, since early last decade, used federal and private grant money to build free course modules infused with "smart" software that supplies instructors with detailed feedback on individual students, along with exercise platforms that adapt to the needs of each student as they work through problem sets. Candace Thille, the head of that long-running project, has posed similar ideas to Khan's about a system of credentialing based on measurable fluency in various concepts.

And yet higher education's traditional teaching, assessment and credentialing models remain largely in place, despite Thille and her Open Learning Initiative having been widely known, and respected, for years. When considering what impact Khan and his band of outsiders could have on the higher education establishment and the digital content providers that serve it, an insider's perspective can be sobering.

At best, Khan Academy is a good idea with a lot to prove, says Carol Twigg, CEO and president of the National Center for Academic Transformation. At worst, it is "just another repository scheme" that will stagnate once the buzz dies down.

Twigg, who for years has been a leading advocate of using technology to upend calcified models of teaching and learning, would seem a natural ally for Khan. But she has also been around enough to have seen more than one ostensibly transformational content platform peter out. In the absence of any evidence that Khan Academy's videos and exercise software can successfully integrate with college classrooms (which, she points out, are not the same as K-12 classrooms, where Khan is road-testing his software through partnerships with several charter schools), and produce better learning outcomes than the stuff that is currently available, Twigg doubts that Khan Academy will transform any college curriculums anytime soon.

"The idea that you can just put stuff out there, and that it will magically be effective and used effectively -- there’s just no evidence of that,” Twigg says. Collecting evidence requires integrating with college classrooms, which requires scale and support, which requires money, she says. Lots of it. "Sixteen million dollars is not chump change," says Twigg, "but you need to be able to support and sustain it." Nonprofit projects in higher education do not have a great track record on this, she points out, not even the most highly regarded ones.

Khan himself comes off as almost cavalier when discussing the future of Khan Academy. "There is no business model," he told Inside Higher Ed. Asked whether he thinks his nonprofit model is sustainable, he replies, "I'm not sure," adding, "It’s not like we have a big strategic plan to raise 'X' dollars by 'X' date or anything like that.”

Then again, Khan has always been confident in his ability to solve for "X."

"We're still young," he points out. If money becomes a problem, Khan and his team will work on a solution. And when that day comes, the method they use will probably depend on what the equation looks like.

For the latest technology news and opinion from Inside Higher Ed, follow @IHEtech on Twitter.