Single Point of Failure

Although no data were lost when the examination software provider ExamSoft stumbled last week, the incident serves as another warning shot about single points of failure in higher education.

Last Tuesday, thousands of budding lawyers feared they would have to put their careers on hold as ExamSoft experienced a major slowdown, in some cases not processing the students' uploaded bar exams until hours after submission deadlines. The company is still investigating what caused the technical issues, according to a spokesman.

For some, last week brought back memories of last fall, when a slew of glitches crippled a new version of the Common Application. The bugs caused hundreds of students to miss early decision deadlines, and eventually led to a leadership change at the organization.

“It is sort of mind-blowing, but a real situation,” Phil Hill, a higher education consultant, said in an email. “[W]ith many online tools we're moving past institutional effects to system- or nationwide effects.... Now, with ExamSoft, Common App and others, we are opening up new situations where multiple states are all subject to single points of failure.”

It is likely a risk boards of law examiners and test-takers will have to endure. The boards -- whose members can sometimes be counted on one hand -- often need to outsource exam services to third parties to manage hundreds, if not thousands, of applicants.

Before taking the bar, students register the laptop they will use on exam day and pre-install SofTest, ExamSoft’s secure testing environment. On test day, students are given a password that locks them out of the Internet or files stored on their computer. In case of a computer crash, the test locations have one backup plan: pen and paper.

But taking the test wasn’t the issue last Tuesday. Law students across the country were able to finish their essays and save them in SofTest, effectively locking their answers in a software vault until they were ready to upload them.

Students in Pennsylvania, for example, finished writing their essays at about 4:30 p.m., giving them more than six hours to connect to the Internet and ensure their answers were uploaded to ExamSoft, before looking ahead to a second day of testing. Proctors encouraged the students to head back to their hotels, the local coffee place or home to space out web traffic and avoid the surge of 1,833 simultaneous uploads.

By some reports, students who uploaded their tests in the late afternoon avoided the issues, but as afternoon turned into evening, technical glitches led to a six-hour processing delay. With the deadline fast approaching, panicked students flooded ExamSoft’s support channels for help, their outrage spilling into social media.

It’s easy to see where their angst came from. According to official documentation from the Pennsylvania Board of Law Examiners, students were informed that “Failure to upload my answer files by 11:00 p.m. on the essay/PT day of the bar examination (Tuesday) may result in the disqualification of my essay/PT answers, and I will be required to sit for a subsequent bar examination.”

Similarly, in Tennessee, students were told, “If you fail to upload your answers by the deadline, your answers from the examination will not be graded, and you will be given a score of zero (0) for each answer.”

Since the bar is offered only twice a year in most states, many students feared missing the deadline would mean having to wait until February to be certified.

“It’s like a six-month unemployment stint,” said Robert G. Wible, director of academic success and bar exam services at the University of Pittsburgh School of Law. “That’s the big stressor.”

At least 16 states announced extended submission deadlines. Forty-two states use ExamSoft to administer the bar exam, according to the company’s website. The cost of a computer-based test varies from state to state; in Pennsylvania, test-takers pay either $115 or $165 depending on when they registered.

Before Wednesday morning, about 1,400 of the Pennsylvania bar exam takers had actually uploaded their essays successfully, but conflicting messages from ExamSoft and the state board led some to second-guess themselves.

By 7:20 p.m. Tuesday, ExamSoft notified Gicine Brignola, the board’s executive director, that the system wasn’t working as intended. Two hours later, when the issue had yet to be resolved, the board extended the deadline for submission until 6 p.m. Wednesday. Students who had already submitted their essays were left staring at two confusing emails: one from ExamSoft, confirming their upload, and another from the board, acknowledging the upload issues.

“The company is still trying to figure out what went wrong, and we will be watching that, because we can’t have a repeat,” Brignola, former assistant dean of career services at the Pennsylvania State University Dickinson School of Law, said in an interview. “I also think they need to work on their communication. Letting us know and letting the applicants know earlier would have helped. I think that largely was part of the problem why people were confused.”

In an email to customers sent over the weekend, ExamSoft CEO Daniel Muzquiz promised to share a full account of what went wrong “in the coming days.” His email also quashed the idea that the company’s systems were unprepared to handle the traffic.

The “overall exam volume was almost the same as last year’s and lower than other weekly periods for final exams, so the volume itself was not the challenge,” Muzquiz wrote.

Most importantly, no test-taker lost his or her essays. SofTest saves a copy of the test every minute, and even if the upload mechanism remained broken, the files could -- with a little work -- be salvaged from the computer.

Brignola said Pennsylvania won’t seek to replace ExamSoft after last week’s “hiccup.” The state “has been using ExamSoft for quite a while and with great success, and we still -- even with this incident -- have no concerns about the exam content,” she said.

With a 10-person staff, building an in-house system isn’t an option. Neither is dropping a computer-based test altogether. “To go back to handwriting at this stage would be shocking for applicants,” Brignola said.