Clear-Eyed Analysis of OER Impact

The student success group known as Achieving the Dream had nice, humble ambitions when it rolled out an initiative two years ago aimed at helping spread the use of open educational resources to build affordable degrees at community colleges.

The freely accessible and openly licensed learning materials "are valuable for colleges and their students in terms of invigorating teaching and learning, lowering costs and laying the groundwork for degree programs that meet up-to-the-minute student and employer needs,” Karen Stout, president and chief executive officer of Achieving the Dream, said in 2016.

Just those few modest goals: revitalizing instructors' teaching and learning, reducing what students spend, and improving the timeliness and value of the colleges' credentials.

A thorough new report from Achieving the Dream offers a modulated assessment of what the OER Degree Initiative accomplished in its first two years, through fall 2017. It shows that the program has stimulated the 38 participating two-year colleges and their instructors to create scores of low- or no-cost materials for courses (numbers that are accelerating as time passes, as a news release at the end of the 2017-18 academic year shows), saving students at least $1.4 million. (The project has also embraced a more conservative methodology for calculating students' savings on digital course materials, contributing to a trend toward more realistic accounting of such figures.)

Students generally gave the openly produced courseware good reviews, and use of OER appears to be leading faculty members to use more student-centered teaching practices.

Significant impediments exist, though, according to the report. Student awareness of the availability of the OER-driven courses remains low, and faculty willingness to do the hard work of creating such courses is limited by the heavy time demands and professors' concerns about losing control of their courses.

The report (which Achieving the Dream hasn't yet released publicly) represents an early status update on the effort's progress. The group's leaders acknowledge what they know so far (what it takes to get OER courses and degrees off the ground, how key constituents like students and professors view the work done so far, initial evidence on the financial impact on students and colleges) and what they don't know.

Foremost among those are any sense of the academic impact on students in terms of degree completion, and therefore of the cost-effectiveness of the initiative, which will be essential in gauging, ultimately, whether the investment in trying to build degrees on open educational resources is worth it and can be sustained when grant funding runs out, as it inevitably does.

“What we’re trying to figure out with this grant is how colleges can do this for the long run,” said Richard Sebastian, director of ATD's OER Degree Initiative. “We’re trying to take a clear-eyed view that this money is going to go away” eventually and colleges “need to figure out strategies for keeping these programs going.”

Driving Change Through Coalitions

Achieving the Dream is an exemplar of the emergent "network" approach to scale and institutional change in higher education -- coalitions of colleges getting together, often on their own but increasingly at urging of (and with funding from) foundations, to try to address common problems or achieve common goals. (Think the University Innovation Alliance or the American Talent Initiative as other recent examples.)

Formed in 2004 with the support of Lumina Foundation, the group has spurred cohorts of community colleges to experiment with and improve on their approaches to student success and, importantly, to study their success in doing so, warts and all. Projects have addressed a range of topics over the years, including the faculty role in student success and the hiring of campus leaders.

The open educational resources initiative, with a price tag of close to $10 million funded in part by the William and Flora Hewlett Foundation and the Bill & Melinda Gates Foundation, aims to lower financial and other barriers to college completion by making curricular materials more widely available to students and, ultimately, by delivering whole degrees at little or no cost to students.

Through fall 2017, participants in the Achieving the Dream project had created 212 OER courses that met certification standards set by Lumen Learning (more on that process later), with 1,704 sections enrolling 41,125 students. (By this summer, according to a news release put out by ATD at its annual meeting, the number of courses had risen to 650, amounting to 52 degrees one could earn without paying for course materials.)

Some analyses of savings generated by lower-cost approaches to college course work derive their calculations by multiplying the bookstore cost of the original textbook by the number of students in the course -- ignoring the reality that many students buy used books, rent books or do not buy them at all. In its report, Achieving the Dream calculates what it calls a "blended" price estimate, which on average was half of the traditional estimate, as seen in the graphic below.

Using the blended and traditional price figures, ATD estimates the cost savings to students from its 212 courses at between $1.3 million and $2.6 million, with the two measures of textbook price representing the end points of the range. The figure subtracts out the modest fees that some institutions charged students to take the OER courses or charged for printed OER course packs.

Based on surveys of students at participating institutions, the report's authors estimate that one in six students had at some point stopped taking courses because they could not afford the textbooks (a figure that was higher for underrepresented minority students). Respondents reported that they would use the savings from the OER courses and degrees for a mix of personal expenses (48 percent), other course supplies and materials (43 percent), and taking additional courses (28 percent).

Evidence of Impediments

Students can't save money from OER courses if they don't know the classes are available, though, a problem Achieving the Dream acknowledges. "A surprisingly high proportion of students -- almost 60 percent -- indicated that they did not know the class they were signing up for was OER at the time of registration," the report notes. "In fact, 19 percent of students indicated that they learned their course was OER by taking the [ATD] survey."

The report says that some member colleges are adding an OER designation to the course catalog, which is how 24 percent of students said they ended up in such a course. "I purposely clicked on courses that had the OER symbol because I knew that would save me money," one student respondent said.

Fewer than a quarter of students said they had enrolled in an OER course at the recommendation of a faculty adviser or classmate, evidence, the report said, of the "varying levels of effectiveness of word of mouth for increasing awareness of and enrollment in OER." More evidence: the vast majority of students in focus groups had no idea their institution offered OER degree pathways.

About six in 10 students were very likely to recommend an OER course to a peer, with some students saying access to technology for course materials was a problem, and small proportions saying the quality of the materials was worse than traditional textbooks.

Most of the campus-specific OER degree efforts grew out of what the report calls "grassroots faculty activity" -- more than emerged from efforts initiated by administrators or academic departments. (Librarians were likeliest to be at the center of institution-led initiatives.)

In most cases, the community colleges used the support from the ATD project to fuel nascent efforts that had already been started by professors, usually with an eye toward expanding the scope to reach the largest number of students.

But just as instructors were usually the champions of OER efforts, they also frequently stood in the way. Professors cited the usual complaints about OER: qualms about the availability of material of sufficient quality, and lack of time to identify and certify the materials as freely available for use.

"Some instructors we interviewed expressed frustration with colleagues who they thought were stuck in their old ways and had become too accustomed to the 'easy' use of pre-made worksheets, assessments and instructions that come with publishers' packages," the report stated.

ATD officials also said they struggled to get some institutions and professors to understand the value of creating fully openly licensed (as opposed to merely "free") courses. Building courses with open educational resources only drives down the costs for institutions and students if the academic programs built with OER can be shared with (and ideally improved upon by) instructors and students at other colleges -- that's how the vision of high-quality, steadily improved OER becomes realized.

But Sebastian said some colleges in the consortium struggled with the requirement (to be certified as truly "open" by Lumen Learning) that every piece of content be openly licensed and correctly attributed. That's hard work, and some preferred shortcuts of using content that they had purchased through an institutional subscription (and hence already paid for) to fill gaps or plug content holes.

That may have worked for that institution's own course -- but at the point where the courses are then designed to be put into the OER Commons and shared with other colleges (which may not have subscriptions to the same copyrighted content), the course falls short.

"This tells me that we need to do a better job helping them see the value of putting fully open courses and programs back into the commons and connecting faculty to the broader global OER community," said Sebastian.

Money Matters

ATD and its partners, including SRI International, the project's research and evaluation analyst, and rpk Group, which is leading the project's financial analysis, are a ways away from having full-blown data on institutions' costs in creating OER degrees.

A limited analysis of five participating colleges hints at some of the issues. Most of the financial backing for the colleges' degree-building efforts came from the foundation grants, with the colleges themselves contributing about a third. About 40 percent of the expenditures on the average program went to pay instructors, and about 30 percent went to administrative overhead.

The average course took roughly 126 hours to develop, and interestingly, programs built by a team of instructors cost the institutions significantly more than those built by individual instructors -- suggesting that the teams had not figured out how to divide up the labor but were doing overlapping work to build the courses.

What the project's leaders have learned so far about the economics of OER suggests that making the programs sustainable will be challenging. What the institutions must spend to build the OER curricula is only part of the calculation; there is forgone revenue for those that own their bookstores (and lose textbook revenue), and a full-blown OER degree approach will require significant administrative oversight, for such things as recruiting and supporting instructors, updating course material, and providing professional development.

The report speculates that more colleges may adopt user fees, but notes that the best bet may come from newfound tuition revenue from improved student performance and retention, "although it is too early to make this claim."

Follow me on Twitter @dougledIHE.