Open Educational Resources: What We Don't Know

In an age of fake news, I took solace in the fact that science has a peer-review process that separates the wheat from the chaff. It was reassuring to think that authors had sifted and winnowed through reams of research to present key findings.

I realize I was letting myself off the hook too easily. Reading with a critical eye and between the lines is essential. Recently Cornell University concluded a yearlong review and found that a prominent food researcher “committed academic misconduct in his research and scholarship, including misreporting of research data.” While this case may be an anomaly and represents intentional obfuscation, it is possible that readers of any research may miss details hiding in plain sight. A quick look at a title or abstract may provide a different story from what the author actually reports. Beyond watching for nuance, we readers must also be watchful for methodological rigor.

I have conducted scholarship of research and teaching for over 20 years now and am co-editor of the American Psychological Association’s journal Scholarship of Teaching and Learning in Psychology. I read a lot of research on higher education and pay close attention to research on new modes of instruction. Editing has made me watchful for issues with writing clarity and research design. Much of the research I review has issues with both. Paying close attention to both is important, especially when student learning is at risk.

In the case of open educational resources (OERs) -- free course materials, openly licensed and able to be used and mixed with permission -- student learning and millions of dollars are at stake. As enrollment pressures and funding shortcomings continue to shape higher education decision making, many schools switch to OERs. Clearly, free is cheaper than alternatives. Clearly, more students, especially low-socioeconomic-status ones, will be better able to afford a textbook and even education in general. But are OERs as good as traditional, albeit costly, resources? It is too early to tell from the research so far.

One of the first reviews of OER efficacy tests included 16 studies (Hilton, 2016). The abstract stated that “ … students generally achieve the same learning outcomes when OER are utilized.” If you stopped there you would be remiss. By the time you get to the results section, you note only nine of the 16 articles that made it into the review analyzed student learning outcomes. The other seven focused only on self-reported perceptions of the material.

All nine studies had major confounds such as method of instruction (e.g., comparing OER sections that were taught online or blended versus traditional texts used in a face-to-face class). Some studies switched exams between comparisons and some changed course design (e.g., went to a flipped model). Most study authors acknowledged that the type of textbook was not the only factor that changed. One study of the 16 minimized confounds by using a common pretest and exams and showed no statistically significant differences in use of an OER for chemistry (Allen, Guzman-Alvarez, Molinaro, & Larsen, 2015).

Astoundingly to the critical reader in me, many studies did not conduct statistical tests on differences (Hilton, 2016). One study reported students using OER performed “slightly better,” but these differences were marginal and not statistically significant. If not statistically significant, “slightly better” does not count.

Another study first reports that OER users had “higher scores on the department final exam” but then notes the exams used by OER users were different from the exams of non-OER users. Again, does not count. The bigger surprise? The authors report there were no analyses performed to determine whether the results were statistically significant. Strange.

Mind you, if there were no statistically significant differences in exam scores between OER users and nonusers, that is good news. It means OERs are as good as more expensive textbooks, and that would be enough for me to consider a switch to an OER. In fact, in testing many educational innovations, “as good as” or “as bad as” is an acceptable version of “not significantly different from.” To get to such a comparison, researchers need to use comparable measures.

Perhaps the biggest issue is that many large-scale studies of educational innovations do not use comparable exams. In one dissertation included in the Hilton (2016) review, Robinson (2015) compared classes across institutions with results spanning the gamut. Student did worse with OERs in some classes or showed no differences. Likewise, Fischer, Hilton, Robinson and Wiley (2015) examined 15 classes with varying results (OER users did better in four, worse in one and were not different in nine), but again without comparable exams. In the one national multisite study using a common exam across classes, OER users did worse than traditional book users (Gurung, 2017).

This issue of benchmarking is most salient in the latest examination of OERs. This summer, I was excited to see a study examining eight undergraduate courses spanning four disciplines (Colvard, Watson and Park, 2018). The sample size was large (21,822 students) and the finding “OER improves end-of-course grades” got my interest. Researchers examined final grades (and other variables) in courses that switched from traditional textbooks to OERs between 2010 and 2016. Students using OERs had higher average scores.

Time to read between the lines. I noted that grades were aggregated over the eight courses and four disciplines. Were courses comparable? Were exams comparable? The article did not say.

When contacted, the lead author stated the data could not be shared. This seemed to be against open science practices, and so I followed up and was told that the information was proprietary and he would check into getting permission. I am still waiting.

Data aside, the big question is the benchmarks used. Did the instructors change their exams to fit the book (OERs or standard books) they were using? The author could not speak to the question. The research methodologist in me cringes. This seems to be a major confound, and research designs need to account for it.

It is also likely that an effective instructor can help students learn from any material equally, so studies having the instructor be constant could explain no differences between books used. An instructor passionate about OER can probably ensure students learn the same or better as they would from a traditional book, a factor rarely discussed.

There is promise in the use of OERs. Beyond the “as good as” findings, some studies suggest they could be beneficial. Jhangiani, Dastur, LeGrand and Penner (2018) found students using print OERs (versus digital) did better on one of three exams tested (no differences on the other two, still good news). Is the promise of OER fulfilled? There is not enough to know yet. We have to be tighter in how we assess the efficacy of such materials in particular and higher education innovation in general.

Methodological challenges abound in classroom research on teaching, as learning is complex. Many challenges can be overcome with strong research design. There are benchmarks for conducting research on teaching and learning (Felton, 2013; Wilson-Doenges and Gurung, 2013), and it would be prudent for more educational researchers to use them.