How Will Unresolved Research Questions Get Answered?

Quite a few of the biggest questions facing higher education in 2018 -- particularly those involving digital technology -- could be answered the same way: “We need more research on that.”

In what combination of classroom settings do students work best? What sorts of instructional approaches work best with which groups of students? How can instructors reproduce face-to-face experiences in digital environments? How can technology expand access to education and diversify the pool of students heading into the work force?

Digital learning thinkers hope research can help them find, or at least approach, the answers. But all kinds of roadblocks keep getting in their way, and the solutions to overcome those roadblocks haven't yet materialized.

From a logistical standpoint, institutions often don’t have the bandwidth to fund research projects as ambitious as the ones necessary to tackle overarching questions. Private foundations and other external organizations, meanwhile, have their own priorities and limited grant budgets.

The challenges aren’t limited to a shortage of resources. There’s also a shortage of understanding -- administrators who aren’t willing to share student outcomes data with researchers, technology that evolves more rapidly than the deliberate pace of research efforts, academics who haven’t landed on the most nuanced questions to ask. Some leaders in digital learning research believe the lack of a formal discipline on such subjects means studies thus far have been scattered and incomplete.

“The lucky thing is we do have faculty and they’re figuring it out on their own. [But] it’s a misuse of resources,” said Tanya Joosten, director of digital learning research and development at the University of Wisconsin at Milwaukee. “We could be much more effective at our jobs as faculty, instructional designers, digital learning leaders if we had these sorts of data and findings that would better inform our practices.”

Shifts in the Funding Landscape

Digital learning research got an aggressive kick start in the early '90s from the Alfred P. Sloan Foundation, which has since rebranded as the Online Learning Consortium. Researchers working today talk in glowing terms about the foundation’s interest in funding digital learning initiatives that would later serve as test cases for new formats and modalities.

But when the foundation’s longtime president, Ralph Gomory, stepped down in 2007, the organization’s focus eventually shifted, ceding ground to organizations like the Bill & Melinda Gates Foundation and the John D. and Catherine T. MacArthur Foundation, which see education innovation as part of a larger effort to drive broad societal transformation.

Those groups’ pockets are deep but not infinite. And they’re not always on the same page with academics. Anthony Picciano, a professor in the School of Education at Hunter College and at the City University of New York Graduate School’s Ph.D. program in urban education, said he’s recently shied away from working with foundations like Gates, as well as technology companies including publishers, because he often finds that their goals aren’t in line with his ambitions.

Picciano believes the Gates Foundation is more convinced than he is that advanced technology will solve existing educational problems. Companies, on the other hand, are always invested above all else in demonstrating the value of their products, he said. After a few instances in which companies wanted to produce reports based on data he collected, Picciano -- and some of his colleagues, he says -- no longer collaborates with such entities.

“I’m a full professor, I get a decent salary, I do what I like,” Picciano said. “I can walk away from contracts.”

The Gates Foundation sees its role as helping to “develop a more sophisticated understanding” of the nuances of new forms of teaching and learning, according to senior program officer Rahim Rajan. The organization doesn’t have a “separate strand of budget around pure applied research,” but rather incorporates efficacy research into the larger package of investment in key spaces, Rajan said.

“I don’t think we will be the only ones who can fund that. I don’t think we can,” Rajan said. “Some of it is not just funding the research -- some of it is actually the practice.”

In Rajan’s view, the Gates Foundation gives researchers wide autonomy to study as they see fit. He and his team have been surprised to find that institutions aren’t inherently adept at delving into their own teaching and learning practices. “This is not an innate capability that higher ed has,” Rajan said.

Picciano has been working for the last two years on a U.S. Department of Education project to assemble a meta-analysis of research on best practices for online learning. He’s been pleased thus far that the department’s only mandate is for all research included in the final product to meet its strict standards -- random selection, pre- and post-tests, and high internal validity experimental designs.

The department, along with the National Science Foundation, has made sporadic forays into funding research into technology-enabled learning, but only in a limited capacity. At Wisconsin-Milwaukee, Joosten leads the National Research Center for Distance Education and Technological Advancements (DETA), established in 2014 with a $1.5 million grant from the U.S. Department of Education.

The grant was the first and only of its kind nationwide. Joosten recalls hearing from the department that she would receive another $1.5 million at some point, but that extension never materialized, and funding has dried up.

“A lot of other folks are struggling out there to become self-funded,” Joosten said. “What a funding agency thinks is important and what we think is important may not always be the same.”

Joosten is grateful for the grant, but working for DETA hasn’t been entirely smooth sailing, she said. One of her main goals has been to synthesize student outcomes data from numerous institutions. But in many cases, she’s found, institutions hesitate to share student data with researchers, even with an assurance that it won’t be used for anything other than aggregation, and that it won’t fall into the wrong hands.

For recent cross-institutional studies, Joosten reached out to more than two dozen institutions and had interest at the director level from several. But in most cases, only a handful of institutions ultimately participated.

“We don’t have the capacity right now to have generalizability of findings,” Joosten said. The one exception was a study on academic success coaching for competency-based education, which mainly involved interviewing staff rather than tapping into student data.

Federal grants often go to experimental design projects, leaving less room for other types of studies, according to Katie Linder, research director at Oregon State University Ecampus. She once collaborated on a proposal for an ethnographic approach to understand system-level approaches to implement educational technology, but it fell outside the scope of funders’ priorities.

Determining the Scope of Research Efforts

Digital learning is such a broad topic that people from many different backgrounds are looking at it in different ways, Joosten said. Although localized studies occasionally pop up, she thinks this research area needs an overall strategy -- but it’s unclear who would provide that direction.

“This is very different from any other discipline,” Joosten said. “It’s not like you are going to have a sociologist having botany studies.”

Academics who care about digital learning are taking research matters into their own hands in various ways. Linder has been noticing for a while that many digital learning research projects are limited in scope to one institution or a small portion of a university.

Linder manages a database that aims to serve as a comprehensive set of efficacy studies around online learning, going back more than two decades. A recent search of the database revealed that fields like communications, criminal justice, foreign language, physical therapy, public health and sociology all have only one published academic article related to online learning.

“That is not something that is reassuring to faculty who are being asked to engage in online teaching and learning,” Linder said. “I do think we have some work to do to help bring people on board.”

Unsatisfied with the pace of progress in this area, Linder and a colleague, Rob Nyland, research and innovation team manager at Boise State University’s Ecampus Center, have created a new initiative: the Online Teaching and Learning Research Seminars program, which will bring together three cohorts of 10 to 12 researchers to combine expertise and collaborate on broad research projects. Each cohort will focus on a theme. The first will tackle learning analytics, and the other two have yet to be determined.

Applications for the program are due Nov. 30, with work set to begin next summer. Linder hopes to see a mix of instructors, instructional designers and administrators with a shared interest in these topics. The program requires some research experience -- it’s not meant to serve as training, but rather as a meeting of the minds asking big questions and combining resources to seek answers.

Oregon State Ecampus has set aside $50,000 for the cohorts to travel to the institution and to compensate Linder and Nyland for their efforts as co-directors.

Linder hopes this program will streamline the protracted process of setting up research projects, particularly ones funded by external grants. Not only is the wait between applying and securing funding often long, the collection of pilot data can take one to two years to complete. If the funder doesn’t extend its contribution, those efforts can go unrealized, or the technology can move on.

“We want to make sure we have time to do the research itself,” Linder said.

Studies conducted among multiple institutions have some potential to broaden available data sets on digital learning, but because each institution defines data differently, "they are often easier to talk about then to get off the ground," Linder said.

One area of higher education that could use more investigation, researchers say, is the for-profit portion of the sector. Several for-profits are among the nation's most attended online institutions, but learning more about for-profit education providers has proved challenging for Eric Bettinger, a professor of economics at Stanford University. He published a paper last year that examined academic offerings at a major for-profit institution.

Bettinger thinks for-profit online institutions can be particularly valuable sources of knowledge because they keep such intricate records of students’ interactions with their products.

“They can identify attendance every day, assignments and study habits,” Bettinger said. “Students are constantly going to some type of electronic record.”

Finding funding for that project wasn’t easy, though. When he contacted the Department of Education’s Institute of Education Sciences and entered other grant competitions, he routinely heard accusations that his efforts were essentially serving as marketing tools for for-profits. His graduate students ended up working on the project for free.

“We just found that the perception of for-profit colleges was so bad was that nobody wanted to fund our study,” Bettinger said.

What's Next

The rapid pace of evolution may be tricky to navigate from a funding standpoint, but it’s also pushing researchers to be more nimble, Linder said.

“There might be a technology tool you want to measure in your classroom, and then next year that technology tool doesn’t exist or its features look entirely different,” Linder said. “I think that’s really kind of helped to develop the methodology of design-based research, and to help you figure out in a very iterative way what’s working, what’s not working.”

Peter Shea, associate provost for online learning at the State University of New York at Albany, has been conducting research in this space since the 1990s. He believes a “full-blown discipline” around digital learning hasn’t yet developed, which means much of the research is being conducted outside the traditional peer-reviewed publication context that serves as the foundation for most social science inquiry.

Aside from a few stray journals like the International Review of Research in Open and Distributed Learning and the Journal of Online Learning, Shea said he’s seen more research coming from people who aren’t necessarily grounded in traditional research principles.

“It’s an imperfect system because the downsides of that are the loss of what’s traditionally been a vehicle for the mechanisms of quality,” Shea said.

Chuck Dziuban, director of the Research Initiative for Teaching Effectiveness at the University of Central Florida, is thankful that his institution has prioritized "data to information to insight to action," allowing him and colleagues to widely publish studies on key issues around the transforming classroom experience. But he feels some other institutions haven't embraced that mind-set. Finding creative solutions to these problems is critical for the future of the sector, Dziuban said.

“We are wasting millions of minds in this country. We must figure out a way to bring out all the talent from our country,” Dziuban said. “We can’t continue to afford to do the kinds of things we’re doing.”