A Natural Experiment

(Before saying anything, I just want to extend well wishes to friends and colleagues on the West Coast. The news coverage has been surreal, and I can only imagine the lived experience of the fires and the orange skies. We’re all rooting for you!)

What’s the optimal allocation of sections between synchronous and asynchronous?

At this point, I think the only responsible answer is “we don’t know.” There just isn’t good enough information yet. But there could be.

My fellow admins out there -- make sure you talk to your institutional research office about tracking enrollments and success rates in the various formats this semester. We’ve found ourselves in what scientists call a natural experiment, and it would be a shame to let it go to waste.

In calling it a natural experiment, I mean that the situation wasn’t built to be an experiment. In this case, it was built to work around a pandemic. But in the course of working around the pandemic, we happened to create a potentially useful testing ground to measure the strengths of the two formats against each other.

As with any natural experiment, it’s imperfect. To the extent that students were offered options, they may have gone with the format that worked better for them. A student who craves structure and real-time interaction might prefer synchronous, whereas a student with a fluid schedule and many commitments may prefer asynchronous. That makes direct comparisons harder. If, say, more academically needy students gravitated toward synchronous classes, then lower success rates in them (if that happens) may reflect the population more than anything inherent in the format. In other words, the performance of students who chose modality A may be better in modality A than it would have been in modality B, even if modality B has higher success rates over all. That’s the sort of question that requires a deeper analysis than just a quick look at aggregate pass rates.

The distribution of sections isn’t necessarily even across disciplines, which confounds matters a bit. And many colleges have far more experience with asynchronous online teaching than with synchronous. My own, for instance, has taught asynchronous online for 20 years and has several degrees available entirely in that format. Its first foray into synchronous online teaching was in March, under duress. This is the first semester in which courses are being taught in the “remote live” format from the start. Comparing success rates for a format we’ve used for 20 years with a format in its first iteration may be misleading. It may take a little while to work out some of the bugs of the new format.

Still, all of those caveats granted, this is a rare institutional learning opportunity, even if it’s one we wouldn’t have asked for. Assuming that the pandemic will subside at some point, it would be silly not to come away from the experience not having learned anything. Is there a cohort of students (or a set of classes) for which remote live works especially well? Are there certain activities for which it’s especially well suited?

In the Before Time, I had seen some literature on the “online paradox,” which says that although on-site classes have higher pass rates than online classes, students who take a mix of formats do better than students who take only one format. Whether something similar is true here would be worth knowing.

In the short term, in the absence of evidence one way or the other, we’ve made a point of offering a good mix of sync and async. If we had good evidence, I’d be happy to follow it, but we’re not there yet.

So, my quick note to administrative colleagues everywhere: if you haven’t yet talked to the institutional research folks, this would be a great time. If you wait too long, you’ll wind up with answers like “we didn’t capture that information.” Now’s the time.