Wednesday, August 8, 2018

Should Online Education Come with an Asterisk on Transcripts? (Michael Simkovic)

The ABA recently voted to permit a dramatic expansion of online legal education.

Online education is controversial in higher education.  It is even more controversial in legal education, which relies more on classroom interaction and less on lectures than most forms of higher education. 

Widespread perceptions that online education is lower quality than live instruction in general—and may be particularly disadvantageous in legal education—are backed by numerous peer-reviewed empirical studies.[1] 

Proponents of online education argue that it is more convenient because students and faculty do not have to commute, or because students can learn at their own pace.  They argue that it is potentially more cost effective, either because physical facilities need not be used, or because it is scalable, or because an artisanal model of teaching through knowledgeable faculty can be replaced with a less expensive, industrial model of low-skill specialized workers who each handle particular aspects of course development and teaching.  Some argue that technology can be used to closely monitor and track students, and that the information gathered can be used to improve the quality of education. 

Critics of online education argue that it is lower quality, that most students learn and absorb less, and that the social dynamic of the classroom and learning from one’s peers and interacting with alumni is a critical part of education.  (In addition to multiple peer-reviewed studies, they point to recent examples of “online education” such as self-paced workplace training modules as examples of the low quality that can be expected.) 

Critics point to the failure of MOOCS—which have extremely low completion rates (see also here)—as evidence of the limits of scalability.  They point to the pricing and cost experience of most universities, which have seen high costs of developing and maintaining online courses and additional software licensing fees which have prevented them from charging much less for online classes than for those taught in person.  And they point to a rash of cheating and distracted learning, which anecdotally seem to be more prevalent online than in person.

Perhaps the most empirically rigorous (and recent) study of online education to date—which relied on an experimental design with random assignment of students to different versions of the same introductory economics course—found evidence that “live-only instruction dominates internet instruction . . . particularly . . . for Hispanic students, male students, and lower-achieving students.”  An earlier study which also used a quasi-experimental approach, found similar results, especially for complex conceptual learning:

“We find that the students in the virtual classes, while having better characteristics, performed significantly worse on the examinations than the live students. This difference was most pronounced for exam questions that tapped the students' ability to apply basic concepts in more sophisticated ways, and least pronounced for basic learning tasks such as knowing definitions or recognizing important concepts . . .

Choosing a completely online course carries a penalty that would need to be offset by significant advantages in convenience or other factors important to the student. . . . Doing as well in an online course as in the live alternative seems to require extra work or discipline beyond that demonstrated by our students, especially when it comes to learning the more difficult concepts.”

One early study relying on observational data also found underperformance of students in online education, but attempted to explain this by worse students selecting into online education.  However, more recent observational studies of community college students using instrumental variable approaches have found strong negative effects of online education on learning and persistence, and also evidence that stronger students are the ones who tend to be selected into online courses. 

In other words, online education probably causes even worse outcomes relative to live education than the raw differences in outcomes suggest. This study found similar results in a different college system.

Another study, which also found a large performance gap, found that underperformance of online education was largest for professional programs (law, nursing, and business) and social sciences, and for black students, males and those with low GPAs.

Some embrace the potential of online education to expand access to disadvantaged and underserved communities.  But the evidence thus far suggests that expanding access to underserved students through online education risks substituting an inferior form of education for a superior one.  Underserved students suffer the most when online education replaces live instruction.  As state budgets tighten, online education may seem like a tempting way to hold down costs by reducing quality under the guise of expanding access.

At this point, there is still a great deal to learn about online education and its many variants.  Several universities that offer online programs carefully distinguish them from their traditional offerings, segregating online classes in separate programs that offer separate degrees.

As universities and law schools begin to experiment integrating online courses into mainstream degree programs, should online courses be disclosed to employers by placing an asterisk on transcripts?

Some employers may be indifferent to online education or even think more highly of it than traditional education.  Many employers--particularly those familiar with the peer reviewed literature--may have serious concerns about quality. 

Should universities and schools be obligated to disclose to prospective students what percent of classes they will take fully online instead of with a live instructor?

Do universities and students have an obligation to disclose the format of a course?

Comments are open and moderated.  Real names only please. 


[1] There are many glossy white papers touting the benefits of online education, often funded or produced by media and technology companies and quite a few unpublished papers or non-peer reviewed work.  I cite only to peer-reviewed research, primarily in well-regarded economics journals.

Guest Blogger: Michael Simkovic, Law in Cyberspace, Legal Profession, Of Academic Interest, Science, Student Advice, Television, Web/Tech, Weblogs | Permalink


Just quickly looking at the first article cited, I'd note that its findings are insubstantial are model-dependent. See Figlio p775 ("the overall effect of live instruction relative to Internet delivery is very modest and positive though not statistically distinguishable from zero in the unconditional mean comparisons)..."

Second, it appears that the investigators did not apply modern standards for online curricular designs. Instead they just posted an video recording of a lecture. By contemporary standards, that would be far from optimal, and indeed negligent, as a way to deliver a course online.

Finally, I'd just note that for some students in some settings, the counterfactual is not an in-person course. It is no course.

Posted by: Christopher Robertson | Aug 10, 2018 3:10:41 AM

Hi Chris,

Thanks for commenting.

Re: the first study (Figlio 2013), the author correctly views the regression results as likely more accurate than the unconditional means because the regression corrects for imperfections in the randomization which led to students with higher GPAs and more educated mothers taking the online version of the course. The control variables that the study uses are unremarkable—standardized test scores, GPA in high school and in previous college classes, sex, race, parental education—so it doesn’t seem like the author engaged in p-hacking to come up with a statistically significant result. (College major is probably not available for students in an introductory economics course).

The online class featured problem sets with feedback, TA support, and live exams, the same as a live economics course. The only *difference* was that the lectures were viewed online.

The second study in the post (Brown and Liedholm) used a fancier design for online instruction:

“The virtual course . . . was the product of a staff of professional web-course producers, designers, programmers, and pedagogical experts operating under the direction of [tenured professors with subject matter expertise]. . . ..”

It’s possible that online courses have gotten better since the early 2000s, but more recent studies (cited later in the post) have similar findings. Xu & Jaggers’ studies used actual online courses in college systems through 2008-2009. The Udacity / Coursera San Jose State debacle (circa 2012-2013) speaks for itself.

Most of the serious peer reviewed studies show that students learn less from online education.

Is online education better than nothing? I would guess so.

But that doesn’t mean we shouldn’t flag it on a transcript to distinguish it from traditional education and prevent dilution with an inferior product.

We wouldn’t allow supermarkets to sell hot dogs and call them steak, would we?

Unless we are talking about people who are literally unable to leave the house because of physical ailments, what is keeping them from making it to a live class?

What kind of additional public investment would it take to get more of those people into a physical classroom with good quality instruction instead of calling online education “good enough”? We know it is not currently very good.

Posted by: Michael Simkovic | Aug 10, 2018 8:32:53 AM

Post a comment