Wednesday, July 10, 2019

Why we need to read scholarship for ourselves and cannot rely on citation counts alone (Michael Simkovic)

Citation counts and other metrics can be a useful starting point for identifying scholarship and scholars that seem promising.  Such measures are quantitative, sortable, and rankable.  Metrics are quick and easy compared to the time-consuming effort of reading scholarship and forming an opinion of its merit based on expert knowledge of the underlying subject matter.  Some scholars argue that metrics should play a larger role in tenure and lateral hiring decisions—perhaps larger than qualitative assessments.  Metrics appear to be “objective” because they are external to an individual reader (although they are in fact subjective—they reflect many choices about what to cite). 

For the past two years, I’ve served on USC’s appointments committee, and I’ve read many academic articles.  My sense is that on average, scholars who are more highly cited tend to be qualitatively strong as well.  However, citations are a noisy signal of quality—some highly-cited work by highly-cited scholars is deeply flawed.  Conversely, some good work slips through the cracks.

While I wish to avoid embarrassing any particular individual—and will therefore avoid using names—I feel that it is necessary to provide illustrative examples.  I use these examples only because I encountered them recently, not because I have reason to believe they are the most egregious.

One extremely highly cited scholar at a reputable institution claimed that unrestricted e-cigarette marketing would improve public health.  The article did not acknowledge any potential downsides.[1]  

I've observed other scientifically questionable claims in the environmental and health law space.  A well-cited article advocating for more state and local environmental regulation and less federal oversight claimed that most environmental problems are local or state-specific.

I asked an expert on environmental science—a well-credentialed environmental engineer for the state of Vermont—about this claim and she wrote that it is:

“Unlikely. Groundwater, surface water, and contaminated air does not stop at state boundaries.”

It is therefore difficult to address issues like air or water quality purely at the state or local level; spillovers are incredibly common, especially along borders.  Air quality in China can reportedly affect the West Coast of the United States.  Similarly, many migratory animals have habitats that extend across state and even national lines. 

Indeed, the most reasonable presumption may be that there are environmental spillovers unless it can be established that there are not.

Legal scholars are free to take a variety of perspectives on policy matters, but those perspectives need to be grounded in a fair, impartial, and accurate description of the current state of scientific knowledge.  Some highly cited work by highly cited authors does not satisfy these basic requirements of scholarship.

Why is work that is flawed so highly cited? 

I do not know. 

I do know that appointments and tenure committees cannot abdicate our responsibilities to read scholarship closely and to develop a well-informed, expert opinion of it.


[1] My understanding of the scientific literature on this issue is that e-cigarettes may be less harmful than combustible cigarettes but they are substantially less healthy than not smoking at all. It remains unclear if deregulating e-cigarettes leads to switching from combustible cigarettes to e-cigarettes more than it leads to switching from not smoking at all to smoking e-cigarettes.  Given extant bans on cigarette advertising, it would seem that unrestricted marketing for e-cigarettes would be likely to bring in new customers rather than simply cannibalize existing business.


UPDATE July 11, 2019:  Adam Levitin (Georgetown) writes:

"Regarding citations, three thoughts.  First, citations aren’t always for important or original ideas.  Sometimes they are just there because an insecure law review editor wants a footnote to support an uncontroversial fact.  A person who has a very clear statement of something obvious might get a lot of “see, e.g.” citations even for an article that’s entirely mediocre.  That’s something that’s unique to law.  Social science disciplines cite only for substantial points.  I don’t know a good way to filter citations in this regard.  

Second, not all citations are positive.  Some citations are to point out limitations or errors in others’ work, bad ideas, etc.  I’ve been the beneficiary of some of those citations and have given some myself. That’s not unique to law, of course.  

Third, I think it’s useful to look at the type of sources in which someone is cited.  Legal scholarship gets cited in a broader range of places than many other types of scholarship:  other legal scholarship, scholarship in other disciplines, judicial decisions, briefs, and administrative materials.  I think it’s useful to see if anyone outside of the legal academy is paying attention to people’s work.  There’s certainly some great inside-baseball legal scholarship, but we’re a field that is entirely enmeshed with the worlds of the courts and policy, and we should be looking to see whose work is relevant in that regard."

Guest Blogger: Michael Simkovic | Permalink