Sisk on Citation Studies

Gregory Sisk (St. Thomas/Minnesota) writes with a useful set of reflections on the Phillips-Yoo citation study discussed last week:

Believing as I do both that U.S. News rankings are flawed (and thus should be supplemented by multiple other ranking approaches) and that scholarly impact or quality is multi-dimensional (and thus also benefits from a diversity of approaches), I too welcome any thoughtful new attempt to evaluate the meaning of citations to legal scholarship.  James Phillips and John Yoo have certainly added a thoughtful contribution to scholarly rankings.  At the same time, I think Brian Leiter’s conclusion is right – the Phillips-Yoo approach is not better than the Leiter Scholarly Impact Score method, but rather is different. 

However, one aspect of the Phillips-Yoo method strikes me as mistaken or at least mistakenly characterized.  As a central critique of Scholarly Impact Scores, Phillips and Yoo complain that the method refined by Brian and that we at the University of St. Thomas applied this year are “bias[ed] against younger scholars.”  Isn’t it odd to describe as “bias” the natural tendency of older, more experienced, and well-published scholars to draw greater attention from other scholars?  If Scholarly Impact Scores were calculated over a lengthy time frame, then tired and semi-retired older scholars – what Brian has aptly described as “once-productive dinosaurs” – would gain misleadingly high scores.  But limiting citations to a five-year period – as we do with Scholarly Impact citation ranking – quite  properly minimizes the impact of no-longer productive scholars, because citation levels naturally fall over time without anything new being contributed.  In general, the fact that an experienced and still active older scholar draws greater attention based on the larger portfolio of work available to be cited in the past five years is hardly a bad or irrelevant thing.

Along the same lines, I don’t know about the wisdom of taking the actual and objective data of current citations and then recalculating scores on the basis of longevity among a particular law faculty.  Unless one is careful to explain that this longevity depreciation factor is being used to separate out and identify up-and-coming young scholars (or to rank schools that have more promising younger scholars than other schools), one could characterized this method as genuinely biased in the opposite direction, that is, against older scholars.

But the better point is not that one or the other is biased so much as that they are doing different things – prediction of the future versus description of the present.  As I see it, Phillips and Yoo seek to devise a method of predicting the likely future scholarly impact of younger scholars (which is commendable and intriguing).  But their introduction of a longevity depreciation factor should not be understood as an improvement on our measurement of current scholarly impact (which it is not).

Let me explain what I mean by an example.  Suppose that Professor A, a recently tenured scholar, has published only 3 articles, each of which has been cited 100 times over the past five years, for a total of 300 citations.  Professor B has an additional ten years of experience as a tenured faculty member, has published 12 articles, each of which has been cited 50 times over the past five years, for a total of 600 citations.  If I understand them correctly, Phillips and Yoo apparently would conclude that the
scholar with the greater impact is Professor A, because each article individually drew more citations and because the number of years in teaching is fewer.  But, if we are measuring which scholar today has a greater scholarly impact, doesn’t the reality remain that it is Professor B?  The authors of 600 articles saw Professor B’s body of work as worthy of citation, while the authors of half as many articles reached that conclusion with respect to Professor A’s work.

Now Phillips and Yoo may be on to something important in predicting that Professor A is more likely to be the more prominent scholar in the future.  Their description of scholars like Professor A as more “relevant” may be shorthand for “making a prediction of the future prediction.”  Of course, it is possible that Professor A will not live up to the prediction, because he fails to remain productive, because his three articles prove to have exhausted his abilities and nothing afterward has the same scholarly luster, or because his work remains of the same high quality but he has saturated the scholarly interest in his particular scholarly message and thus he experiences diminishing returns in citation to his future articles that are along the same vein.  In fairness to Phillips and Yoo, however, those disappointing possibilities simply reflect that, by focusing on longevity and citations per article, they are attempting to predict the future and any prediction includes an element of uncertainty.

By contrast, the 600 citations garnered by hypothetical Professor B over the past five years is not a prediction but a present reality.  Whether by exceptionally prolific writing or diligence in promoting a point of a view through a series of articles or something else, she has succeeded in drawing the attention of the authors of 600 articles.  To dismiss or dilute that accomplishment by constructing a depreciation formula that incorporates number of years in teaching is to ignore the reality of the current impact.  Again, if our purpose is predictive, we might prognosticate that Professor B’s influence will decline or at least be surpassed in the future by Professor A.  But as a description of the present scholarly impact, haven’t the authors of 600 articles in my hypothetical already reached a definite conclusion?

As a final note, the Phillips-Yoo study appears mostly to provide confirmation of the Scholarly Impact Score method, as the changes in rankings among the 16 schools studied are mostly modest.  Moreover, because I expect that, however defined, All-Stars and Super-Stars would make up decreasingly smaller percentages of law faculties as one moves down through the ranking, the Phillips-Yoo method is likely to have decreasing significance as one moves from the 16 schools they chose to study to include the larger sets of 96 law faculties we studied in 2012 (and the full 200 law faculties studied in
2010).