Mary Dudziak isn't happy with the new citation rankings

Her comments on the "Legal History" listing are here and she felt the need to post a link to her comments here as well after I linked to that post.  Professor Dudziak was actually one of the runner-ups in "Legal History," so this isn't just sour grapes on her part.  But let's see what she has to say and whether it has any merit:

Brian Leiter's rankings are not a true measure of "scholarly impact," especially in a field like legal history. The study is confined to the Westlaw JLR database which only includes legal publications.

The study is a "true measure" of what it purports to measure, namely, scholarly impact in legal scholarship.  It is, as I explicitly note, an imperfect measure (of influence, of quality, of importance, etc.), but that's a different matter.  Whether or not a different database would produce significantly different results is an empirical question; Professor Dudziak appears to assume an answer, but I don't know of any actual evidence supporting her assumption.  Contrary to the impression Professor Dudziak gives, the Westlaw JLR database includes a large number of interdisciplinary journals (including, e.g., American Journal of Legal History and Law and History Review), as well as many foreign legal periodicals (the majority from Anglophone countries not surprisingly).

What does this miss? Leading scholars will have an impact that ranges beyond their fields and beyond their nations. But the Westlaw database cannot measure impact beyond the legal academy, and the important global reach of many American legal scholars is not measured. All but a very few journals in the database are U.S.-based.

Some "leading scholars will have an impact that ranges beyond their fields and beyond their nations" and some won't, so it's silly to generalize.  It will depend on what we mean by "leading," and, more importantly, by the sub-field we are discussing.  Many specialties within legal scholarship are nation-specific which, quite reasonably, means their influence "beyond their nations" tends to be slight. 

It is true that the Westlaw JLR database is a lousy database if one is trying to measure influence "beyond the legal academy."  Should anyone have thought that's what this exercise was about, I hereby reiterate that it is not. 

The impact of interdisciplinary scholars, in particular, will be under-counted. For serious interdisciplinary scholars, especially J.D./Ph.D.s, the true measure of scholarly success is to be seen as leading figure both within the legal academy and within the Ph.D. field. To further one’s scholarship within the Ph.D. field, an interdisciplinary scholar will publish in the field’s leading peer-reviewed journals. If in the humanities and perhaps social sciences, they will publish books.

I am puzzled, again, by the confidence with which Professor Dudziak issues pronouncements about what interdisciplinary scholars aspire to achieve.  No doubt she speaks for some (maybe even the majority), but not for others.  Surely Professor Dudziak knows that there are some non-law disciplines in which the study of law and legal phenomena is not held to be very important or held in high esteem; that is one reason some interdisciplinary scholars might prefer to be in law schools and to write for academic lawyers.

This leads to two under-counting problems. First, the Westlaw JLR database will miss citations to the scholar’s work in journals other than law reviews -- this includes journals in the Ph.D. field.

This is indisputably true, but what does it mean?  On the evidence I've seen in books and non-law journals, I'm probably the most-cited Nietzsche scholar in the English-language secondary literature in recent years, and all of that, alas, counts for naught in my own study!  How sad.  But is it significant?  Not in a study measuring impact in legal scholarship.

For American legal historians, this would include citations in the Journal of American History, American Historical Review, and other history journals. Second, legal scholars often confine their research to the same Westlaw database, and so they don’t find and cite to relevant books and articles.

Fair point:  the case of legal historians may be quite different.  Perhaps if citations in these journals were counted, the top ten list would change a bit (maybe quite a bit, though I'm skeptical about that).  I hope Professor Dudziak will do the study, since this would illuminate the empirical issue she raises.

The limitations of this sort of study are not ameliorated by separating out a field like legal history. Using the Westlaw database will undercount those scholars who have a stronger impact across scholarly journals (beyond those in the legal database), and who do more publishing in books and peer reviewed history articles.  Even a more comprehensive citation study will skew in favor of scholars in larger sub-fields (e.g. American history as compared to medieval studies).

Again, these are empirical claims, that may be true, or may not.  The one I'm confident is true is that the Westlaw JLR database will be skewed, as Professor Dudziak notes, towards American history, which explains the under-counting of extremely eminent and influential legal historians like R.H. Helmholz who work on earlier and non-U.S. periods.

It is also important to point out that Leiter does not count legal historians with appointments outside of law schools. A number of leaders in the field have such appointments.

I'm not sure why it's "important" to point out what should be obvious, given that the study was explicitly confined to law professors, i.e., those holding tenure-stream positions in law schools.  It was so confined because my "law school ranking" site (that's its name) is a source of information for prospective law students, not prospective PhD students in history.

UPDATE:  More thoughts from Professor Dudziak here.