Westlaw searches: misspellings, multi-author articles and other problems

Greg Sisk gave me permission to share his response to an inquiry (on which I was cc'd) about some deficiencies of the Westlaw database and the searches Prof. Sisk and colleagues perform.  One difficulty that has come up is common misspellings of names; another is that in multi-author articles or books, sometimes (not all the time) only the first author is listed.   Here is what Prof. Sisk had to say, which seems to me sensible:

Your email raises an issue that we’ve seen and thought about every time we do this.  And it affects me as well, as I’ve conducted empirical research that has left me at the end of a list of authors on a piece.

Any methodology has limitations, which we've always forthrightly acknowledged.  The strength of Westlaw is also sometimes a weakness – that it is quite literal.  Being a literal search engine means that if a name is missing in a citation, then a Westlaw search simply will not uncover it.

When it comes to methodology, we have to consider what is practical and possible in a large-scale study involving thousands of tenured faculty members at a hundred law schools and how to implement that in a manner consistent across-the-board.  When we are looking a literally hundreds of thousands of citations for thousands of law professors, we have to rely mostly on a mechanical counting method.

Your email illustrates why we’re unable to integrate a resolution to the et al. issue into our methodology.  To do so consistently across the board, we’d first have to know each individual professor that is affected by this (which of course wouldn’t be apparent in the Westlaw database but would require outside information), and then run not merely one alternative search but potentially multiple alternatives for each type of “et al.” citation.  Within each of those searches, we’d then have to eyeball each of the citations to determine whether it is a correct hit and is duplicative of results from another alternative, and then calculate the right formula for coming up with a final count.  And we’d have to replicate the process across 3,378 law professors.

Importantly, our primary objective is comparison of law faculties, and this issue is not isolated to a particular law school’s faculty.  I've run some test searches in the past -- admittedly on an ad hoc basis and not thoroughly empirical in nature -- and it appears that this problem is vanishingly small when looking at the collective impact of a law school's faculty, which is the central feature of the Leiter Scholarly Impact ranking.  In other words, given that this phenomenon exists at any school with productive faculty, it washes out across the comparison of one faculty to another.  Indeed, as a rough calculation, citation counts for the typical school would have to be under-stated by a few hundred before it likely would affect a school’s overall ranking.

By contrast, for a dean conducting an annual evaluation, it would be quite right for these individual re-calculations to be made to come up with a better count.  Indeed, as we’ve noted, for individual evaluation, one might consult other databases, such as Google Scholar which allows for setting up a profile that, on a case-by-case basis, pulls these citations into one measure.  While that’s not practical for a large-scale study like ours, it may be indispensable for an individual evaluation.

And, because this affects me as well, it is one more reason that I have a policy of insisting with law reviews that citations in my articles include the names of all authors, at least out to three (and sometimes I’ve been able to insist that it go out to four).

Now this is probably far more than you wanted to know.  But I hope it helps explain things and at least shows that we really do take methodology matters seriously and try to think them through.

Of course, in the lists of high-impact scholars in particular areas, this may matter more, though whether it would have significant effects on the results (as opposed to just affecting one or two ordinal placements, which are meaningless anyway) is not clear.

UPDATE:  Ted Sichelman (San Diego) sent me an example, and its effect was to move the author from 4th to 3rd in the law & social science category.  That strikes me as quite minor, but I did make the change.