Brian Leiter's Law School Reports

"How to Rank Law Schools"

Here's a draft (Download how_to_rank_law_schools.doc) of my contribution to the Indiana Law Journal symposium on law school rankings (other papers from that symposium have been referenced here and here).  My contribution is largely a commentary on papers by others.  Since I still have time to correct egregious errors and ridiculous statements, I've opened comments and feedback is welcome.

A provocative excerpt for those who don't want to bother to peruse the document:

All of these criticisms [of U.S. News] presuppose, of course, that a ranking of academic institutions ought to reflect certain relevant attributes which serve as a benchmark for critiquing the U.S. News result.  Here I part company with Professor Korobkin, who reiterates his well-known Marxist view that rankings essentially serve a coordination function—allowing good students to find good employers and vice-versa—such that the criteria by which schools are ranked hardly matters.[1]  On this view, legal education is really about pedigree and certification, not education and training; as I once heard a prospective law student put it:  “I'm going to law school to get my ticket punched. Everyone knows you learn the material on your own anyway.”

There is certainly something to this:  if, sotto voce, the Fordham faculty were swapped for the Yale faculty next year, Yale would still continue to produce hugely successful graduates for the foreseeable future.  But that is surely, in significant part, because the Fordham faculty is rather good, and so the real question should be:   what if we swapped, say, the Baylor faculty for the Yale faculty tomorrow?  While the Yale “name” would continue to carry forward for a short while, surely it would not be long before both students and judges and employers noticed that something significant had changed—and not only that Yale students were being taught by folks who actually knew how to practice law!

But what is it exactly that they would notice?  According to Professor Korobkin, it would be nothing that matters to either the students or the employers, and, therefore, the only reason to prefer a ranking that favors the Yale faculty over the Baylor faculty is that we have made a societal value judgment to encourage the kind of scholarly work Yale faculty do.  Perhaps this is right, though I am skeptical.

I am still attracted to the old-fashioned view that those who are smarter and more learned can provide higher-quality instruction.  (I am not saying that this is true of the Yale faculty, though it may be in some cases.)  This is not to say that the best scholars are the best teachers:  that plainly is not true, since there are a variety of pedagogical skills that are unrelated to intellectual acumen.  But it is to say that no set of pedagogical skills can compensate for lack of intellectual depth in one’s subject-matter, and I’m reasonably confident, based on experience on both sides of the podium, that this is true.  That difference may be lost on many students, to be sure, but it won’t be lost on the better ones.  And whether noticed or not, if the old-fashioned view is correct, then it will affect educational outcomes.  With all that in mind, I think an assessment of academic institutions ought to weigh heavily the intellectual and scholarly caliber of the faculty, not to the exclusion of other factors, but as a way of putting education at the center of any evaluation of institutions in the business of educating.

Let me conclude by suggesting four general guidelines for how law schools can be meaningfully and usefully ranked.

            First, rankings of academic institutions should emphasize and reward academic values:  namely, scholarly excellence, pedagogical skill, and student ability and achievement.  It is odd to have to lay emphasis on this, but in an era in which U.S. News ranks schools based on the inefficiency of their spending and their self-reported, and thus largely fictional, job placement statistics, it is, I fear, necessary to state the obvious.

            Second, it is desirable to evaluate law schools along dimensions where there can be measurable change and constructive competition.  Not all the elements of academic value are equally susceptible to measurement, but some certainly are.  If Professor Stake is correct in his contribution to this symposium[2] (and I am persuaded that he is), then one of the many deficiencies of U.S. News is that its reputational surveys of academics are so poorly conducted that they have simply become echo chambers of the prior year’s U.S. News ranking.  But this doesn’t mean faculty quality can not be measured more reliably by better-designed surveys or by the use of “objective” measures like citations.  So, too, measures of student quality in terms of LSAT scores are hostage both to a similar echo chamber effect, as well as the many other factors identifies by Professors Henderson and Morriss in their contribution.[3]  To the extent more academically sound rankings proliferate, serious students will begin making better-informed choices, and rankings of student quality may tell us more than how U.S. News recently ranked particular schools.

            Third, those elements worth measuring should be measured separately rather than aggregated on the basis of unprincipled and unrationalizable schema.  One can rank schools based on SSRN downloads, student LSAT scores, faculty reputation, scholarly impact as measured by citations, job placement, Supreme Court clerkships, and so on, but there is no way these criteria can be meaningfully amalgamated.

            Fourth, we should encourage and welcome many different kinds of academic rankings from many different sources;[4] that is the only way to counteract the excessive influence of the academically unreliable U.S. News rankings.  If the Association of American Law Schools were not in contention for being recognized as the most useless professional organization in the United States, it would have long ago taken the lead in promoting alternatives, instead of giving students the laughable advice that they should discount prestige and reputation in choosing schools.  There is, in fact, a sizable audience looking for rankings that convey genuine academic information:  contrast my own academically-oriented ranking site,[5] which garners upwards of 10,000 hits per week during the peak admissions season and which has been frequently discussed in this symposium, with the bizarre Thomas M. Cooley law school rankings,[6] which contain no useful information and are uniformly ignored by students, faculty, and in most discussions of rankings.  If Professor Korobkin were right, though, then the Cooley method of simply aggregating A.B.A. data without regard to its meaning or importance would have worked as well for U.S. News as the methods it actually adopted, which at least attempt to identify some factors of relevance to legal education.

            Academic rankings that provide actual information on matters of educational value have a useful role to play for students, quite obviously, but they also have a constructive role to play for faculty.  Professor Korobkin suggests that in ranking schools we want to discourage “status competition.”[7]  I guess my own view is more Nietzschean, and so let me close with a quote I have used before.[8]  This is Nietzsche from his early essay on “Homer’s Contest”:

Jealousy, hatred, and eveny…spur[] men to activity:  not to the activity of fights of annihilation but to the activity of fights which are contests.  The Greek is envrious, and he does not consider this quality a blemish but the gift of a beneficial godhead….  The greater and more sublime a Greek is, the brighter the flame of ambition that flares out of him, consuming everybody who runs on the same course….

     Every talent must unfold itself in fighting:  that is the command of Hellenic popular pedagogy, whereas modern educators dread nothing more than the unleashing of so-called ambition…And just as the youths were educated through contests, their educators were also engaged in contests with each other.

We should produce rankings, and more of them, that unleash academic talent and ambition, not rankings that reward decanal connivance at manipulating ranking schemes cooked up by journalists.  Although many of the scholarly critiques of U.S. News in this symposium are devastating, only alternative ranking schemes, that embody academic values we share, will counteract the pernicious impact of U.S. News on legal education.  And, in the process, the right kinds of academic rankings may also stimulate and strengthen our scholarly community in law.

---

[1] Korobkin, supra n. __ at __.  On the Marxian nature of Professor Korobkin’s analysis, see Brian Leiter, Measuring the Academic Distinction of Law Faculties, 29 J. OF LEGAL STUDIES  451, 454 (2000).

[2] [cite]

[3] [cite]

[4] One must note, however, that even in the world of business schools, where there are five different media outlets ranking schools, faculty still bemoan the effect of rankings.  See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=766404

[5] [cite].

[6] See http://www.cooley.edu/rankings/index.htm

[7] Korobkin, supra n. __ at (p. 10 of typescript).

[8] Leiter, Measuring the Academic Distinction of Law Faculties, supra n. __ at 451.

A note on "Comments"

Typepad has a new function whereby all comments must be approved before being published; as a result, there may be a delay between when you post your comment and when it shows up.  So far, I've published almost all the comments submitted, though I have edited some for typos, accuracy, and relevance.  Anyone unhappy with the result should contact me, and I will remove the comment.  I'm being somewhat laissez-faire in terms of "accuracy" and "relevance", at least for now.  I do not intend to let the comments, though, deteriorate in the way they do at most sites.  Thanks to the readers who are making that job fairly easy to date.

Problems with the SSRN Rankings

There was a bit of chatter among law profs about the new SSRN download rankings of law schools and individual faculty awhile back.  (Free registration required for access.)  I noted some of the limitations of this measure at the time.  (My colleague Bernie Black and Cincinnati's Paul Caron [emperor of Law Prof Blog land!] discuss SSRN rankings here.)  Now an anonymous commenter on one of Professor Morriss's recent postings on rankings at the Volokh site has given a particularly good summary of the problems with the SSRN rankings; some excerpts:

1. Economics professors, corporate scholars, and IP scholars use SSRN much more often than other legal professors. Their papers dominate the most downloaded papers lists. While these are interesting fields, they are extremely overrepresented in the ratings.

2. Blogs make a big difference: With no offense given to the fine writers at volokh, of course, who would no doubt be well received without their blog, a mention on one of the more popular blogs can have a very substantial impact on downloads. Take a look at the SSRN page for that "Perfect Crime" article in Georgetown that Volokh posted about a few months ago. It's a fun article, but should it really have far more downloads than anything else Georgetown is publishing this year?

3. Specialty articles make too big of a difference: Articles that rate law schools or law faculties, for example, get a huge number of downloads; typically more than 10-15 good articles at well-rregarded journals get combined. Do these articles have that much more of an academic impact?

4. A few big name scholars drastically skew things: There are a handful of scholars that get more reads than entire other schools. These scholars deserve recognition, but do they actually have more scholarly impact then entire faculties?

5. Older scholars are underrepresented: At least at my school, a number of older scholars are still quite prolific but don't use SSRN unless forced.

6. There's no standard for what to upload: Some professors only upload articles they are still working on and would like feedback on. Others upload articles they wrote 6 years ago, along with their various book chapters.

7. Downloads just aren't the same as actual citations: Setting aside whether even citations mean anything important about scholarship, a citation, at the very least, means that another scholar respected an article enough to reference it in his or her own work.  [Ed.-I doubt citations even mean this.] Downloads don't mean the same thing. An abstract could sell a paper as having an entirely new schema for analyzing an area of law, then the paper could turn out to be garbage. But, SSRN would already reflect the paper as influential. Heck, the attached paper could be a blank .PDF with a catchy title, and it would do well.

SSRN downloads are certainly a very good measure of...who is getting downloaded on SSRN a lot!  But more than that:  some of the problems noted above may wash out across whole faculties, and the resulting lists of individuals and schools does bear, in many instances, a stronger relationship to faculty quality than, say, U.S. News.  And one consequence of creating the SSRN rankings will be that more faculty will be inspired to put their papers on SSRN--and that's a good thing, independent of any ranking.  The important story about SSRN--and this is related to our discussion yesterday about student-edited law reviews--is that it may, in fact, be on the verge of supplanting law review publication altogether.

Red Alert: Fordham Law Review is *Still* Interested in the "Quality" of Articles--unlike Columbia, Cornell, Harvard, Michigan, Stanford, Texas, Virginia et al.

In the annals of ill-conceived solicitations from law reviews, the one I received today (a mass mailing to law professors nationwide) from the Fordham Law Review is likely to be added to the list of "what not to do":

As you may have heard, a group of law reviews, including Columbia, Cornell, Duke, Georgetown, Harvard, Michigan, Stanford, Pennsylvania, Texas, Virginia, and Yale, recently announced their intention to limit the length of articles.  For example, Harvard Law Review will not publish articles longer than 70-75 law review pages "except in extraordinary circumstances."

The Fordham Law Review...disagrees with this policy.  We believe that quality is more important than quantity.  Therefore, we will continue to focus only on merit in choosing what articles to accept.

Yes, indeed, that's exactly what the top law reviews were saying:  "We don't care about quality or merit, we care only about number of pages."  But, thankfully for the verbose and undisciplined, there remains a journal courageous enough to publish unreadable, reinvent-the-wheel tomes posing as articles:  the Fordham Law Review!  What would we do without it?

The decision by the top law reviews to emphasize a strong preference for shorter articles (and 70 pages is not exactly short!) was, of course, driven by qualitative considerations, not quantitative ones:  namely, the entirely correct perception that, with rare exceptions, articles clocking in at 80 or 90 or 120 pages were unnecessarily and gratuitously long, and that their scholarly merit suffered accordingly.   The idea that, in the wake of this decision, only the student editors at the Fordham Law Review will be considering "merit" is, shall we say, a bit amusing.

On the other hand, one smart thing the Fordham Law Review editors do say in this solicitation letter is the following:  "we realize that a student editor's role should be to aid the scholar who wrote the piece, not to burden him or her needlessly."  That's the real mystery about the student-edited law reviews which are, as we've remarked previously, a scandal:  any of the 17-or-so high-profile student-edited law reviews could gain an immediate edge over the competition by adopting the policy of no editorial harassment of authors (no subliterate "editing for style", no demands for meaningless parentheticals, no requirements of "authority" in support of the banal and the trivial, and so on).  Why hasn't it happened?

What to tell the new students on the first day of law school?

Professor Richard Garnett (Law, Notre Dame) seeks advice.

Sextonism Watch: University of Michigan Law School

MOVING TO THE FRONT FROM FRIDAY, AUGUST 12:  NEW UPDATE!

======================================================

And our first "winner" (from a nominator who asked not to be credited) in our on-going effort to purge ludicrous hyperbole from the legal academy is the University of Michigan Law School, which welcomes prospective students on a public web site with the following Sextonian overstatement:

The law comes alive [here] because of our faculty. Our scholar-teachers are of a quality and breadth unsurpassed in legal education. The grand stone walls of our School prove permeable, as for the past several decades our faculty has become internationally renowned for their interdisciplinary approach to the study of law and legal institutions. My colleagues display a broader range of academic interests and expertises than you will find in any other law school.

Since the bolded lines simply aren't true, and are surely known not to be true by everyone at Michigan, why say them?

Nominations for future Sextonism watches are welcome.

UPDATE:  The plot thickens, since alert readers have pointed out to me that the University of Pennsylvania law

faculty is [also!] unparalleled in the depth and breadth of their intellectual interests, the quality and magnitude of their scholarship, and their dedication to teaching.

Moreover, the Harvard Law School faculty is

home to the world's most intellectual interesting and diverse law faculty.

Now somebody here must be fibbing.

And don't forget, of course, that the UCLA law

faculty is [still] emerging as the strongest faculty in the country.

"Emerging," as Professor Brennan instructed us, is, though, a term of art here:

You don't appreciate the meaning of the word 'emerging.' 'UCLA School of Law is *emerging* as the strongest law faculty in the United States' has to be understood in line with such statements as:

'Saddam Hussein was an *emerging* threat to the United States.'

In other words, it's roughly a negation-operator, as in: 'Three is an *emerging* even number.'

Perhaps a similar gloss will explain how Michigan, Penn, and Harvard are all telling the truth about their faculties.

What makes a good law school Dean?

A law professor writes:

Having just started as an associate dean, I have been thinking about what makes for a good law school administrator and remembered your comments on your own dean, when the UT president stepped down - "without a doubt the most talented academic administrator I've ever seen: excellent political instincts, shrewd psychological judgment, strong academic and intellectual values, and an outstanding ability to traverse the worlds of 'theory' and 'practice.'" 

Are there other qualities you would add?  How would you expand on these?  Order them?  Where does fundraising come in? Perhaps you would be willing to address this question on your new blog -- I would love to hear your further thoughts and others' comments.

I've opened comments, and invite others to discuss.  I'll weigh in with my own additional comments a bit later.

On the Leiter Side...

This is pretty funny. 

Legal Historian Hamburger from Chicago to Columbia

Legal historian Philip Hamburger at the University of Chicago Law School has accepted a senior offer from Columbia Law School, to start January 2006. 

Hiring Unemployed Grads as Research Assistants: A Dean Comments

Dean Lauren Robel at Indiana University, Bloomington writes in response to the “hiring grads for short-term employment at graduation” gripe:

I have  looked carefully at the US News questionnaire, the ABA questionnaire that it references, and the National Association of Law Placement (NALP) "Instructions for Compiling and Reporting  Employment Report and Salary Survey Data."  Let me walk you through the convoluted reporting issue, and then make a comment. 

US News asks for employment statistics "at graduation" and at 9 months after graduation (Feb. 15 of the year following graduation).  It references the ABA Questionnaire Part I, Section 12, Question 34B.  However, that section of the ABA Questionnaire only asks for the Feb.15 data.  It does not ask for "at  graduation" data.  NALP's survey does not ask for "at graduation" data either, but it does ask schools to code each student as having received a job offer "at graduation" or at one of 3 other times ( before bar, after bar, and 9 months out).  When asked  (and I am sorry this sounds Clintonian) what a "job" was, Judy Collins at NALP confirmed what the NALP instructions suggest, and I quote, "A job is a job."  (I asked if I could quote her.)  It doesn't matter whether it is legal or nonlegal, or permanent or temporary.  And to complete the loop, the instructions for the ABA questionnaire refer back to the  NALP Graduate Survey Form.

So the bottom line is: There is no collection by either the ABA or NALP of ONLY full-time legal employment at graduation (or 9 months out for that matter, although that latter data is at least segmented in ways from which you could tease out most legal employment).  And the only group that collects anything like at-graduation employment data is explicitly NOT collecting only full-time legal employment.  Thus, the "at graduation" data  reported by every school to US News is only employment, and, as I  confirmed with Ms. Collins at NALP, that can, and should, include  flipping burgers as well doing legal research for law professors.

I think that, for purposes of ultimate legal employment,  students are better off doing research.  Most public school students are going to have to do something to  make ends meet between graduation and full-time legal employment.  From NALP’s perspective, if they are bartending, they are employed.  Cleaning tables---employed. Legal research---employed.  At IU, we ask students to apply for short-term positions on the basis of their debt load and their lack of immediate prospects for full-time employment.  It keeps body and soul together while they are studying for the bar.  As dean, I make zero apologies for this.

Finally, the idea that this is misleading students assumes that "pure" full-time legal employment statistics exist and are requested by US News.  Both assumptions are false. No one reports only full-time legal employment at graduation, because no accrediting or other agency requests it, and neither does US News.   If anything is misleading students, it is the way in which US News presents the employment statistics.  Apparently representatives of NALP met with Bob Morse of US News recently to talk this  through.  Perhaps there will be further developments. 

I will just observe that prior to U.S. News, I am not aware of any law school hiring its graduate as research assistants because they are "better off" doing that than being unemployed.  It is, of course, true that even the A.B.A. and NALP don't distinguish between legal employment and flipping burgers.  Perhaps this point needs to be hammered home to prospective students, who routinely misinterpret the U.S. News data.