Henderson and Morriss Reply Regarding U.S. News

I invited Professors Henderson and Morriss to reply to my criticism of their defense of U.S. News; their comments follow:

As Brian notes, many of the underlying inputs used by U.S. News, including the employment numbers (which are supplied by the schools to the ABA), are remarkably unreliable. This is an embarrassing problem that only the law schools and the ABA can solve.  Yet, because of its dominant position, the value that U.S. News provides to applicants does not necessarily hinge on high quality data.

Specifically, in terms of employment prospects, U.S. News helps to coordinate the market for human capital:  By enrolling in a highly ranked school, a student signals to employers that he or she has impressive intellectual ability; and employers in the market for that talent know where recruit.  See Korobkin (1998); Korobkin (2006).  Although we agree that overall U.S. News rankings cannot be defended as a measure of educational quality, the rankings are nonetheless correlated (albeit imperfectly) with future employment options.  This information is clearly valuable to prospective students contemplating a 3-year, $100K+ investment.

We believe a lot of vitriol over the U.S. News methodology (which we saw firsthand at the Rankings sessions at the 2007 AALS meeting) is not really about the methodology, but rather the discomfort of being ranked at all.  Indeed, U.S. News has ratcheted up the competition for students.  Law schools can and should provide all information available to aid students in deciding how to make a major investment of time and money.  (And the ABA has recently made a giant step in this direction.  See here. [Ed.--see also here.])  With more information in the public domain—and perhaps more alternative rankings—prospective students will have the ability make more informed and targeted enrollment decisions.  This dynamic will make U.S. News less influential; we think this would be a very good development.

If the U.S. News rankings went away tomorrow, law faculty would be better off, but not prospective students.  Indeed, by placing its methodologically flawed ranking system into the stream of commerce, U.S. News is forcing the legal academy to disclose more information.  (Note that in the late 1990s, the ABA-LSAC Official Guide was made more comprehensive and uniform in partial response to the outcry over the false information that was being submitted to U.S. News.) On that criterion, we are happy to defend the U.S. News rankings.

Two further thoughts of my own by way of reply.  First, the Korobkin argument about the coordination function served by rankings (even a set of rankings based wholly (or in the case of U.S. News, partially) on nonsense criteria could perform such a function!) works only if those in the market for human capital--the sellers (the students) and the buyers (e.g., the elite firms)--pay attention to the rankings.  There seems to be good evidence that the sellers pay attention, but I don't know of any evidence that the buyers do.  (Maybe such evidence exists?)  My impression--admittedly anecdotal, but accumulated over a long period of time--is that almost all legal employers operate with their own "internal" rankings of schools based on past experience, and that this trumps U.S. News and anything else.   U.S. News only appears "imperfectly correlated" with these job market realities because it uses enough measures that track professional opinion among insiders.  Second, if U.S. News stopped ranking schools tomorrow, how exactly would students be worse off?  The job placement data the magazine reports is not reliable.  There are now lots of ways to figure out that Columbia provides better job opportunities than, say, Cardozo (but did students really need U.S. News to figure that out ever?).  So what exactly is lost?  True enough, U.S. News will send some students to Washington & Lee over Illinois based on ranking differentials, but will they have really performed a service or simply misled students who might have made a better choice in the absence of the U.S. News nonsense number? 

Comments may take awhile to appear; post only once.  Signed comments are far more likely to appear, as usual.

Comments

[Jason Ratigan]

I am an entering 1L at W&L and vividly remember the application process and can confidently say that the process would be extremely difficult if it were not for US News. The most difficult aspect of the application process for me (a student with few family members actually going to college, let alone law school) was the very start (i.e. Where do I apply?). Prior to taking the LSAT, I had put together a list of the usual suspects, but after the LSAT found that those were going to be a waste of 80$ each. Where to turn?

Luckily I did well enough in undergrad and, only just, on the LSAT for your rankings to be of use, but if I had not (say with 3 points lower on the LSAT, or as the vast majority of test takers had done) they would have been less valuable to me. Then again, it is possible that without US News, admissions offices might actually be interested in information deeper than a test, gpa and 600 words in assessing my value, but I doubt that accuracy would somehow trump speed and ease (I was tempted to say laziness, but thought that may be unfair to admissions offices generally).

Although I am in full agreement that US News is more of a negative influence than a positive one (especially with respect to admissions offices priorities), I feel that it is certainly a valuable tool to prospective students as things stand today. However, if US News were substituted for an interactive, composite ranking system in conjunction with a reputable specialization ranking--not only as a valuable tool in itself, but also to soothe the self-respect one might lose going to a school outside the "usual suspects"--the world may be a better place.

Jason Ratigan
Entering 1L at W&L

[frankcross]

If the sellers pay attention to the rankings, it doesn't really matter if the buyers do, does it? So long as they know the sellers are paying attention. Assuming the criteria for admission are relevant, the students would be selecting themselves into discrete talent pools. The buyers wouldn't pay attention to the rankings, they'd pay attention to the fact that the sellers pay attention to the rankings.

[Brian Leiter]

Frank, this just pushes the question back one level: is there any reason to think that the buyers are paying attention to what the sellers are paying attention to? Perhaps there is an indirect effect insofar as buyers adjust their assessment (their "internal" ranking of schools) as sellers adjust their behavior to coordinate with U.S. News. But this is all pretty speculative, not sure if it has any basis in reality.

[Bill Henderson]

Brian,

Thanks for posting this. Re your point on coordination and the need for the "buyer" to care in order for the theory to have traction. That is a good point. We have three pieces of evidence that collectively suggest that buyers (i.e., employers) pay attention to rankings:

1. We predicted 84% of the variance in Am Law 200 OCI interviews through 1) school size (larger schools give better odds to employers), 2) median LSAT, 3) Top 16 status, 4) Tier 1 status, 5) geography. If rankings were irrelevant to employers, inclusion of the LSAT (correlated at .9 with rankings) should render the coefficients for Top 16 and Tier 1 statistically insignificant. That was not the case. See Table 4 of our 2006 Rankings paper. Besides LSAT and school size, we cannot think of any other attributes that are correlated with USNWR rank that would logically affect employer behavior. So, we think it is probably rank that is producing this effect.

2. The Large Firm sphere is growing faster than the supply of elite students. Firms need to go beyond conventional / traditional past practice to fill hiring slots.

3. BCG Attorney Search, one of the largest legal recruiters in the nation, prepares an annual recruiting manual (280+ pages), available to any law firms, that provides resume parsing information on the Top 50 school. They use US News to make the cut off.

None of this is conclusive, but collectively, it is suggestive (I think strongly so) that employers care about rankings.

Re your second point on how students would be unaffected if US News went away: it is important to note that this is a dynamic process--the ABA just released a boatload of data largely because of efforts of researchers who study rankings and argued that more information and transparency was needed. I do think that US News leads a lot of students to make very bad decisions. But more information in an accessible format will put US News in its place. None of this would be happening without US News in the first instance.

Perhaps we should have said "If US News had never existed … " rather than "went away tomorrow … " That would be closer to our point.

bh.