Brian Leiter's Law School Reports

Fortune's Best Graduate Degrees Based on Earnings, Job Satisfaction, and "Stress"

An interesting, and not implausible, list.  The JD comes in 6th, though most of the other options are unlikely to be pursued by an undergraduate humanities major--one reason, among others, why we have probably hit bottom in terms of the applicant pool and will probably see a slight uptick in the next couple of years.

Geography Matters (Michael Simkovic)

A number of critics have argued against extrapolation from Professor Merritt’s study of the Ohio legal market to the national legal market.  In her response, Professor Merritt makes some good points, and also several key points with which I disagree. 

Professor Merritt suggests that an important contribution of her study is providing up-to-date information about national legal employment through the prism of Ohio. However, there is no shortage of up-to-date data that can provide a more accurate picture of national trends than a study specifically focused on Ohio.*  The primary value of Professor Merritt’s study is as an isolated snapshot of a single cohort in Ohio at a particular point in time.  Without additional information, it is hard to know how much, if at all, Professor Merritt’s findings should be generalized to other legal markets or other time periods.

There is no reason to believe that the single Ohio cohort tracked by Professor Merritt will better predict outcomes for those currently enrolling in law school than a national cohort.  The single Ohio cohort will likely be less predictive than a long-term national average across multiple cohorts.  Indeed, as Professor Merritt acknowledges, her study is not a study of going to law school in Ohio because of selection issues from law graduates leaving for larger markets, coming to Ohio from other markets, and from non-bar passage. **

Year-to-year changes in employment, earnings, and economic growth can vary widely from state to state.  Absent evidence of a history of correlated economic activity, a single state should not be used as a proxy for the U.S. as a whole or for other states.

There is no reason to believe that the trajectory of Ohio’s legal market from year to year will closely track national trends, particularly when the national legal market is heavily concentrated elsewhere.  Washington D.C. and the top 5 states by size of legal market*** collectively account for more than half of the national legal market. 

If Professor Merritt wishes to use Ohio as a proxy for the rest of the U.S., then she should supply evidence that Ohio tracks national trends, and she should compare Ohio to Ohio at different points in time and Ohio to the U.S. at the same point in time.

Second, Professor Merritt suggests that focusing on Ohio is just as reasonable as focusing on New York or California.  New York and California collectively constitute 28 percent of the national legal market.***  Ohio constitutes 2.5 percent of the national legal market.  Moreover, the New York legal market is unusually large relative to the New York economy, while Ohio has a legal market that is small relative to its economy. 

Third, Professor Merritt suggests that Ohio can be made nationally representative by deflating salaries elsewhere by cost of living differences.  Cost of living differences are not the reason corporations—who can send legal work anywhere— pay a premium for lawyers in the major legal markets such as New York, D.C., Los Angeles, Boston and Houston.  Rather, corporate clients believe that differences in quality of work justify higher billing rates for important matters.  New York, D.C. and other high-paying markets are importers of top legal talent from across the country. 

Differences in costs of living are not random, but rather reflect real differences in quality.  Cost of living indexes often focus on quantitative rather than qualitative factors.  For example, a restaurant meal in Manhattan may cost more than a restaurant meal in Buffalo, but the quality of the experience in the restaurant in Manhattan will on average be higher because the high prices restaurants in Manhattan can charge will attract the most talented restaurateurs.  Similarly, there may be differences in the quality of healthcare, legal services, education, policing, parks and recreation, environmental safety, transit, housing and other factors.  Money attracts talent.  Some amenities or opportunities may only be available in particular locations, and people are willing to pay for proximity to consumption, employment, and social opportunities. 

Many costs are not local, but rather national.  These include automobiles, items ordered online, higher education at major universities, and investments (stocks, bonds, etc.).  For law school graduates—who will typically be able to earn far more than they consume in a given year—it is financially better to work where both income and costs are proportionately higher because this will maximize the dollar value of savings.  Law graduates can always retire to a lower-cost location later in life if they wish.

One quantitative measure for differences in quality of life is differences in life expectancy.****   High cost, high income, high infrastructure states like New York, Connecticut, and Massachusetts generally rank well on this measure, while lower cost, lower income states rank less well.  This pattern can also be seen internationally and individually—higher income and higher life expectancy are correlated.*****  

There will indeed be some lucky individuals who find low-cost locales both more attractive and less expensive, and some unlucky individuals who find high cost locales unworthy of the price.  Costs of living reflect the aggregation by the market of many individual preferences, not any particular person’s idiosyncratic views.  Nevertheless, local prices can contain important information about quality of life that we should not assume away.

* There are numerous sources of up-to-date (2013 or even 2014) national information, including data from:

• the U.S. Department of Labor Bureau of Labor Statistics (BLS) 
• the U.S. Census Bureau’s
  • American Community Survey (ACS),
  • Current Population Survey (CPS), and
  • Survey of Income and Program Participation (SIPP),
• the National Association for Law Placement (NALP), and
• from the American Bar Association (ABA) 

NALP and ABA data are for the most recent graduating class shortly after graduation.  SIPP earnings data includes earnings as recently as 2013, but only through the class of 2008.  ACS and CPS have young lawyers and young professional degree holders, but cannot specifically identify young law degree holders.  The Department of Education also has information on student loan default rates for recent cohorts.  Default rats remain much lower for former law students than for most other borrowers.

Another valuable source of information is After the JD III.  Professor Merritt notes that response rates for higher income individuals may be higher in After the JD, but the After the JD researchers, like the U.S. Census, weight their sample to take into account differences in response rates.

**The selection bias issues may be more severe than Merritt has acknowledged.   Looking at Ohio State’s 509 report for 2011, there were 24 students who took the NY bar vs. 136 who took the Ohio bar—a substantial percentage of the class taking a bar in a non-adjacent state.  The New York bar takers had much higher bar passage rates (11% above the state average for N.Y. vs. 1.3% above the state average for Ohio), which is consistent with positive selection out of state. In any given year, roughly 25 to 50 percent of Ohio State law school graduates who are employed 9 or 10 months after graduation are employed outside of Ohio.  For Case Western graduates, employment seems to be even less Ohio-centered than Ohio State.

*** Size of the legal market calculated using ACS data, multiplying number of lawyers by average total personal income per lawyer to get aggregate pay to all lawyers.  In other words, the measure is a dollar count, not a body count.

**** It is probably preferable to consider life expectancy within race (life expectancy varies by race, and racial demographics vary by geography).

***** After controlling for GDP per capita, societies with less income dispersion tend to have higher life expectancy.  Another issue is selection effects vs. causation.  For example, those with higher life expectancy to begin with may choose to pursue additional education and therefore have the opportunity to live in high cost, high income states.

Northwestern Law to pick up one year of loan interest...

...for some graduates.  Interesting idea.

In Memoriam: Greg Lastowka

Professor Gregory Lastowka of Rutgers-Camden Law passed away last night, after a year-long battle with cancer.  He was 46.  Greg was a serious scholar - writing across areas of Cyberlaw and IP - but was also a particularly humane colleague.  He is missed.

New York Times relies on unrepresentative anecdotes and flawed study to provide slanted coverage of legal education (Michael Simkovic)

Just when you thought The New York Times was rounding the corner and starting to report responsibly about legal education based on hard data and serious labor economics studies, their reporting reverts to the unfortunate form it has taken for much of the last 5 years*—relying on unrepresentative anecdotes and citing fundamentally flawed working papers to paint legal education in a negative light.

Responsible press coverage would have put law graduate outcomes in context by noting that:

(1) law graduates continue to do better in terms of employment (both overall and full time) and earnings than similar bachelor’s degree holders, even in an economy that has generally been challenging for young workers

(2) law students, even from some of the lowest ranked and most widely criticized law schools, continue to have much lower student loan default rates than the national average across institutions according to standardized measurements reported by the Department of Education

(3) law graduate earnings and employment rates typically increase as they gain experience

(4) Data from After the JD shows that law graduates continue to pay down their student loans and approximately half of graduates from the class of 2001 paid them off completely within 12 years of graduation

Instead, The New York Times compares law graduate outcomes today to law graduate outcomes when the economy was booming.  But not all law graduates.  The Times focuses on law graduates who have been unusually unsuccessful in the job market or have unusually large amounts of debt.  For example, The New York Times focused on a Columbia law school graduate working as an LSAT tutor** as if that were a typical outcome for graduates of elite law schools.  But according to the National Law Journal, two-thirds of recent Columbia graduates were employed at NLJ 250 law firms (very high paying, very attractive jobs),*** and the overwhelming majority of recent Columbia graduates appear to work in attractive positions.   (Columbia outcomes are much better than most, but the negative outcomes discussed in The New York Times are substantially below average for law graduates as a whole).

In Timing Law School, Frank McIntyre’s and I analyze long term outcomes for those who graduated into previous recessions, using nationally representative data and well-established econometric methods.  Our results suggest that law graduates continue to derive substantial benefits from their law degrees even when graduating into a recession.  The recent recession does not appear to be an exception. (See also here and here).  This analysis is not mentioned in the recent The New York Times article, even though it was cited in The New York Times less than a month ago (and alluded to in The Washington Post even more recently).

The implication of The New York Times’ story “Burdened With Debt, Law School Graduates Struggle in Job Market” is that there is some law specific problem, when the reality is that the recession continues to negatively affect all young and inexperienced workers and law graduates continue to do better than most.   Law school improves young workers’ chances of finding attractive employment opportunities and reduces the risk of defaulting on debt.  The benefits of law school exceed the costs for the overwhelming majority of law school graduates.

The New York Times relies heavily on a deeply flawed working paper by Professor Deborah Merritt of Ohio State.  Problems with this study were already explained by Professor Brian Galle:

“My problem is that instead DJM wants to offer us a dynamic analysis, comparing 2014 to 2011, and arguing that the resulting differential tells us that there has been a "structural shift" in the market for lawyers.  It might be that the data exist somewhere to conduct that kind of analysis, but if so they aren't in the paper.  Nearly all the analysis in the paper is built on the tend line between DJM's 2014 Ohio results and national-average survey results from NALP.  

Let me say that again.  Almost everything DJM says is built on a mathematical comparison between two different pools whose data were constructed using different methods.  I would not blame you if now stopped reading."

In other words, it is difficult to tell whether any differences identified by Professor Merritt are:

(1) Due to differences between Ohio and the U.S. as a whole

(2) Due to differences in methodology between Merritt, NALP, and After the JD

(3) Actually due to differences between 2011 and 2014 for the same group

After Professor Galle’s devastating critique, journalists should have been extremely skeptical of Merritt’s methodology and her conclusions.  Professor Merritt’s response to Galle’s critique, in the comments below his post, is not reassuring:

“Bottom line for me is that the comparison in law firm employment (62.1% for the Class of 2000 three years after graduation, 40.5% for the lawyers in my population) seems too stark to stem solely from different populations or different methods—particularly because other data show a more modest decline in law firm employment over time. But this is definitely an area in which we need much, much more research.”

Judging from this response and the quotes in The New York Times, Merritt appears to be doubling down on her inapposite comparisons rather than checking how much of her conclusions are due to potentially fatal methodological problems.  What Professor Merritt should have done is replicate her 2014 Ohio-only methodology in 2000/2001 or 2010/2011, compared the results for Ohio only at different points in time, and limited her claims to an analysis of the Ohio legal employment market.

There are additional problems with Professor Merritt’s study (or at least the March 11 version that I reviewed).**** 

• Ohio is not a representative legal employment market, but rather a relatively low paying one where lawyers comprise a relatively small proportion of the workforce.  

• A disproportionate share of the 8 or 9 law schools in Ohio (9 if you include Northern Kentucky) are low ranked or unranked, and this presumably is reflected in their employment outcomes. 

• Merritt’s sample is subject to selection bias because of movement of the most capable law graduates out of Ohio and into higher paying legal markets.  Ohio law graduates who do not take the Ohio bar after obtaining jobs in Chicago, New York, Washington D.C., or other leading markets will not show up in Merritt’s sample.  

• Whereas Merritt concludes that law graduate outcomes have not improved, the data may simply reflect the fact that Ohio is a less robust employment market than the U.S. as a whole. 

• Merritt’s analysis of employment categories does not take into account increases in earnings within employment categories.  After the JD and follow-ups suggests that these within-category gains are substantial, as does overall increases in earnings from Census data. 

• Merritt makes a biased assumption that anyone she could not reach is unemployed instead of gathering additional information about non-respondents and weighting the results to take into account response bias. Law schools may have been more aggressive in tracking down non-respondents than Professor Merritt was. 

For the benefit of those who are curious, I am making my full 8 page critique of Professor Merritt's working paper available here, but please keep in mind that it was written in mid March and Professor Merritt may have addressed some of these issues in more recent versions of her paper.  If that is the case, I trust that she’ll highlight any changes or improvements in a blog post response.

 

*    A few weeks ago I asked a research assistant (a third year law student) to search for stories in The New York Times and Wall Street Journal about law school.  Depending on whether the story would have made my research assistant more likely or less likely to want to go to law school when he was considering it or would have had no effect, he coded the stories as positive, negative, or neutral.  According to my research assistant, The New York Times reported 7 negative stories to 1 positive story in 2011 and 5 negative stories to 1 positive story in 2012.  In 2013, 2014, and 2015, The New York Times coverage was relatively balanced.  In aggregate over the five-year period The New York Times reported about 2 negative stories for every 1 positive story.  The Wall Street Journal’s coverage was even more slanted—about 3.75 negative stories for every positive story—and remained heavily biased toward negative stories throughout the five-year period.

**   Professor Stephen Diamond notes the LSAT tutor’s relatively high hourly wage, more lucrative opportunities the tutor claims he turned down, and how the tutor describes his own work ethic.

***  For the class of 2010, the figure at Columbia was roughly 52 percent 9 months after graduation, but activity in the lateral recruitment market suggests things may be looking up.

**** The comments that follow summarize a lengthy (8 page) critique I sent to Professor Merritt privately in mid March after reviewing the March 11 draft of her paper.  I have not had a chance to review Professor Merritt’s latest draft, and Professor Merritt may have responded to some of these issues in a revision.  

 

UPDATE:  Additional responses from Professors Galle and Merritt.

I call "bullshit" on this one...

...a story allegedly about a fearful closeted Christian law professor at an elite school?  The late William Stuntz at Harvard, Michael McConnell at Stanford, David Skeel at Penn, Stephen Bainbridge at UCLA all seem to have done rather well at elite schools, despite being quite openly religious.  (I'm sure there are others, but the preceding scholars have incorporated their religious commitments into at least some of their scholarly and popular writing.)  I invite the alleged subject of this article to contact me; I will also preserve his anonymity, but I'd like to pose some further questions about what it is about his institutional environment that would lead to the behavior described.  I'm afraid it just doesn't ring true to what I've seen at the institutions I've taught.  Yes, levels of religiosity among law professors are not high (though they are higher than among philosophy professors); but norms of collegiality and respect for differences have generally created environments in which no one would reasonably feel a need to go into the closet as described.  Maybe I've been lucky, but...

Apparently if you don't agree with Paul Campos about law schools...

...you're like a Holocaust denier.   This latest smear from Crazy Campos was directed at Michael Simkovic, whose grandparents, by the way, were Holocaust survivors.

Even when one disagrees with Prof. Simkovic, there's no doubt that a debate that had been far too dominated by clowns like Campos and various amateurs has been raised to a far more serious level by his systematic interventions, both on this blog and in the peer reviewed literature.  CC's latest smear job just gives more urgency to Dean Bales's question.

LSAC reports applicants down just 2.6% as of April 17...

...here and see the chart, which suggests we've hit bottom in terms of the applicant pool (barring dramatic economic jolts, that is, which could move things either way).  Of course this is a bottom last  seen in the 1970s when there were 50 fewer law schools.  But given how many law schools have refrained from hiring faculty due to uncertainty about the future, my guesstimate is that we'll see a slight uptick in law school hiring next year, since many schools have unfilled needs.

Six law professors elected to American Academy of Arts & Sciences

They are:  Noah Feldman (Harvard), David Garland (NYU), Jane Ginsburg (Columbia), Philip Hamburger (Columbia), Lewis Kornhauser (NYU), and Robin West (Georgetown). 

A somewhat more balanced piece on law schools and the legal profession...

...in The Washington Post.