Brian Leiter's Law School Reports

Kyle McEntee, Derek Tokaz, and Law School Transparency: "Profits of Doom"

Apt comments by Stephen Diamond (Santa Clara) in the wake of this!

I was surprised to discover recently that Mr. Tokaz's primary extra-curricular activity--when not posing as a high-minded law school reformer--is running a website devoted to insulting, ridiculing, and defaming law professors, including as Prof. Diamond notes, one post calling for Diamond to be fired for having the temerity to disagree with Mr. Tokaz and other know-nothings about law schools.  A class act.  (Mr. Tokaz also uses the moniker BL1Y when posting his harassment and abuse of law faculty.)

UPDATE:  In response to a reader query:  you can reach Mr. Tokaz at derek.m.tokaz@gmail.com.  He seems to respond to being called out on false statements of fact, which is one of his specialties.

Ten Million Dollar "unrestricted" gift to Northwestern Law

That's a good one for Northwestern.

Reflections on "The Economic Value of a Law Degree" and the Response to It

It has been a curious two weeks in the legal academic blogosphere.   Michael Simkovic, a law professor at Seton Hall (who, I learned, never went on the law teaching market, he was plucked out of practice at Davis Polk by a savvy hiring committee), and Frank McIntyre, a labor economist at the Rutgers Business School, produced a serious piece of empirical analysis of the economic outcomes for those earning a JD as opposed to those stopping with the BA.  The result was hardly surprising:  JDs do better, considerably better, at both the top and the bottom.

Although the result isn't surprising on reflection, it clearly created a problem for those heavily invested in the idea that it's stupid to go to law school.  Predictable know-nothings--like Elie Mystal (a nice guy, I hasten to add, but out of his depth with respect to almost any serious topic) at the Above the Law cesspool--did scandalous hatchet jobs and obviously didn't read the article.  All the bottom-feeders of cyberspace stick together, of course, so we find Matt Leichter, another benighted blogger, endorsing Mystal's mistakes and adding more of his own!  The Dunning-Kruger Effect is alive and well!  Certain familiar charlatans, like Paul Campos, were exposed once again as having no idea what they were talking about (we even had the remarkable spectacle of Campos, an actual academic fraud, calling an actual scholar (Stephen Diamond) an "academic fraud," even as both Simkovic and Bainbridge noted the correctness of Diamond's criticisms of Campos.)

Most disappointing, however, were Brian Tamanaha's badly confused interventions.  As someone who endorsed Tamanaha's book originally, I've been embarrassed by the arrival of serious research on the topic.  Tamanaha's work, in the areas where I have expert knowledge, has always been a bit notorious for its confusions and theoretical overreaching, especially in its desire to make startling claims, the evidence and the arguments be damned. I had thought his book on law schools was different.  (I suppose I should have been given pause by the way in which Tamanaha repeatedly tried to legitimize some of the most disgusting and deranged "scam" blogs out there, even posting encouraging comments on some of them.)  In any case, it now turns out that Failing Law Schools is consistent with the rest of his work, and I was mistaken in commending it. 

Well, maybe not wholly mistaken, since Tamanaha has scaled back his claims in the wake of the Simkovic & McIntyre analysis (also here--and see this reply).  It's not law schools, per se, that are failing, but just Thomas Jefferson, Phoenix, Southwestern, and a few others he singles out for calumny.   To be honest, I don't even know that they are failing, it really depends on whether the long-term economic outcomes for their graduates fall well below the 25th-percentile in the Simkovic & McIntyre study--Tamanaha has no evidence on that, nor do I.   As Stephen Diamond (Santa Clara) aptly put it in responding to another tissue of confusions about the paper:  "The challenge for an individual law student is to determine where they are likely to fall along the distribution. I don’t think the Simkovic/McIntyre paper was intended to be a calculator for prospective law students and so criticizing them for that issue is unfair. However, the paper does provide concrete evidence that such a distribution actually exists and that for most points on the distribution the present value of the earnings premium associated with a JD is positive."

Tamanaha's response to the Simkovic & McIntyre article has been, in short, a spectacular intellectual embarrassment and travesty:  the conclusion now seems inescapable that he has no idea what he is talking about.  (Naturally, the Cato Institute hacks thought his intellectual mess was brilliant!)  A colleague at another top law school put it to me aptly in an e-mail last week:

Tamanaha is embarrassing himself terribly.   His responses have the feel of someone desperate to defend a profitable franchise, and they are manifestly deficient on the merits.  Simkovic and McIntyre have comported themselves like true scholars.  The contrast is striking.

Tamanaha should apologize in public and simply admit that he was wholly out of his depth.  I certainly apologize to anyone who took Failing Law Schools seriously based on my opinion.  It has some good anecdotes, and some interesting history of law school regulation, but I should have been more cautious.  (I still agree with Tamanaha that the ABA ought to lighten up on regulation, to allow more models of legal education--that point is quite independent of the Simkovic & McIntyre analysis.)

So here's the amazing fact:  not a single meritorious criticism of the paper was voiced in the blogosphere despite nearly two weeks of intense discussion.*  It has all been careless, in some cases idiotic, mispresentations; technical and conceptual confusions; or irrelevant ranting by ignoramuses.  It all reminds me why, as I said long ago, blogs are bad for legal scholarship, since they provide a voice, sometimes--as with the ATL cesspool--a loud voice for people who literally have nothing to say because they have no relevant competence or skill.

Here's what I think we know and don't know after the first serious scholarly intervention in the debate about an economic value of a law degree:

1.  The vast majority of those who got a JD over the last two decades are better off, financially, than similar individuals who stopped with the BA.

2.  We don't know if other post-graduate degrees are more worthwhile, financially, than the JD; Simkovic & McIntyre did not attempt to address that systematically.

3.  We don't know if the economic pattern for JD-holders will hold for the future, though Simkovic & McIntyre adduce some evidence that it will.  [BL EDIT:  rather good evidence, I will add, in light of a recent dishonest misreading]

And that, I think, is a fair summary of where we are.  The critics of law schools, it turns out, simply do not understand the basic math behind valuation--net present value.  They think law school is a bad value because they apparently don't know how to value anything that pays off over time. 

We have had a massive downturn in applications to law schools the last three years, primarily as a result of ABA-induced better reporting of employment outcomes and New York Times coverage of the bad job market for new JDs.  That's fine, some of these individuals may have made the correct decision by foregoing law school.  But we've also had an hysterical cyber-reaction by miscreants and charlatans (like Campos), as well as by unemployed law school graduates desperate for someone to blame for the latest crisis of capitalism.  Journalists have generally done a poor job sorting the wheat from the chaff, though there have been honorable exceptions (I think especially of Debra Weiss at the ABA journal).

*Perhaps I missed an intelligent response to the Simkovic & McIntyre paper buried somewhere in the bowels of cyberspace.

UPDATE:  Predictably, Brian Tamanaha was not happy with the preceding, but I'll try to be succinct:  (1) it is false that he and I have "skirmished on various subjects over the past fifteen or so years"; the one time I engaged his work was in the review of his book three years ago (anyone interested in the actual objections should read the review essay); (2) he did not "challenge" my "interpretation of the formalists and realists"--on the formalists, because I have no views on the historical figures denominated formalists, and on the realists, because he did not engage seriously with either mine or Fred Schauer's readings of the Realists in his book; (3) he wholly misrepresents the contents of David Rabban's recent book and also misrepresents, by implication, Rabban's opinion of Tamanaha's book, which was the same as mine (Rabban agrees with me that Tamanaha stated a prima facie plausible case against the attribution of "vulgar" formalism to 19th-century writers, but that does nothing to resolve the issue of what other kinds of formalism were common in the 19th-century--again see my review essay); (4) anyone who reviews Tamanaha's postings can see how utterly ludicrous it is for him now to pretend that he acknowledged the evidence that Simkovic & McIntyre adduced in support of the proposition that the current recession in the legal market is well within the parameters of past economic cycles in the legal profession; (5) I am agnostic on Tamanaha's motives, and do not think my correspondent meant "profitable enterprise" literally.  I do think his work, in both legal theory and on legal education, exemplifies a pattern of careless scholarship, which is why I noted my one discussion of the former by way of evidence.

ANOTHER:  Apt comments by Frank Pasquale, also a blogger at the Balkinization site.

Brian Tamanaha’s Straw Men (Part 4): We would have to be off by 85 percent for our basic conclusion to be incorrect

Update:

Brian Tamanaha’s latest blog post has been hailed by Stephen Diamond and others as a major concession.  But in a comment below the post, Tamanaha explains to one of his followers:

 “I believe the doubts I raised about the study in my previous three posts have not been answered satisfactorily.”

We therefore continue our response to Tamanaha’s first three posts before addressing Tamanaha’s fourth post.

BT Claim 4:  Historical economic data tells us nothing about the future

"It is exeedingly rare to find reliably predictive 'historical norms' in the social sciences because social life is too complex and circumstances are constantly changing . . . S&M have produced a narrow, partial, time-bound study that has zero predictive relevance for anyone thinking about attending law school today." A proper study "may require data over several centuries."  

Response: We would have to be off by 85 percent for our basic conclusion to be incorrect

In finance, valuation entails using historical data to establish a baseline scenario.  This baseline is generally viewed as the center of a distribution of possible future outcomes.  The baseline can be modified to construct upside and downside scenarios to get a sense of what could happen if the future is better or worse than the past.  Scenario analysis can help understand how robust the findings are--that is, how much the future would need to deviate from the past to change the basic directional conclusion of the valuation analysis.  For the extreme downside, this is sometimes called "break-even analysis."

For general background focused on the corporate context, I recommend Tim Koller, Marc Goedhart & David Wessel, McKinsey, Valuation: Measuring and Managing the Value of Companies (4th Edition), and Brealey, Myers & Allen, Principles of Corporate Finance.

We estimate the present value of a law degree at the median as $610,000 as of the start of law school.  This figure is pre-tax and pre-tuition, but includes opportunity costs and financing costs. 

In other words, some combination of the student and the federal government could pay up to $610,000 for the law degree and break even.  The government might contribute to the cost through debt forgiveness through Income Based Repayment, or through some other method.

As we note in the paper, ABA data suggest that the typical tuition cost for law school, less scholarships and grants, is roughly around $30,000 per year.  Spread over 3 years, and assuming tuition rises 6 percent per year nominal (i.e., at our discount rate), this comes to $90,000 in present value terms as of the start of law school.

For law school to cease to be a value-creating investment for the majority of law students, the present value of the lifetime earnings premium would have to fall to below $90,000—a drop of 85 percent.

At the “25^th percentile” (more like the 15^th because of regression to the median), toward the bottom of the distribution, the law school earnings premium is $350,000.  Assuming tuition (less scholarships and grants) remains at $30,000, the 25^th percentile premium would need to fall by 74 percent for a law degree to no longer be value-creating proposition toward the bottom of the distribution.  At the mean, we’d have to be off by 91 percent.

These would be extreme deviations from the pattern seen in 1996-2011.

Presumption of innocence should not be a Bayesian prior

Interesting analysis, as always, from philosopher Larry Laudan (Texas & UNAM).

Diamond wipes the floor with Tamanaha

Amusingly so.

Repetitive (and avoidable) mistakes

At the American Lawyer, Matt Leichter repeats many misrepresentations of our research that originally appeared in the tabloid Above the Law, even after Above the Law posted corrections and after we refuted many of these misrepresentations.  He also refers to anonymous comments attacking our research from people who did not read it.  

• He erroneously claims that we "assume law school pays off equally for non-lawyers" when we in fact measure the earnings premium regardless of occupation.  We do not assume anything.  
• Leicther's description of our take on BLS projections is lifted out of context, since we note that even BLS economists are skeptical of these sorts of projections.
• Leicther ignores our careful controls for ability sorting and selection.
• He ignores the fact that we show not only current student loan default data, but also data that predates IBR.  Law school student loan default rates were low even before IBR was available.

Steve Harper makes many of the same mistakes, and throws in a few disparaging remarks to boot.

• Harper repeats Tamanaha's claim--which the Washington Post reported as false--that we only look at means and do not consider different points in the distribution.  And he throws in a red herring about a bi-modal distribution.
• Harper gets confused about present value and about the difference between medians and means, much like Campos and Tamanaha.  Harper incorrectly reports that  "a [law] degree returns at most a lifetime average of $687 a month" spread "over a 40-year career."  
• The average (mean) is in fact around $53,000 per year before taxes and the median is around $32,000 per year in real terms (after taking inflation into account).  After taxes, the annual average benefit is greater than $37,000 per year. 

 

• Harper gets confused about causal inference and controls for ability sorting and selection, and repeats erroneous claims from Paul Campos that the United States Census Bureau's Survey of Income and Program Participation does not constitute a representative sample.  Harper throws in some new errors about the relationship between sample size and statistical significance.
• Harper incorrectly claims that our findings of a premium depend on certain assumptions when--as we explicilty note in the paper--our findings are robust and do not depend on those assumptions.  And he overlooks the data on which those assumptions are based.
• Harper incorrectly claims that half of law graduates will remain "below the median income" even after they graduate.  In fact, the median income for law graduates is 60 percent higher than the median income for similar bachelor's degree holders.  

We've already responded to many of these same misrepresentations of our research from Above the Law, Brian Tamanaha, and Paul Campos. Simple fact-checking, either by reading the article or by checking our blog posts, could have prevented these errors.

Hopefully the editors at the American Lawyer will promptly post corrections and have a serious discussion with Mr. Leichter and Mr. Harper about the differences between critiquing research on the merits and misrepresenting the contents of that research--and impugning the integrity of its authors--in a nationally distributed publication.

Brian Tamanaha’s Straw Men (Part 3): We use better (and more) data than studies Tamanaha praised in his book

BT Claim 3:  16 years of data is not enough

“S&M’s bold assertion that their 16-year study establishes valid ‘historical norms’ on law degree earnings would be scoffed at by social scientists who take the notion of ‘historical norms’ seriously. That is more than enough time to confirm norms governing the mating behavior of fruit flies, but 16 years is laughably inadequate for predicting something as complex and subject to change as the lifetime earnings of future law grads.”

Response Part 1:  A fine idea for historical research

We will be delighted to read the results of similar work on earnings premia carried back into the distant past.  We certainly are not claiming to have uncovered hundreds of years of data on law school earnings premia.  But, ultimately, we are not sure how valuable such a retrospective would be for today's graduate. 

Response Part 2:  Professor Tamanaha and other critics of law school relied on—and praised—studies that use far less than 16 years of data

The literature has numerous studies using smaller data sets than ours (citations available), including several studies using only 3 years of data that were cited by Professor Tamanaha in Chapter 11 (starting on page 137-38) of his recent book, Failing Law Schools.  Professor Tamanaha cited these studies without comment or criticism regarding the number of years of data used (although he did criticize them on other grounds), so we find it odd that he views our study as somehow deficient on this ground.

Professor Tamanaha and other law school critics have cited and praised studies that were much less rigorous and used much less data, including Herwig Schlunk’s “Mama’s Don’t Let Your Babies Grow Up to Be Lawyers.”  Schlunk used a couple of years of data from Payscale.com, law.com, AbovetheLaw.com, and other websites.

The Economic Value of a Law Degree uses 16 years of data regarding earnings across age groups from the United States Census Bureau.

On page 217, note 18 of his book, Tamanaha called Schunk’s study “An excellent example of how [to determine whether a law degree is a good investment] in economic terms.”  Tamanaha also praised an article by Jim Chen that used only starting salaries.

While we're happy to admit that no study is perfect, if these studies have enough years of data for Professor Tamanaha to cite in his book, then we struggle to understand his objection to 16 years of data across age groups. 

And we're now going to take this opportunity to cite a personal favorite line from Professor Tamanaha's post:

“Let me also confirm that [Simkovic & McIntyre’s] study is far more sophisticated than my admittedly crude efforts.”

We'll take what we can get.

For a brief critique of Professor Schlunk’s work, see the discount rate appendix of The Economic Value of a Law Degree.   A more thorough critique of Professors Schlunk’s work—and Professor Tamanaha’s reliance on it—is contained in our book review of Failing Law Schools, which will be posted on SSRN soon.

 

Reminder: Mike Simkovic is still guest-blogging...

...about his important paper with Frank McIntyre, and will continue blogging into next week.  I may have a few items too, including some reflections on the remarkable display of cyber-stupidity their serious research has elicited.

UPDATE:  As if on cue, we now have this from an adjunct law professor who has been riding the "don't go to law school" bandwagon for awhile.  Amusingly, he refers to Campos as a "respected" academic.  At least one adjunct thinks so. 

ANOTHER:  Stephen Diamond (Santa Clara) comments on Mr. Harper's opnion piece.

Brian Tamanaha’s Straw Men (Part 2): Who's Cherry Picking?

BT Claim 2:  Using more years of data would reduce the earnings premium

BT Quote: There is no doubt that including 1992 to 1995 in their study would measurabley reduce the 'earnings premium.'"  

Response:  Using more years of historical data is as likely to increase the earnings premium as to reduce it

We have doubts about the effect of more data, even if Professor Tamanaha does not.

Without seeing data that would enable us to calculate earnings premiums, we can’t know for sure if introducing more years of comparable data would increase our estimates of the earnings premium or reduce it.

The issue is not simply the state of the legal market or entry level legal hiring—we must also consider how our control group of bachelor’s degree holders (who appear to be similar to the law degree holders but for the law degree) were doing.   To measure the value of a law degree, we must measure earnings premiums, not absolute earnings levels.

As a commenter on Tamanaha’s blog helpfully points out:

“I think you make far too much of the exclusion of the period from 1992-1995. Entry-level employment was similar to 1995-98 (as indicated by table 2 on page 9).

But this does not necessarily mean that the earnings premium was the same or lower. One cannot form conclusions about all JD holders based solely on entry-level employment numbers. As S&M's data suggests, the earnings premium tends to be larger during recessions and their immediate aftermath and the U.S. economy only began an economic recovery in late 1992.

Lastly, even if you are right about the earnings premium from 1992-1995, what about 1987-91 when the legal economy appeared to be quite strong (as illustrated by the same chart referenced above)? Your suggestion to look at a twenty year period excludes this time frame even though it might offset the diminution in the earnings premium that would allegedly occur if S&M considered 1992-95.”

There is nothing magical about 1992.  If good quality data were available, why not go back to the 1980s or beyond?   Stephen Diamond and others make this point.

The 1980s are generally believed to be a boom time in the legal market.  Assuming for the sake of the argument that law degree earnings premiums are pro-cyclical (we are not sure if they are), inclusion of more historical data going back past 1992 is just as likely to increase our earnings premium as to reduce it.  Older data might suggest an upward trend in education earnings premiums, which could mean that our assumption of flat earnigns premiums may be too conservative. Leaving aside the data quality and continuity issues we discussed before (which led us to pick 1996 as our start year), there is no objective reason to stop in the early 1990s instead of going back further to the 1980s.

Our sample from 1996 to 2011 includes both good times and bad for law graduates and for the overall economy, and in every part of the cycle, law graduates appear to earn substantially more than similar individuals with only bachelor’s degrees.

 

 

This might be as good a place as any to affirm that we certainly did not pick 1996 for any nefarious purpose.  Having worked with the SIPP before and being aware of the change in design, we chose 1996 purely because of the benefits we described here.  Once again, should Professor Tamanaha or any other group wish to use the publicly available SIPP data to extend the series farther back, we'll be interested to see the results.