« June 2013 | Main | August 2013 »

July 24, 2013

Brian Tamanaha’s Straw Men (Part 1): Why we used SIPP data from 1996 to 2011

BT Claim:  We could have used more historical data without introducing continuity and other methodological problems

BT quote:  “Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity . . .”

Response:  Using more historical data from SIPP would likely have introduced continuity and other methodological problems

SIPP does indeed go back farther than 1996.  We chose that date because it was the beginning of an updated and revitalized SIPP that continues to this day.  SIPP was substantially redesigned in 1996 to increase sample size and improve data quality.  Combining different versions of SIPP could have introduced methodological problems.  That doesn't mean one could not do it in the future, but it might raise as many questions as it would answer.

Had we used earlier data, it could be difficult to know to what extent changes to our earnings premiums estimates were caused by changes in the real world, and to what extent they were artifacts caused by changes to the SIPP methodology. 

Because SIPP has developed and improved over time, the more recent data is more reliable than older historical data.  All else being equal, a larger sample size and more years of data are preferable.  However, data quality issues suggest focusing on more recent data. 

If older data were included, it probably would have been appropriate to weight more recent and higher quality data more heavily than older and lower quality data.  We would likely also have had to make adjustments for differences that might have been caused by changes in survey methodology.  Such adjustments would inevitably have been controversial.

Because the sample size increased dramatically after 1996, including a few years of pre 1996 data would not provide as much new data or have the potential to change our estimates by nearly as much as Professor Tamanaha believes.  There are also gaps in SIPP data from the 1980s because of insufficient funding.

These issues and the 1996 changes are explained at length in the Survey of Income and Program Participation User’s Guide

Changes to the new 1996 version of SIPP include:

Most government surveys topcode income data—that is, there is a maximum income that they will report.  This is done to protect the privacy of high-income individuals who could more easily be identified from ostensibly confidential survey data if their incomes were revealed.

Because law graduates tend to have higher incomes than bachelor’s, topcoding introduces downward bias to earnings premiums estimates. Midstream changes to topcoding procedures can change this bias and create problems with respect to consistency and continuity.   

Without going into more detail, the topcoding procedure that began in 1996 appears to be an improvement over the earlier topcoding procedure.

These are only a subset of the problems extending the SIPP data back past 1996 would have introduced.  For us, the costs of backfilling data appear to outweigh the benefits.  If other parties wish to pursue that course, we'll be interested in what they find, just as we hope others were interested in our findings.

Posted by Michael Simkovic on July 24, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Science, Weblogs | Permalink

Brian Tamanaha’s Straw Men (Overview)

Brian Tamanaha previously told Inside Higher Education that our research only looked at average earnings premiums and did not consider the low end of the distribution.  Dylan Matthews at the Washington Post reported that Professor Tamanaha’s description of our research was “false”.  

In his latest post, Professor Tamanaha combines interesting critiques with some not very interesting errors and claims that are not supported by data.   Responding to his blog post is a little tricky as his ongoing edits rendered it something of a moving target.  While we're happy with improvements, a PDF of the version to which we are responding is available here just so we all know what page we're on.

Stephen Diamond explains why Tamanaha apparently changed his post: Ted Seto and Eric Rasmusen expressed concerns about Tamanaha’s use of ad hominem attacks.

Some of Tamanaha’s new errors are surprising, because they come after an email exchange with him in which we addressed them.  For example, Tamanaha’s description of our approach to ability sorting constitutes a gross misreading of our research.  Tamanaha also references the wrong chart for earnings premium trends and misinterprets confidence intervals.  And his description of our present value calculations is way off the mark.

Here are some quick bullet point responses, with details below in subsequent posts:


Posted by Michael Simkovic on July 24, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Science, Weblogs | Permalink

July 23, 2013

"How to fix law schools"? TNR asked six "experts" [sic]...

 ...meaning one gossip columnist who went to law school, one longtime law professor, two journalists who went to law school, one "journalist moonlighting as a law professor", and one in-house counsel.  In short, they asked no experts on legal pedagogy, the economics of legal education, or the legal profession, just a group of pontificators (plus one lawyer) with varying degrees of familiarity with law schools.  Most of what they have to say is silly and irrelevant, with two exceptions:  the longtime law professor (Dershowitz) offers his own version of the 2-year JD idea that others have floated; and the in-house counsel for Cisco reveals an interesting plan, which should offer some useful experience to some students:

Colorado University Law School's visionary dean Philip J. Weiser is working to implement this program next year: Students will work as interns at Cisco for seven months–from June of the second year of law school until the following January, and potentially part-time during the following spring. We will pay them as we do our customary interns, and the students will not be required to pay tuition to the law school for the fall semester. The students will be supervised by a faculty member during the fall through an intensive credit-earning independent study project, and the students will also take extra courses during the rest of law school to ensure sufficient ABA-approved credits to graduate.  

 One can guess which member of the Colorado law faculty will not be asked to supervise anyone!

TNR also ran a sensationalistic and fact-free article on Chicago-based Mayer Brown in the same issue.  Even I would expect a bit more from TNR, but when it comes to law schools, careless journalism is increasingly the norm.

Posted by Brian Leiter on July 23, 2013 in Legal Profession, Of Academic Interest | Permalink

July 22, 2013

Above the Law Needs to Read More

Elie Mystal and Joe Patrice at Above the Law misrepresented the contents of The Economic Value of a Law Degree.   Mr. Mystal and Mr. Patrice are entitled to disagree with us but it would help if they would first carefully read what we wrote.  I realize the paper is long, but a lot of the problems below appear to come from jumping to conclusions without reading.

 Here are six of ATL’s inaccurate claims, with the facts listed below:


(1)   ATL Claim: The article doesn’t present any earnings data after 2008 and ignores changes since the financial crisis


Facts: We present earnings data from 1996 to 2011.  We also check for trends.  As shown in the figure below, the earnings premiums in recent years are within historical norms. 





Earnings declined for both law degree holder and bachelor degree holders, and law degree holders maintained their relative advantage.

We note in the article that the most recent college and law graduates in our sample graduated in 2008.  This is because the Census Bureau reports educational attainment at the start of each Panel and the most recent panel started in 2008.  Each panel tracks earnings for several years.

We separately analyze earnings premiums for young law degree holders and bachelor's degree holders, and do not find evidence that the earnings premium has compressed.


Youngster Cycles


(2)   ATL Claim: The article relies on averages (or means) and doesn’t provide any information relevant to students who attended lower ranked law schools


Brian Tamanaha made the same misrepresentation. We have already responded to it here.  The Washington Post reported that the averages-only claim is "false."


Lifetime earnings distribution slide


(3)   ATL Claim:  The article doesn’t present after-tax earnings


Facts: We present both pre-tax and after-tax present value figures, and discuss both the private benefit of the law degree to the student and the public benefit to the federal government in the form of higher tax revenue. 

The discussion of taxes appears on pages 42 to 44 of the article, but we will relieve the suspense: the Federal government takes about a third of the extra earnings. 


(4)   ATL Claim:  The article doesn’t consider the cost of tuition


Facts: The article presents both benefits (higher earnings) and costs (such as tuition).  We include a range of annual tuition values between 0 and $60,000 in our Internal Rate of Return calculation in Table 10 on page 68 of the article.  The Internal Rate of Return is typically between 10 and 25 percent (in real terms, net inflation), depending on tuition and the point in the distribution where the law degree holder falls.


IRR Table short




We list present value figures prior to deducting tuition in the article so that the cost of the degree—tuition and fees less grants and scholarships—can readily be subtracted from the value of the degree—the earnings premium. We also cite to ABA data about law school tuition, both before and after scholarships and grants.

Mr. Mystal is correct that the $1 million figure is pre-tax and pre-tuition—the article says as much. However, after deducting taxes, and tuition, and even close to the bottom of the distribution, it appears that a law degree is very often a profitable investment. 

How profitable varies from person to person.  There are surely a few individuals who do not benefit at all or even do worse.  If we had a way to identify these unlucky people before law school we would encourage them to avoid law school.    Judging from student loan default rates, this is a relatively rare outcome.  Further, income based repayment of loans should substantially mitigate this downside, even as it makes it difficult for us to nail down exactly how much a given person pays for law school. 


Student Loan Defaults



(5)   ATL Claim:  The article simply “assumes” that law degree holders who don’t practice law benefit from a law degree


Facts:  We don’t assume.  We measure earnings for all law degree holders in sample, regardless of eventual occupation.  The data suggests that the earnings premium is significant, even including those who do not practice law. 


Slide 20 Blog Post 1


(6)   ATL Claim:  The article compares apples to oranges and fails to control for differences between bachelor’s and law degree holders


Facts:  We control for many demographic, academic, and socio-economic characteristics other than law school attendance that predict earnings. In a supplemental analysis using data from the National Education Longitudinal Study, we incorporate additional control variables and tests for ability sorting and selection.   

Controls 1


Our controls for ability sorting and selection are described at length in Part II and Appendix A of The Economic Value of a Law Degree.

Bottom line: read the paper.

Posted by Michael Simkovic on July 22, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Weblogs | Permalink

July 21, 2013

The Economic Value of a Law Degree: Week 1 Summary

In The Economic Value of a Law Degree Frank McIntyre and I measure differences in annual earnings and hourly wages between those with law degrees and similar individuals who end their education with a bachelor’s degree.   We account for unemployment and disability risk.

We also control for many demographic, academic, and socio-economic characteristics other than law school attendance that predict earnings. In a supplemental analysis using data from the National Education Longitudinal Study, we incorporate additional control variables and tests for ability sorting and selection.

The Economic Value of a Law Degree was covered by:

Our first blog post at Concurring Opinions ends with a presentation of annual earnings premiums at the mean and median, as well as at the high and low end of the distribution. 

Questions and Critiques

The low and and high ends of the distribution

Representativeness of the data

Present Value and Net Present Value



Confusion at Above the Law


Posted by Michael Simkovic on July 21, 2013 in Guest Blogger: Michael Simkovic, Law Professors Saying Dumb Things, Legal Profession | Permalink | Comments (0) | TrackBack

July 20, 2013

Diamond on Campos on the economic value of a law degree

This is amusing.

Posted by Brian Leiter on July 20, 2013 in Legal Profession, Of Academic Interest, Student Advice | Permalink

July 19, 2013

Professor Simkovic will be guest-blogging here next week

He will be addressing some of the criticisms of his co-authored work on the economic value of a law degree, as well as correcting many of the errors about the content of the piece that have been put into circulation by people who most likely did not read the paper, or, if they did, plainly didn't understand it.

Posted by Brian Leiter on July 19, 2013 | Permalink

July 18, 2013

Some adult coverage and commentary on "The Economic Value of a Law Degree"


Posted by Brian Leiter on July 18, 2013 in Legal Profession, Of Academic Interest, Student Advice | Permalink

Simkovic v. Tamanaha on the economic value of a law degree

This round to Simkovic.

UPDATE:  This is also helpful.

Posted by Brian Leiter on July 18, 2013 in Legal Profession, Of Academic Interest, Student Advice | Permalink

July 17, 2013

"The Economic Value of a Law Degree"

This article ought to fundamentally change the conversation about the economic value of legal education; it's considerably more sophisticated methodologically than anything I've seen, and it sensibly compares the value of the JD to the alternatives, such as only having a BA.  (A peculiarity of the cyber-ranting about "don't go to law school" is that it never explains, or even seems to care, what happens or would happen to those who forego the JD.)   There's also a powerpoint version which contains most (not all) of the key data and analysis.

UPDATE:  Predictably, serious research and analysis is met with derision and and insults from the know-nothing crowd.  Professor Simkovic sent the following measured reply to Mr. Mystal's nonsense, which makes clear that Mystal didn't even read the article before ranting about it:

Your coverage of our research, "The Economic Value of a Law Degree" contains several inaccuracies that you may wish to correct.  

1)  The study does not only look at averages, as you state in your post.  It also considers the median, 25th percentile and 75th percentile outcomes.  Even at the 25th percentile, the value of a law degree exceeds the cost.

2) The study includes earnings data from 1996 to 2011--the coverage does not stop in 2008 as you state.  The most recent law school and college graduates in sample for the earnings portion of the study are from the class of 2008, but earnings are reported through 2011.

3) The study notes the typical tax rate on the earnings premium and reports both pre-tax and after tax-values.  It does not only report the value of a law degree before taxes, as you state.

4) The study includes a range of possible tuition values in our internal return calculations and explains clearly how to compare the cost of the degree to the value of the degree.  It does not simply provide "theoretical never-never-land [estimates] where things like TUITION don’t matter", as you state.

5) The study presents student loan default rates for many low-ranked law schools.  The default rates are lower than average default rates for former college and graduate school students.

UPDATE:  Professsor Simkovic will be blogging about his research here.

Posted by Brian Leiter on July 17, 2013 in Legal Profession, Of Academic Interest, Student Advice | Permalink