July 24, 2013
Brian Tamanaha’s Straw Men (Part 1): Why we used SIPP data from 1996 to 2011
BT Claim: We could have used more historical data without introducing continuity and other methodological problems
BT quote: “Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity . . .”
Response: Using more historical data from SIPP would likely have introduced continuity and other methodological problems
SIPP does indeed go back farther than 1996. We chose that date because it was the beginning of an updated and revitalized SIPP that continues to this day. SIPP was substantially redesigned in 1996 to increase sample size and improve data quality. Combining different versions of SIPP could have introduced methodological problems. That doesn't mean one could not do it in the future, but it might raise as many questions as it would answer.
Had we used earlier data, it could be difficult to know to what extent changes to our earnings premiums estimates were caused by changes in the real world, and to what extent they were artifacts caused by changes to the SIPP methodology.
Because SIPP has developed and improved over time, the more recent data is more reliable than older historical data. All else being equal, a larger sample size and more years of data are preferable. However, data quality issues suggest focusing on more recent data.
If older data were included, it probably would have been appropriate to weight more recent and higher quality data more heavily than older and lower quality data. We would likely also have had to make adjustments for differences that might have been caused by changes in survey methodology. Such adjustments would inevitably have been controversial.
Because the sample size increased dramatically after 1996, including a few years of pre 1996 data would not provide as much new data or have the potential to change our estimates by nearly as much as Professor Tamanaha believes. There are also gaps in SIPP data from the 1980s because of insufficient funding.
These issues and the 1996 changes are explained at length in the Survey of Income and Program Participation User’s Guide.
Changes to the new 1996 version of SIPP include:
- Roughly doubling the sample size
- This improves the precision of estimates and shrinks standard errors
- Lengthening the panels from 3 years to 4 years
- This reduces the severity of the regression to the median problem
- Introducing computer assisted interviewing to improve data collection and reduce errors or the need to impute for missing data
- Introducing oversampling of low income neighborhoods
- This mitigates response bias issues we previously discussed, which are most likely to affect the bottom of the distribution
- New income
topcoding procedures were instituted with the 1996 Panel
- This will affect both means and various points in the distribution
- Topcoding is done on a monthly or quarterly basis, and can therefore undercount end of year bonuses, even for those who are not extremely high income year-round
Most government surveys topcode income data—that is, there is a maximum income that they will report. This is done to protect the privacy of high-income individuals who could more easily be identified from ostensibly confidential survey data if their incomes were revealed.
Because law graduates tend to have higher incomes than bachelor’s, topcoding introduces downward bias to earnings premiums estimates. Midstream changes to topcoding procedures can change this bias and create problems with respect to consistency and continuity.
Without going into more detail, the topcoding procedure that began in 1996 appears to be an improvement over the earlier topcoding procedure.
These are only a subset of the problems extending the SIPP data back past 1996 would have introduced. For us, the costs of backfilling data appear to outweigh the benefits. If other parties wish to pursue that course, we'll be interested in what they find, just as we hope others were interested in our findings.
Brian Tamanaha’s Straw Men (Overview)
Brian Tamanaha previously told Inside Higher Education that our research only looked at average earnings premiums and did not consider the low end of the distribution. Dylan Matthews at the Washington Post reported that Professor Tamanaha’s description of our research was “false”.
In his latest post, Professor Tamanaha combines interesting critiques with some not very interesting errors and claims that are not supported by data. Responding to his blog post is a little tricky as his ongoing edits rendered it something of a moving target. While we're happy with improvements, a PDF of the version to which we are responding is available here just so we all know what page we're on.
Some of Tamanaha’s new errors are surprising, because they come after an email exchange with him in which we addressed them. For example, Tamanaha’s description of our approach to ability sorting constitutes a gross misreading of our research. Tamanaha also references the wrong chart for earnings premium trends and misinterprets confidence intervals. And his description of our present value calculations is way off the mark.
Here are some quick bullet point responses, with details below in subsequent posts:
- Using more historical data from SIPP would likely have introduced continuity and other methodological problems
- Using more years of data is as likely to increase the historical earnings premium as to reduce it
- If pre-1996 historical data finds lower earnings premiums, that may suggest a long term upward trend and could mean that our estimates of flat future earnings premiums are too conservative and the premium estimates should be higher
- The earnings premium in the future is just as likely to be higher as it is to be lower than it was in 1996-2011
- In the future, the earnings premium would have to be lower by **85 percent** for an investment in law school to destroy economic value at the median
- 16 years of data is more than is used in similar studies to establish a baseline. This includes studies Tamanaha cited and praised in his book.
- Our data includes both peaks and troughs in the cycle. Across the cycle, law graduates earn substantially more than bachelor’s.
errors and misreading
- We control for ability sorting and selection using extensive controls for socio-economic, academic, and demographic characteristics
- This substantially reduces our earnings premium estimates
- Any lingering ability sorting and selection is likely offset by response bias in SIPP, topcoding, and other problems that cut in the opposite direction
- Tamanaha references the wrong chart for earnings premium trends and misinterprets confidence intervals
- Tamanaha is confused about present value, opportunity cost, and discounting
- Our in-school earnings are based on data, but, in any event, “correcting” to zero would not meaningfully change our conclusions
- “Let me also confirm that [Simkovic & McIntyre’s] study is far more sophisticated than my admittedly crude efforts.”
July 23, 2013
"How to fix law schools"? TNR asked six "experts" [sic]...
...meaning one gossip columnist who went to law school, one longtime law professor, two journalists who went to law school, one "journalist moonlighting as a law professor", and one in-house counsel. In short, they asked no experts on legal pedagogy, the economics of legal education, or the legal profession, just a group of pontificators (plus one lawyer) with varying degrees of familiarity with law schools. Most of what they have to say is silly and irrelevant, with two exceptions: the longtime law professor (Dershowitz) offers his own version of the 2-year JD idea that others have floated; and the in-house counsel for Cisco reveals an interesting plan, which should offer some useful experience to some students:
Colorado University Law School's visionary dean Philip J. Weiser is working to implement this program next year: Students will work as interns at Cisco for seven months–from June of the second year of law school until the following January, and potentially part-time during the following spring. We will pay them as we do our customary interns, and the students will not be required to pay tuition to the law school for the fall semester. The students will be supervised by a faculty member during the fall through an intensive credit-earning independent study project, and the students will also take extra courses during the rest of law school to ensure sufficient ABA-approved credits to graduate.
One can guess which member of the Colorado law faculty will not be asked to supervise anyone!
TNR also ran a sensationalistic and fact-free article on Chicago-based Mayer Brown in the same issue. Even I would expect a bit more from TNR, but when it comes to law schools, careless journalism is increasingly the norm.
July 22, 2013
Above the Law Needs to Read More
Elie Mystal and Joe Patrice at Above the Law misrepresented the contents of The Economic Value of a Law Degree. Mr. Mystal and Mr. Patrice are entitled to disagree with us but it would help if they would first carefully read what we wrote. I realize the paper is long, but a lot of the problems below appear to come from jumping to conclusions without reading.
Here are six of ATL’s inaccurate claims, with the facts listed below:
(1) ATL Claim: The article doesn’t present any earnings data after 2008 and ignores changes since the financial crisis
Facts: We present earnings data from 1996 to 2011. We also check for trends. As shown in the figure below, the earnings premiums in recent years are within historical norms.
Earnings declined for both law degree holder and bachelor degree holders, and law degree holders maintained their relative advantage.
We note in the article that the most recent college and law graduates in our sample graduated in 2008. This is because the Census Bureau reports educational attainment at the start of each Panel and the most recent panel started in 2008. Each panel tracks earnings for several years.
We separately analyze earnings premiums for young law degree holders and bachelor's degree holders, and do not find evidence that the earnings premium has compressed.
(2) ATL Claim: The article relies on averages (or means) and doesn’t provide any information relevant to students who attended lower ranked law schools
(3) ATL Claim: The article doesn’t present after-tax earnings
Facts: We present both pre-tax and after-tax present value figures, and discuss both the private benefit of the law degree to the student and the public benefit to the federal government in the form of higher tax revenue.
The discussion of taxes appears on pages 42 to 44 of the article, but we will relieve the suspense: the Federal government takes about a third of the extra earnings.
(4) ATL Claim: The article doesn’t consider the cost of tuition
Facts: The article presents both benefits (higher earnings) and costs (such as tuition). We include a range of annual tuition values between 0 and $60,000 in our Internal Rate of Return calculation in Table 10 on page 68 of the article. The Internal Rate of Return is typically between 10 and 25 percent (in real terms, net inflation), depending on tuition and the point in the distribution where the law degree holder falls.
We list present value figures prior to deducting tuition in the article so that the cost of the degree—tuition and fees less grants and scholarships—can readily be subtracted from the value of the degree—the earnings premium. We also cite to ABA data about law school tuition, both before and after scholarships and grants.
Mr. Mystal is correct that the $1 million figure is pre-tax and pre-tuition—the article says as much. However, after deducting taxes, and tuition, and even close to the bottom of the distribution, it appears that a law degree is very often a profitable investment.
How profitable varies from person to person. There are surely a few individuals who do not benefit at all or even do worse. If we had a way to identify these unlucky people before law school we would encourage them to avoid law school. Judging from student loan default rates, this is a relatively rare outcome. Further, income based repayment of loans should substantially mitigate this downside, even as it makes it difficult for us to nail down exactly how much a given person pays for law school.
(5) ATL Claim: The article simply “assumes” that law degree holders who don’t practice law benefit from a law degree
Facts: We don’t assume. We measure earnings for all law degree holders in sample, regardless of eventual occupation. The data suggests that the earnings premium is significant, even including those who do not practice law.
(6) ATL Claim: The article compares apples to oranges and fails to control for differences between bachelor’s and law degree holders
Facts: We control for many demographic, academic, and socio-economic characteristics other than law school attendance that predict earnings. In a supplemental analysis using data from the National Education Longitudinal Study, we incorporate additional control variables and tests for ability sorting and selection.
Our controls for ability sorting and selection are described at length in Part II and Appendix A of The Economic Value of a Law Degree.
Bottom line: read the paper.
July 21, 2013
The Economic Value of a Law Degree: Week 1 Summary
In The Economic Value of a Law Degree Frank McIntyre and I measure differences in annual earnings and hourly wages between those with law degrees and similar individuals who end their education with a bachelor’s degree. We account for unemployment and disability risk.
We also control for many demographic, academic, and socio-economic characteristics other than law school attendance that predict earnings. In a supplemental analysis using data from the National Education Longitudinal Study, we incorporate additional control variables and tests for ability sorting and selection.
The Economic Value of a Law Degree was covered by:
- Steven Davidoff at the New York Times Dealbook
- Dylan Matthews at the Washington Post
- Jordan Weissmann at the Atlantic Monthly
- Lauren Ingeno at Inside Higher Education
- Debra Cassens Weiss at the ABA Journal
- Christina Sterbenz at Business Insider
Our first blog post at Concurring Opinions ends with a presentation of annual earnings premiums at the mean and median, as well as at the high and low end of the distribution.
Questions and Critiques
The low and and high ends of the distribution
- Brian Tamanaha mischaracterized our research as only presenting means in a quote to Inside Higher Education.
- We pointed out the error.
- Dylan Matthews at the Washington Post reported that Professor Tamanaha’s description of our research was “false.”
- Brian Tamanaha posted and emailed several new questions and comments, which we will begin to respond to this week.
Representativeness of the data
- We responded to questions about the representativeness of the U.S. Census Bureau’s Survey of Income and Program Participation (“SIPP”).
- Paul Campos, Jack Graves, Brian Tamanaha (in a comment below the post), and Derek Tokaz (in a comment below the post) misunderstood net present value and double-counted opportunity costs. Campos, Graves, and Tokaz arrived at median after-tax, after-tuition net present values for a law degree that are too low by hundreds of thousands of dollars.
- Tamanaha erroneously included undergraduate debt as a cost of attending law school.
- Stephen Diamond explained Net Present Value and Opportunity Cost and performed the correct calculation, and noted that the median after-tax, after-tuition net present value of the law degree was approximately $330,000 as of the start of law school.
- Adam Levitin, Jordan Weissmann, and Deborah Merritt ask how much we can learn about the future from data about the past. We will respond the week after next.
- John Steele at Legal Ethics Forum reports that according to NALP, median full time starting salaries increased dramatically between 1996 to 2011. He forgets to take inflation into account. In real terms, median starting salaries exhibited a pattern of cyclicality.
- Adam Levitin asks whether law school is underpriced.
Confusion at Above the Law
- Above the Law subsequently posted a correction.
- Above the Law mischaracterizes our research again, overlooking our careful controls for ability sorting and selection, described at length in Part II and Appendix A of The Economic Value of a Law Degree.
July 20, 2013
Diamond on Campos on the economic value of a law degree
July 19, 2013
Professor Simkovic will be guest-blogging here next weekHe will be addressing some of the criticisms of his co-authored work on the economic value of a law degree, as well as correcting many of the errors about the content of the piece that have been put into circulation by people who most likely did not read the paper, or, if they did, plainly didn't understand it.
Posted by Brian Leiter on July 19, 2013 | Permalink
July 18, 2013
Some adult coverage and commentary on "The Economic Value of a Law Degree"Here.
Simkovic v. Tamanaha on the economic value of a law degree
UPDATE: This is also helpful.
July 17, 2013
"The Economic Value of a Law Degree"
This article ought to fundamentally change the conversation about the economic value of legal education; it's considerably more sophisticated methodologically than anything I've seen, and it sensibly compares the value of the JD to the alternatives, such as only having a BA. (A peculiarity of the cyber-ranting about "don't go to law school" is that it never explains, or even seems to care, what happens or would happen to those who forego the JD.) There's also a powerpoint version which contains most (not all) of the key data and analysis.
UPDATE: Predictably, serious research and analysis is met with derision and and insults from the know-nothing crowd. Professor Simkovic sent the following measured reply to Mr. Mystal's nonsense, which makes clear that Mystal didn't even read the article before ranting about it:
UPDATE: Professsor Simkovic will be blogging about his research here.
Your coverage of our research, "The Economic Value of a Law Degree" contains several inaccuracies that you may wish to correct.
1) The study does not only look at averages, as you state in your post. It also considers the median, 25th percentile and 75th percentile outcomes. Even at the 25th percentile, the value of a law degree exceeds the cost.
2) The study includes earnings data from 1996 to 2011--the coverage does not stop in 2008 as you state. The most recent law school and college graduates in sample for the earnings portion of the study are from the class of 2008, but earnings are reported through 2011.
3) The study notes the typical tax rate on the earnings premium and reports both pre-tax and after tax-values. It does not only report the value of a law degree before taxes, as you state.
4) The study includes a range of possible tuition values in our internal return calculations and explains clearly how to compare the cost of the degree to the value of the degree. It does not simply provide "theoretical never-never-land [estimates] where things like TUITION don’t matter", as you state.
5) The study presents student loan default rates for many low-ranked law schools. The default rates are lower than average default rates for former college and graduate school students.