April 27, 2015
New York Times relies on unrepresentative anecdotes and flawed study to provide slanted coverage of legal education (Michael Simkovic)
Just when you thought The New York Times was rounding the corner and starting to report responsibly about legal education based on hard data and serious labor economics studies, their reporting reverts to the unfortunate form it has taken for much of the last 5 years*—relying on unrepresentative anecdotes and citing fundamentally flawed working papers to paint legal education in a negative light.
Responsible press coverage would have put law graduate outcomes in context by noting that:
(1) law graduates continue to do better in terms of employment (both overall and full time) and earnings than similar bachelor’s degree holders, even in an economy that has generally been challenging for young workers
(2) law students, even from some of the lowest ranked and most widely criticized law schools, continue to have much lower student loan default rates than the national average across institutions according to standardized measurements reported by the Department of Education
(3) law graduate earnings and employment rates typically increase as they gain experience
(4) Data from After the JD shows that law graduates continue to pay down their student loans and approximately half of graduates from the class of 2001 paid them off completely within 12 years of graduation
Instead, The New York Times compares law graduate outcomes today to law graduate outcomes when the economy was booming. But not all law graduates. The Times focuses on law graduates who have been unusually unsuccessful in the job market or have unusually large amounts of debt. For example, The New York Times focused on a Columbia law school graduate working as an LSAT tutor** as if that were a typical outcome for graduates of elite law schools. But according to the National Law Journal, two-thirds of recent Columbia graduates were employed at NLJ 250 law firms (very high paying, very attractive jobs),*** and the overwhelming majority of recent Columbia graduates appear to work in attractive positions. (Columbia outcomes are much better than most, but the negative outcomes discussed in The New York Times are substantially below average for law graduates as a whole).
In Timing Law School, Frank McIntyre’s and I analyze long term outcomes for those who graduated into previous recessions, using nationally representative data and well-established econometric methods. Our results suggest that law graduates continue to derive substantial benefits from their law degrees even when graduating into a recession. The recent recession does not appear to be an exception. (See also here and here). This analysis is not mentioned in the recent The New York Times article, even though it was cited in The New York Times less than a month ago (and alluded to in The Washington Post even more recently).
The implication of The New York Times’ story “Burdened With Debt, Law School Graduates Struggle in Job Market” is that there is some law specific problem, when the reality is that the recession continues to negatively affect all young and inexperienced workers and law graduates continue to do better than most. Law school improves young workers’ chances of finding attractive employment opportunities and reduces the risk of defaulting on debt. The benefits of law school exceed the costs for the overwhelming majority of law school graduates.
The New York Times relies heavily on a deeply flawed working paper by Professor Deborah Merritt of Ohio State. Problems with this study were already explained by Professor Brian Galle:
“My problem is that instead DJM wants to offer us a dynamic analysis, comparing 2014 to 2011, and arguing that the resulting differential tells us that there has been a "structural shift" in the market for lawyers. It might be that the data exist somewhere to conduct that kind of analysis, but if so they aren't in the paper. Nearly all the analysis in the paper is built on the tend line between DJM's 2014 Ohio results and national-average survey results from NALP.
Let me say that again. Almost everything DJM says is built on a mathematical comparison between two different pools whose data were constructed using different methods. I would not blame you if now stopped reading."
In other words, it is difficult to tell whether any differences identified by Professor Merritt are:
(1) Due to differences between Ohio and the U.S. as a whole
(2) Due to differences in methodology between Merritt, NALP, and After the JD
(3) Actually due to differences between 2011 and 2014 for the same group
After Professor Galle’s devastating critique, journalists should have been extremely skeptical of Merritt’s methodology and her conclusions. Professor Merritt’s response to Galle’s critique, in the comments below his post, is not reassuring:
“Bottom line for me is that the comparison in law firm employment (62.1% for the Class of 2000 three years after graduation, 40.5% for the lawyers in my population) seems too stark to stem solely from different populations or different methods—particularly because other data show a more modest decline in law firm employment over time. But this is definitely an area in which we need much, much more research.”
Judging from this response and the quotes in The New York Times, Merritt appears to be doubling down on her inapposite comparisons rather than checking how much of her conclusions are due to potentially fatal methodological problems. What Professor Merritt should have done is replicate her 2014 Ohio-only methodology in 2000/2001 or 2010/2011, compared the results for Ohio only at different points in time, and limited her claims to an analysis of the Ohio legal employment market.
There are additional problems with Professor Merritt’s study (or at least the March 11 version that I reviewed).****
- Ohio is not a representative legal employment market, but rather a relatively low paying one where lawyers comprise a relatively small proportion of the workforce.
- A disproportionate share of the 8 or 9 law schools in Ohio (9 if you include Northern Kentucky) are low ranked or unranked, and this presumably is reflected in their employment outcomes.
- Merritt’s sample is subject to selection bias because of movement of the most capable law graduates out of Ohio and into higher paying legal markets. Ohio law graduates who do not take the Ohio bar after obtaining jobs in Chicago, New York, Washington D.C., or other leading markets will not show up in Merritt’s sample.
- Whereas Merritt concludes that law graduate outcomes have not improved, the data may simply reflect the fact that Ohio is a less robust employment market than the U.S. as a whole.
- Merritt’s analysis of employment categories does not take into account increases in earnings within employment categories. After the JD and follow-ups suggests that these within-category gains are substantial, as does overall increases in earnings from Census data.
- Merritt makes a biased assumption that anyone she could not reach is unemployed instead of gathering additional information about non-respondents and weighting the results to take into account response bias. Law schools may have been more aggressive in tracking down non-respondents than Professor Merritt was.
For the benefit of those who are curious, I am making my full 8 page critique of Professor Merritt's working paper available here, but please keep in mind that it was written in mid March and Professor Merritt may have addressed some of these issues in more recent versions of her paper. If that is the case, I trust that she’ll highlight any changes or improvements in a blog post response.
* A few weeks ago I asked a research assistant (a third year law student) to search for stories in The New York Times and Wall Street Journal about law school. Depending on whether the story would have made my research assistant more likely or less likely to want to go to law school when he was considering it or would have had no effect, he coded the stories as positive, negative, or neutral. According to my research assistant, The New York Times reported 7 negative stories to 1 positive story in 2011 and 5 negative stories to 1 positive story in 2012. In 2013, 2014, and 2015, The New York Times coverage was relatively balanced. In aggregate over the five-year period The New York Times reported about 2 negative stories for every 1 positive story. The Wall Street Journal’s coverage was even more slanted—about 3.75 negative stories for every positive story—and remained heavily biased toward negative stories throughout the five-year period.
** Professor Stephen Diamond notes the LSAT tutor’s relatively high hourly wage, more lucrative opportunities the tutor claims he turned down, and how the tutor describes his own work ethic.
*** For the class of 2010, the figure at Columbia was roughly 52 percent 9 months after graduation, but activity in the lateral recruitment market suggests things may be looking up.
**** The comments that follow summarize a lengthy (8 page) critique I sent to Professor Merritt privately in mid March after reviewing the March 11 draft of her paper. I have not had a chance to review Professor Merritt’s latest draft, and Professor Merritt may have responded to some of these issues in a revision.
April 27, 2015 in Advice for Academic Job Seekers, Guest Blogger: Michael Simkovic, Law in Cyberspace, Legal Profession, Of Academic Interest, Professional Advice, Science, Student Advice, Web/Tech, Weblogs | Permalink
April 23, 2015
...here and see the chart, which suggests we've hit bottom in terms of the applicant pool (barring dramatic economic jolts, that is, which could move things either way). Of course this is a bottom last seen in the 1970s when there were 50 fewer law schools. But given how many law schools have refrained from hiring faculty due to uncertainty about the future, my guesstimate is that we'll see a slight uptick in law school hiring next year, since many schools have unfilled needs.
April 09, 2015
It was another tight year on the law teaching market, probably even more difficult than last year. Happily most of our candidates once again were successful at securing tenure-track positions. They are:
Laura Napoli Coordes '10 who will join the faculty at Arizona State University, where she is presently a VAP. She graduated with Honors from the Law School, where she was a member of the Law Review and served as a Legal Fellow at the Student Press Law Center upon graduation, before working at Weil, Gotshal & Manges as a bankruptcy associate in New York. Her teaching and research interests include bankruptcy, commercial law, contracts, and corporate finance.
Goldburn P. Maynard, Jr. '05, who will join the faculty at the University of Louisville. He was a member of the Law Review at Chicago,and also earned an LL.M. in tax at Northwestern. He was a tax associate at Skadden Arps in Chicago, and then an estate tax attorney with the I.R.S. for four years. He was a VAP at Washington University, St. Louis and at Florida State University. His research and teaching interests include federal tax, estates and trusts, and estate and gift tax.
Joshua Sellers '08 who will join the faculty at the University of Oklahoma, Norman. At Chicago, he was Articles Editor of the Law Review and also earned a Ph.D. in Political Science with a dissertation on "The 'Crown Jewel' at a Crossroads: Appraising the Contemporary Political Function of the Voting Rights Act." He clerked for Judge Barkett on the U.S. Court of Appeals for the Eleventh Circuit, and was an associate at Jenner & Block in Washington, D.C. for three years, where he primarily litigated insurance claims. Most recently, he was a post-doc in the Maxwell School of Public Policy at Syracuse University. His research and teaching interests include election law, civil rights, constitutional law, legislation, insurance law, and torts.
Sloan G. Speck '07 who will join the faculty at the University of Colorado, Boulder. He graduated with Honors from the Law School, where he also served as Articles Editor of the Law Review. He also earned an M.A. in History from Chicago (where he focused on the history of tax and business) and an LL.M. in tax (2010) from New York University, where he has been Acting Assistant Professor of Tax Law since 2013. He will receive his PhD in History from Chicago in 2016. He was a tax associate at Skadden Arps in Chicago for five years. His teaching and research interests include tax law and policy (including corporate and international tax), as well as legal and business history.
Matthew J. Tokson '08 who will join the faculty at the Salmon P. Chase College of Law at Northern Kentucky University. He graduated with High Honors and Order of the Coif from the Law School, where he served as both Executive Articles Editor and Book Review Editor of the Law Review. He clerked for Judge Randolph on the U.S. Court of Appeals for the D.C. Circuit, served first as a Kauffman Fellow then as a Bigelow Fellow at the Law School from 2009-2011, before clerking on the U.S. Supreme Court for both Justice Ginsburg and Justice Souter in 2011-12. He was also a litigation associate at WilmerHale in Washington, D.C. His teaching and research interests include criminal procedure, privacy, intellectual property, judicial behavior, criminal law and torts.
(I will have a separate post about our Bigelows and other Fellows once their situations are all settled; as always, they all have tenure-track offers.)
March 25, 2015
Many legal educators believe that shrinking class sizes will help the students they do admit find higher paid work more easily and boost the value of legal education. They reason that if the supply of law graduates shrinks, then the market price law graduates can command should increase.
According to another hypothesis, now popular in the press, a decline in the number of law school applicants reflects the wisdom of the crowds. Students now realize that a law degree simply isn’t worth it, and smaller class sizes reflect a consensus prediction of worse outcomes for law graduates in the future.
Frank McIntyre and I investigated whether changes in law cohort size predict earnings premiums. Historically, they have not. Not for recent graduates, and not for law graduates overall. Nor have changes in cohort size predicted much of anything about the entry-level measures used by the National Association for Law Placement (NALP)—starting salary, initial employment, initial law firm employment.
How can both of these theories be wrong? One possibility is that they are both right, but the two effects offset each other. This seems unlikely however. If neither macroeconomic data nor Bureau of Labor Statistics (BLS) employment projections can predict law employment conditions at graduation, then how likely is it that recent college graduates with less information and less expertise could make a better prediction?
A more likely possibility is that there are other factors at play that prevent any strong predictions about the relationship between cohort size and outcomes / value added. For example, law schools may become less selective as cohort size shrinks and more selective as it increases. In addition, the resources available to law schools, and therefore the quality of education and training they are able to provide, may also change with cohort size. Since physical facilities expenses are not particularly variable in the ordinary course, most budgetary adjustments at law schools presumably take place with respect to personnel.
Anecdotally, many law schools appear to be managing the recent decline in enrollments by shrinking their faculties and administrations and using remaining personnel to teach classes and perform functions outside of their areas of expertise. Reduced specialization and a lack of economies of scale could affect the quality of service provided to students, offsetting any benefits to students from less competition at graduation.
Previous research in labor economics has found that resources per student are an important predictor of value added by college education, and that the use of adjuncts can lead to worse outcomes for students. (See here for a review)
Much of this is speculative—we do not yet understand why changes in cohort size do not predict law graduate outcomes, only that they do not predict outcomes. Given the historical data, it is probably not advisable to read too much into what the decline in law school enrollment means for students who will graduate over the next few years.
Instead, we should focus on the long-term historical data and the value of a law degree across economic cycles and enrollment levels.
January 26, 2015
October 26, 2014
LSAC data here. Since the trend appears to be for applicants to apply later in the cycle, the final decline for this year is likely to be less than 8% (in the last few years, the September/October drop was always greater than the final total decline). But a continued decline of any kind means that law schools uncertain about whether to hire new faculty will likely err on the side of not hiring.
October 20, 2014
October 15, 2014
Here's a couple of words of advice I typically share with Chicago candidates, but others might appreciate:
First, although this can be stressful, it should also be fun: lots of law faculty will want to talk about you and your ideas over the next couple of days! You will form intellectual and professional relationships even from interviews that don't lead to callbacks. Enjoy the scholarly dialogue and learn from it.
Second, remember that every hiring committee is a black box: you don't know its internal priorities and squabbles, its biases and agendas. So don't waste time speculating about how you did (candidates, in my experience, are uneven judges of their performance, in both directions), and remember you are bound to bomb an interview, but life will go on. Forget about it.
Third, bear in mind that hiring committees come to the hiring convention with different charges from their home schools. Some will be authorized to offer some callbacks even before the weekend is out; others will have to report back to the rest of the committee at home before doing anything. Don't draw inferences from silence, or from the fact that someone you know got a callback before the weekend was over--even when hiring committees are allowed to make some quick callback offers, it's almost always the case that the full hiring committee back home will make decisions about other callbacks at a later date.
Best of luck to all the job seekers out there!
October 08, 2014
Barry Friedman (NYU) writes with an excellent set of questions and observations:
Here’s a thought worth maybe tooting on your blog. It never ceases to catch my attention how much school hiring is driven by signals from other schools. School X will interview candidate Y and love him/her, or will love him/her on paper, but will never move forward for an interview absent a strong signal from some number of schools they consider competitive. Yet, in this tight market, those signals get fewer – especially at the call back and offer stage. It has the effect I think of killing candidates that otherwise would get interviews or offers. Yet, paradoxically, if schools had confidence in their internal assessments (and it is not like this is one person deciding; it is an entire faculty or faculty committee) this sort of market provides a real opportunity to steal that person you loved without a fight.
So why do schools do this? I think in most cases it is because they lack confidence in their own judgments. But what do readers think? I would prefer signed comments, but you must, in any case, include a valid e-mail address, which will not appear.
September 22, 2014
Schools vary in their procedures for scheduling interviews with candidates at the "meat market," but a typical pattern is this: after an initial cull of candidates in the first AALS distribution, schools begin doing "due diligence," which typically means talking to references and reading work by the candidates. Appointments committees usually only meet once a week. At each meeting, the Committee will take a decision on some of the candidates they've been reviewing, and then contact them to schedule interviews. The same thing will happen the following week and so on, until all the spots are filled. For schools that do a lot of 'due diligence,' the process of scheduling 15 or 20 candidates to see could easily take four weeks. It's important for candidates to realize that this is how many schools proceed, so the fact that an anonymous person on some blog reports they have an interview at school X does not mean you, a hopeful candidate, will not get an interview with school X. School X may have only just begun, and may be scheduling interviews for weeks to come.