January 19, 2017
Established datasets, proxies, and customized data collection: The case of international LLMs (Michael Simkovic)
How should researchers make tradeoffs between the costs of data collection, the speed of the analysis, the precision of the measurements, reproducibility by other researchers, and broader context about the meaning of the data: how we might compare one group or one course of action to another, how we might understand historical trends, and the like?
Must we always measure the precise group of interest, with zero tolerance for over-inclusion or under-inclusion? Or might one or a series of proxy groups be sufficient, or even preferable for some purposes? What if the proxies have substantial overlap with the groups of interest and biases introduced by use of proxy groups are reasonably well understood? How close must the proxy group be to the group of interest?
These are important questions raised by a group of legal profession researchers which includes several of the principal investigators of the widely used After the JD dataset.
Professors Carole Silver, Ethan Michelson, Robert Nelson, Nancy Reichman, Rebecca Sandefur, and Joyce Sterling (hereinafter, Silver et al.) recently wrote a three-part response (Parts 1, 2, and 3) to my two-part blog post from December about International LLM students who remain in the United States (Part 1) and International LLM students who return to their home countries (Part 2). The bulk of Silver et al.’s critique appears in Part 2 of their post, and focuses mainly on Part 1 of my LLM post.
My post, which I described as “a very preliminarily, quick analysis intended primarily to satisfy my own curiosity” used U.S. Census data from the American Community Survey and two proxy groups for international LLM (“Masters of Law”) graduates to make inferences about the financial benefits of LLM degrees to international students who remain in the U.S. Silver et al. agree with several of the limitations of this analysis that I noted in paragraphs 5 through 8 of Part 1 of my post. They also note that historically, many LLMs have returned to their home countries and argue that the benefits of LLM programs to returning students may be greater than the benefits to those who remain in the United States. (While I am skeptical of this last claim—especially if we focus exclusively on pecuniary benefits—it seems likely that both groups benefit).
Silver et al. have also helpfully made several additional points about limitations in my proxy approach and ways in which proxies could over-count or under-count foreign LLMs. The most important of these limitations can be addressed with a few modifications to the LLM proxy group approach. Those interested in the technical details are encouraged to read footnote 1 below.
Returning to broader questions about the use of proxy groups, my view is that proxy groups can be helpful and potentially necessary for certain kinds of analysis.
Suppose that we wish to know the temperature in New York’s Central Park before we take a stroll, but we only have temperature readings for LaGuardia and Newark airport. While neither of those proxies will tell us the precise temperature in Central Park, they will usually be sufficiently close that we can ascertain with a reasonable degree of certainty whether we should bring our winter coats, wear sweaters, or proceed with short sleeves. Indeed, readings from Boston or Philadelphia will probably suffice, particularly if we’re aware of the direction and magnitude of typical temperature differences relative to Central Park.
Should we refuse to venture out until we can obtain a temperature reading from Central Park itself?
December 14, 2016
Republican Tax Plan likely to cause "an explosive rise in federal debt" according to Centrist former Treasury Secretary Lawrence Summers (Michael Simkovic)
Former Secretary of the Treasury Lawrence Summers recently warned that President Elect Donald Trump's proposed tax reform plan "will massively favor the top 1 per cent of income earners, threaten an explosive rise in federal debt, complicate the tax code and do little if anything to spur growth."
Summers served as Secretary of the Treasury during the Clinton Administration, during one of the few periods in the last 4 decades when the Federal Government ran a surplus budget. Summers is a Professor of Economics at Harvard University, served as the President of Harvard University, was the Chief Economist of the World Bank, was the Director of the National Economic Council, and was a managing partner at hedge fund D.E. Shaw.
Summers is widely regarded as data-driven, rigorous, and centrist (Summers has complained about "absurd political correctness" in academe and his potential nomination as Chairman of the Federal Reserve was opposed by Progressive Democrats).
"Unfortunately, neither the Trump plan, nor the one put forward by Paul Ryan, speaker of the House of Representatives, provides for nearly enough base-broadening to finance all the high-end tax cutting they include.
Steven Mnuchin, Treasury secretary-designate, asserts there will be no absolute tax cut for the upper class because deductions would be scaled back. The rub is that totally eliminating all deductions for those with incomes over $1m would not even raise enough revenue to cover reducing their marginal tax rates from 39 to 33 per cent, let alone offset their benefit from huge rate reductions on business and corporate income, and the elimination of estate and gift taxes.
Estimates of the Trump plan suggest that it will raise the average after-tax income of the 0.9 per cent of the population with incomes over $1m by 14 per cent, or more than $215,000. This contrasts with proposed tax cuts for those in the middle of the income distribution of $1,000, or about 2 per cent.
The repeal of estate and gift taxes is especially problematic because it would provide a window for the very rich to use gift and trust structures to ensure that their wealth passes without tax not just to their children but to their grandchildren and great grandchildren, regardless of subsequent legislation. . . .
The envisioned Trump tax cut is about the same size relative to the economy as the 1981 Reagan tax cut. It is worth remembering that Reagan, hardly a fan of reversing course or raising taxes, found it necessary to propose significant tax increases in 1982 and 1984 (the equivalent in today’s economy of $3.5tn over a decade) due to concerns about federal debt.
August 19, 2016
"There's a new book out about the student loan crisis [Student Debt: Rhetoric and Realities of Higher Education], or what author Sandy Baum suggests is a "bogus crisis." Baum, a financial aid expert and senior fellow at the Urban Institute, claims it has been [sensationalized and exaggerated] by the media in search of a spicy story and fueled by politicians pushing "debt free college" proposals. . . . "
"People who earn bachelor's degrees, by and large, do fine.
The problem is that we have a lot of people actually borrowing small amounts of money, going to college, not completing [a degree] or completing credentials that don't have labor market value. They tend to be older. They tend to come from disadvantaged or middle-income families and they're struggling. [But] not because they owe a lot of money. . . .
Its not realistic to say we're going to pay people to go to college [for free]. Someone has to pay. We can have everyone pay much higher taxes. But short of that, it's not clear how we would pay. . . .
There are some people who borrowed under fraudulent, deceptive situations and their debt should be forgiven. There are people for whom education did not work out through no fault of their own and their debt should be forgiven. . . . We don't give people very much advice and guidance about where [and] when to go to college, how to pay for it, what to study. . . .
[[There are facts that] get little or no attention because they don't fit the "crisis" narrative:
- A third of college students who earn a four-year degree graduate with no debt at all. Zero.
- A fourth graduate with debt of no more than $20,000.
- Low-income households hold only 11 percent of all outstanding [student] debt.
- Almost half of the $1.3 trillion in student loan debt is held by 25 percent of graduates who are actually making a pretty high income.]
This is an investment that pays off really well. The median earnings for young bachelor's degree recipients is about $20,000 a year higher than the median earnings for high school graduates.
Student debt is really creating a lot of opportunities for people. People wouldn't be able to go to college otherwise."
Baum notes that many graduates with high debt levels (>$100,000) have advanced degrees, high expected incomes, and low default rates.
"The highest debt levels are for those earning professional degrees . . . Despite high debt levels, default rates among graduate borrowers are very low." However, Baum expresses some concern about those pursuing expensive master's degrees in fields "that rarely lead to the kind of earnings that doctors, lawyers, and MBAs can expect."
Baum's findings are broadly consistent with recent research by Beth Akers and Matthew Chingos, reviewed by David Leonhardt for the New York Times. Akers and Chingos have a new book coming out this fall.
August 17, 2016
New York Times journalist Elizabeth Olson recently reported that the law school graduating class of 2015--which was very close to the size of the class of 1996--had about the same number of private sector jobs 9 months after graduation as the class of 1996. That's a pretty good outcome considering that the economy-wide employment population ratio in February 2016 was 3.6 points lower than in February 1997. Olson puts a negative spin on the non-story.
UPDATE: Casey Sullivan at Bloomberg provides more balanced coverage, noting the smaller class size at the outset of his story and focusing on overall earnings rather than job counts in one segment of the market.
For previous coverage, see
Timing Law School (forthcoming in JELS)
August 15, 2016
“Glass Half Full” author concedes problems with estimates of solo practitioner incomes and headcounts (updated 8/18)
Professor Benjamin H. Barton recently responded to critiques of his estimates of solo practitioner incomes. Barton does not answer the specific questions that I posed about his use of IRS data, but he generally concedes that the IRS data is problematic.
- Barton wrote:
“Is it possible that the IRS data undersells the earnings of solo practitioners? Yes, for the reasons I state above and for some of the reasons that you and Professor Diamond point out.”
- Barton wrote:
“Do I think that the IRS data are off by a factor of 3.5 or even 2? No.”
I encourage Professor Barton to present a revised estimate that he thinks is more accurate. Several studies that he cites for support suggest that his solo income estimates are off by a factor of approximately 2 to 3 (see below for details).
- Barton defends his use of IRS data on three grounds, each of which is problematic:
a. “The IRS data on lawyer earnings is the longest running data I could find and thus the best dataset for a discussion of long term trends.”
Professor Barton overlooked the U.S. Census Bureau’s Decennial Census, which has data on Lawyer’s incomes since 1950 (which reports 1949 incomes).[i] The IRS data presented by Barton starts 18 years later, in 1967.
When considering long term trends in occupational incomes, it’s important to consider changes in the race and sex of members of the occupation. Across occupations, women and minorities generally earn less than white men. Race and sex variables are available in Census Household data, but not public-use IRS data.
b. The IRS data “separates lawyer earnings into solo practitioners and law firm partners”
Professor Barton acknowledges that his data misses incorporated self-employed lawyers, and that this group likely has higher incomes than those that he captures.[ii]
This means that Professor Barton’s IRS data is much less useful for identifying small and solo practitioners in 2013 than it was in 1970. This is because the proportion of solo and small attorneys who incorporated has likely increased dramatically. In 1970, 5 percent of full-time self-employed lawyers were incorporated. By 2014, the share increased to more than 50 percent.[iii]. Barton is missing many solo and small time practitioners. If trends toward incorporation continue, his data will become less useful every passing year. The IRS data has different biases at different points in time, making trends potentially unreliable.
August 11, 2016
Benjamin H. Barton Responds to Critics of Solo Practitioner Income Estimates: "IRS and Census Data Not that Far Apart Upon Closer Inspection"
The following is a response from Professor Benjamin H. Barton to critiques and questions about his use of IRS data to estimate solo practitioner incomes. It has not been edited or altered from the form in which Professor Barton submitted it.
IRS and Census Data Not that Far Apart Upon Closer Inspection
On July 25th Professor Stephen Diamond criticized my use of IRS income statistics to discuss the earnings of solo practitioners on his blog. I responded to Professor Diamond in the comments. On July 26, 2016 Professor Michael Simkovic published a number of critiques here. Two days later Professor Simkovic followed up with a second post asking me a series of questions and challenging me to respond to both of his posts. Here I accept Professor Simkovic’s invitation.
Below I explain more about the IRS data and how I use it, but I will not bury the lede. The data that Professors Simkovic and Diamond use to criticize my work, ACS data for lawyers who are in the category of “self-employed, not incorporated,” is not appropriate data for defining the earnings of solo practitioners. That Census category likely includes two very different types of self-employed lawyers – solo practitioners (the lowest paid lawyers in private practice) and law firm partners (the highest paid lawyers in private practice). The Census Department does not make it easy to figure out exactly which lawyers are counted in the category of “self-employed, not incorporated,” but combining this definition with this one and looking at the ACS form itself it seems pretty clear that partners in law firms are included in this category.[i]
Because the ACS data includes an indeterminate number of partners and solos, the average earnings in that category ($165-200,000) are a misleading proxy for the earnings of American solo practitioners. If there was a data category of “professional baseball players” that included minor league (low paid) and major league (highly paid) baseball players, and there was no way to tell how many of each were in the sample, you could not use the average earnings of “all professional baseball players” as a proxy for minor league salaries, since some members of the sample earn much, much more than other members of the sample.
The ACS data is inappropriate, but is the IRS data better? I use the IRS data in my book, Glass Half Full – The Decline and Rebirth of the Legal Profession (Oxford 2015) and in later work to talk about several trends in the market for legal services. Here is an updated version of a chart I first created for the book:
July 28, 2016
Some questions for Professor Benjamin H. Barton about his use of IRS data to estimate solo practitioner incomes (Michael Simkovic)
After Tuesday's post explaining why IRS schedule C data dramatically underestimates incomes for solo practitioners and other sole proprietors, Professor Benjamin H. Barton emailed to indicate that his views remained unchanged and he did not intend to respond beyond his previous comments on Professor Stephen Diamond's blog. Barton's comments did not address many of the issues I raised.
On Wednesday, I asked Professor Barton to consider the following questions:
1) Do you think that 20 million or so U.S. small business owners are living below the poverty threshold for a 2 person household?
2) Do you think the IRS is wrong about its own data and schedule C does not in fact understate net income? Why do you think that you understand IRS data, IRS enforcement capabilities, and the level of tax evasion better than the IRS?
3) Do you think that everyone who files schedule C has no other sources of income?
4) Do you think that Treasury and JCT estimates of tax expenditures are way off and exclusions and deductions from tax concepts of income are negligible?
5) If apples to apples comparisons using schedule C data show that legal services sole proprietorships are more profitable than 97 percent of sole proprietorships, is that something you should mention? Would you at least agree that using schedule C data for legal services and census data for everyone else is a methodological error?
Professor Barton has not yet responded.
Aug. 11, 2016. Professor Barton responded without specifically answering the questions above, but generally conceded that IRS data is problematic.
Aug. 15, 2016. I replied to Barton.
June 24, 2016
Last week I wrote an open letter to New York Times reporter Noam Scheiber discussing problems with his law school coverage and his reliance on low quality sources such as internet blogs and "experts" who lack relevant expertise rather than peer reviewed labor economics research. By email, Scheiber insisted that there was nothing wrong with his coverage, but he'd be happy to hear of any specific factual problems I could identify.
I identified 6 clear factual errors and multiple misleading statements. I also reinterviewed his lead source, John Acosta and found important discrepancies between how Scheiber depicted Acosta as someone who was suckered into un-repayable debt, while Acosta describes his own situation as hopeful and law school as a worthwhile and carefully researched investment. New York Times Dealbook reporter and U.C. Berkeley Professor Steven Davidoff Solomon weighed in, citing my research and supporting my points.
Scheiber posted a response to his facebook page, after running it by his editors at the New York Times. The New York Times agreed to correct the most minor of the six errors I identified. They also "tweaked" two sentences so that the language was less definitive.
Scheiber's response includes some good points (many students from Valparaiso might be below the 25th percentile of law school graduates) as well as strained interpretations of the language of his original article: "fewer" did not actually mean "fewer"'; "Harvardesque" did not actually mean "similar to Harvard." Scheiber describes my presentation of data that contradicts his factual claims as "strange", "bizarre", "odd", "overly-literal" and (on Twitter) "gripes." Interestingly, Scheiber thinks that "most law school graduates who pass the bar are going to have at least a few hundred thousand dollars in assets like 401k and home equity by the time they work for 20 years." This level of savings would make them far more financially secure than the vast majority of the U.S. population.
My response to Scheiber is below. I explain why The New York Times has an obligation to its readers to correct the remaining uncorrected factual errors in Scheiber's story.
Scheiber embedded his response in my explanation of the 6 clear factual errors in his story, and I in turn embedded my response within his response. To ease readability, I have color coded Scheiber's response in orange, and my new response in blue. Scheiber's response is indented once, and my new response is indented twice. The least indented black text at the beginning of each thread is from the list of 6 clear factual errors, and can be skipped (scroll down until you see orange or blue text) by those who have followed the discussion thus far.
UPDATE: June 25, 2016: Yesterday, The New York Times posted an additional minor correction to its discussion of taxation of debt forgiveness, stating that debt forgiveness would "probably" be treated as taxable income. This is an improvement over the original, but could still mislead or confuse readers. It also leaves many of the most important errors uncorrected.
Scheiber tells me that the "tweaks" to the language which he communicated to me in his facebook post from Tuesday 6/21 actually happened on Friday evening 6/17. This would make them coincide with the timing of my open letter, but before my more detailed explanation of 6 clear factual errors. Scheiber tells me that these "tweaks" were not made in response to my letter, although he has not specified when on Friday evening the changes were made. They appear to have been made after I sent him the letter.
June 18, 2016
New York Times reporter Noam Scheiber was kind enough to respond to my open letter and ask if I could point to anything specifically factually wrong with his story. My response is below.
Thanks so much for responding. Yes, there are at least 6 factual errors in the article, and several misleading statements.
I’ll start with my interview with Acosta from earlier today, and then we can discuss empirics. Here’s what Acosta said:
"There’s no way I could pay back my student loans under a 10-year standard payment plan. With my current income, I can support myself and my family, but I need to keep my loan payments low for now. I’ve been practicing law since May, and I’m on track to make $40,000 this year. I think my income will go up over time, but I don’t know if it will be enough for me to pay back my loans without debt forgiveness after 20 years. What happens is up in the air. I’m optimistic that I can make this work and pay my student loans. I view the glass now as half full.
Valparaiso did not mislead me about employment prospects. I had done my research. I knew the job market was competitive going in. I knew what debt I was walking into. I think very few Americans don’t have debt, but for me it was an investment. I saw the debt as an investment in my career, my future, and my family.
Valparaiso gave a guy like me, a non-traditional student a shot at becoming a lawyer. Most law schools say they take a holistic approach, but they don’t really do it. I had to work hard to overcome adversity, and they gave me a shot to go to law school and to succeed. They gave me a shot at something that I wanted to do where most law schools wouldn’t.
My situation might be different from other law students who start law school right out of college. I was older and I have a family to support."
On to empirics.
The story states that:
“While demand for other white-collar jobs has rebounded since the recession, law firms and corporations are finding that they can make do with far fewer full-time lawyers than before.”
This is incorrect.
First, the number of jobs for lawyers has increased beyond pre-recession levels (2007 or earlier), both in absolute terms and relative to growth in overall employment. (error #1)
Focusing only on lawyers working full-time in law firms or for businesses (I’m not sure why you exclude those working in government), there are more full-time corporate and law firm lawyers in 2014 according to the U.S. Census Bureau’s Current Population Survey (CPS)—870,000—than in 2007—786,000. There have been more full-time corporate and law firm lawyers in every year from 2009 on than there were in 2007 and earlier.
You were looking at NALP or ABA data, which is measured at a single point in time—9 or 10 months after graduation—and is therefore much less representative of outcomes for law graduates—even recent law graduates—than Census data. Indeed, many law graduates who will eventually gain admission to a state bar will not have done so as of the date when NALP collects data. NALP and the ABA also use different definitions from the Census, so you cannot readily use their data to compare law graduates to others.
The trend of growth in lawyer jobs holds true for other cuts of the data (all lawyers; all full time lawyers) using other data sources—U.S. Census or Department of Labor (BLS OES) data.[i]
This is in spite of large declines in law school enrollments, which would be expected to reduce the number of working lawyers.
Second, employment has not rebounded to pre-recession (2007 or earlier) levels outside of law. (error #2)
June 17, 2016
An Open Letter to New York Times Journalist Noam Scheiber: Journalists Should Consult Peer-Reviewed Research, Not Bloggers (Michael Simkovic)
Dear Mr. Scheiber:
Have you seen this line of peer-reviewed research, which estimates the boost to earning from a law degree including the substantial proportion of law graduates who do not practice law?
- Michael Simkovic & Frank McIntyre, The Economic Value of a Law Degree, 43 J. Legal Stud. 249 (2014)
- Michael Simkovic & Frank McIntyre, The Economic Value of a Law Degree (2013)
- The Economic Value of a Law Degree PowerPoint Presentation
High quality nationally representative data from the U.S. Census Bureau, analyzed using standard and widely accepted econometric techniques, shows that even toward the bottom of the distribution, the value of a law degree (relative to a terminal bachelor’s degree) is much greater than the costs.
All of the data suggests that this has not changed since the financial crisis. The economy is worse and young people are facing more challenges in the job market, but law graduates continue to have the same relative advantage over bachelor’s degree holders as they have had in the past:
These findings have been covered in the New York Times before:
- Michael Simkovic, Overall Stagnation in Legal Jobs Hides Underlying Shifts, The New York Times Dealbook, April 1, 2016
- Steven Davidoff Solomon, Law Schools and Industry Show Signs of Life, Despite Forecasts of Doom, The New York Times Dealbook, March 31, 2015
- Steven Davidoff Solomon, Debating, Yet Again, the Worth of Law School, The New York Times Dealbook, June 18, 2013
Data from the U.S. Census and the Department of Labor Bureau of Labor Statistics shows that the number of lawyers has grown since the financial crisis, both in absolute terms and relative to overall employment.
Data from the Department of Education shows that law school graduates, even from very low-ranked law schools, have exceptionally low student loan default rates.
I have a number of concerns about factual inaccuracies in your recent story, “An Expensive Law Degree, and No Place to Use It” and your reliance on “experts” such as Paul Campos who lack any technical expertise or even basic financial or statistical literacy.
Your readers would receive more reliable information if you concentrated less on sources like Paul Campos and internet “scamblogs” and focused instead on peer-reviewed research by professional economists using high quality data and well-established methods of statistical analysis.
June 18, 2016: Noam Scheiber replies and I respond by re-interviewing Acosta and pointing out specific factual errors in Scheiber's story.
June 20, 2016: I explain different data sources that are useful for counting lawyers.
June 21, 2016: Steven Davidoff Solomon weighs in at N.Y. Times Dealbook, citing my research and supporting my points.
June 21, 2016, 10:05pm EST: Noam Scheiber sent a lengthy response by email and posted his response to his facebook page. Scheiber informs me that his response was reviewed by his editors at the New York Times.
June 24: I responded to Scheiber and explain Why The New York Times Should Correct The Remaining Factual Errors in Its Law School Coverage. In response, the New York Times posted a correction to the most minor of the 5 remaining errors.