May 07, 2015
One question in the labor economics literature is why education increases earnings. The dominant view is Human Capital Theory—education increases earnings by increasing productivity of educated workers. A minority view, signaling theory, holds that education increases earnings by revealing information about potential employees to employers and facilitating the employer-employee matching process, similar to marketing expenditures matching products with customers. In other words, education indicates to employers which employees were the best all along.
On some level, the distinction is irrelevant—from the perspective of potential students it does not really matter how education increases earnings, only that it does. Whether education makes workers more efficient individually or makes the economy in aggregate more efficient by moving workers to where they can do the most good does not change the fact that employers, employees, and the economy as a whole are better off with additional education as long as the marginal benefits of additional education exceed marginal costs.
Critics of higher education sometimes claim that there is over-investment in signaling and credentialing, a kind of arms race producing negative externalities in which the social returns to education are lower than the private returns. This view, though popular with political movements seeking to reduce public support for education, does not hold much sway in the modern peer-reviewed labor economics literature.
A recent literature review by Professors Fabian Lange at Yale and Robert Topel at the University of Chicago explains the problems with the view that employers would expend valuable resources paying a premium for employees who had over-invested in education as a signal of ability: “Employer learning about productivity occurs fairly quickly after labor market entry, implying that the signaling effects of schooling are small.”
In other words, employers can quickly and efficiently sort employees on their own by observing their productivity on the job, retaining and promoting strong performers, and terminating weaker employees. Employers do not need educational institutions to perform this task for them, nor are employers willing to pay premium wages for information they can more cheaply obtain themselves.
Suppose, for example, that the only value of a Harvard Law degree were as signal of ability to employers. Harvard admits students based almost exclusively on standardized test scores and undergraduate grades, which are also observable to employers. Almost everyone who is admitted to Harvard graduates.
A clever employer realizes it can save money by employing bachelor’s degree holders with admissions letters from Harvard, but who have not yet attended. Harvard gets wise and refuses to confirm admissions. Employers now look directly at LSATs and GPAs, and perhaps hire former admissions officers to consider softer factors. The employer and employee can effectively split the cost savings of not getting the Harvard degree and also the value of the time that would be spent in schooling instead of working.* If additional information is required, personality testing, assessments of physical and mental health, assessments of writing samples, background checks and the like can all be performed for a fraction of the cost and in a fraction of the time as a Harvard degree.
In spite of potentially massive efficiency gains and financial reward for employers and employees, this is not what actually happens in the real world (with the exception of a stacked "experiment" by Peter Thiel). Why not? Because employers believe that the schooling itself is valuable and makes employees more productive.
Lange and Topel also explain how “sheepskin effects” (disproportionately higher earnings premiums for college completers than for college dropouts with a few years of college) have been misinterpreted as support for the signaling hypothesis, when they actually reflect selection effects and dynamic decision making about educational investments:
"Diploma effects are often presented as evidence for screening theories of schooling. We disagree. Instead we view diploma effects as evidence that individuals face uncertainty about their individual returns to schooling and that this uncertainty is revealed as individuals acquire schooling. Those least capable to profit from schooling drop out before the completion of degree years. Those graduating exhibit larger returns than those who dropped out at lower levels of schooling. This reasoning was informally developed by Chiswick (1973). Since then, a number of authors (Altonji (1993), Heckman, Lochner and Todd (2003), Keane and Wolpin (1997) and Eckstein and Wolpin (1999)) examined different aspects of sequential schooling choice under uncertainty. We build a simple model in this spirit with the intention to show how individual’s schooling decisions can generate (large) diploma effects if individuals learn about their returns while in school."
There are additional problems with signaling theory, which I explained in Risk Based Student Loans:
“Signaling Theory implies that labor market outcomes should not depend on what students study, but only on how well they perform academically relative to other students with similar standardized test scores, or perhaps whether they demonstrate a strong work ethic by choosing a challenging major. Differences in earnings by field of study appear to reflect the value of field-specific skill development rather than differences in ability levels. Even within engineering, there are large starting wage differences by specialty.
Human Capital Theory also helps explain higher average per-capita productivity and wages in states and nations with higher levels of educational attainment. If education only sorted workers according to ability, it would presumably only increase the variance of wages (i.e., income inequality), while leaving the mean unaltered.
Further, Human Capital Theory helps explain the willingness of many employers to pay for professional degree programs for successful employees. Employers’ willingness to educate workers whom employers already know to be of high quality suggests that employers believe that professional education has skill-development value rather than mere sorting value.”
In The Knowledge Tax I expand on this, explaining the evidence linking investment in education to more rapid economic growth rather than merely redistributing income from the less educated to the more educated.
* Note that in many states, including New York and California which collectively comprise 28 percent of the law market, a law degree is not legally required to sit for the bar exam. Instead, “law office study”, or an apprenticeship under the supervision of an attorney, possibly following one year of law school, will suffice.
May 05, 2015
A better grading system
Professor Merritt argues that mandatory grading curves can be unfair when one class has stronger students than another. I agree.
Statistician Valen Johnson—whom I cite in my last post as an authority on grade inflation— has developed a clever solution to this problem which involves adjusting grading curves within each class based on the ability levels of the students. A Johnson-inspired proposal was nearly adopted at Duke University in the late 1990s, but was blocked by departments that offered higher grades and attracted weaker students.
Most law schools try to balance their sections in term of student ability levels and overall quality of faculty. Nevertheless, anomalies like a “smart section” (as Professor Merritt calls it) may occasionally occur. Johnson’s proposal would be an excellent solution to this problem.
Professor Merritt asserts that there is some sort of problem with the market for lawyers and law graduates that makes competition and inequality uniquely bad in the context of law. These assertions are implausible given the low barriers to entry for both law schools and lawyers, aggressive competition between law schools for students and between lawyers for clients, and widespread inequality outside of law school and legal practice. Some form of regulation is the norm in many areas of employment and in many industries, and a licensing regime for lawyers and an accreditation system for law schools do not in any way make these occupations and institutions unique or unusual. According to a recent study, nearly a third of U.S. workers are licensed, licensing is more common as education and skill levels increase, and licensing does not affect inequality among the licensed.
As a general matter, deregulated market competition and greater inequality are a package deal. Inequality can be reduced through regulation, taxation, and politicization of compensation through unionization or growth of public sector employment.
Professor Merritt’s critiques follow the standard playbook of law school critics—take something about law schools that is widespread and common out of context, claim that it is somehow unique to law schools when it is neither unique nor unusual, and then demonize it.
Jeremy Telman responds.
In her latest post, Deborah Merritt maintains that scholarships conditioned on maintaining a minimum GPA or class ranking are troubling when used by law schools, even though such conditions are routinely used by other educational institutions and state government programs.
According to Professor Merritt, the problem is that the mandatory curve in law school is such that not all students can keep their conditional scholarships. But Professor Merritt presents no evidence that conditional scholarships retention rates are any higher for undergraduate or government programs than for law schools. She infers nefarious motives on the part of law schools based only on the fact that law schools require students to compete for scholarship funds that are in limited supply.
Perhaps Professor Merritt believes that competition for scarce and valuable resources is inherently immoral. She does not explain why this is so or whether these views apply outside the context of law school scholarships. If only one out of ten associates hired at a law firm will make partner and earn $1 million per year, is it inherently immoral to ask associates to work hard and compete for the opportunity? If only one actor will be selected for a part, is it immoral to ask more than one actor to try out? Is any competition for promotions, clients, or recognition immoral? If so, we are living in a wicked, wicked world.
Perhaps Professor Merritt believes it is inherently immoral to limit “A” grades to students whose academic performance is superior to most of their peers, since an “A” is simply a data point and can be replicated and distributed to everyone at zero marginal cost. But liberally handing out “A” grades is costly for students and employers. Labor economics studies suggest that grade inflation is associated with reduced effort by students and reduced learning. Educators are not doing students or employers favors if they allow high grades to become a birthright rather than a marker of distinction that must be earned through hard work and exceptional performance.
Statistician Valen Johnson and others have argued that many perverse incentives in undergraduate education could be ameliorated if mandatory grading curves were imposed across majors and grade inflation and grade shopping were stamped out. If certain undergraduate majors have succumbed to the pressure to inflate grades in order to keep student-customers happy, that is quite troubling. Employers will likely distrust grades from such programs, question how much students have learned, and harbor suspicions about the work ethic of students who would opt into programs known for awarding easy “A’s” for minimal effort. Programs that have resisted the pressure to inflate grades and maintained more rigorous academic standards are more likely to retain the confidence of employers and to teach students knowledge and skills that are valued in the labor market. Indeed, grades are notoriously lower in STEM fields than in the humanities, even though STEM majors spend more time studying and have higher standardized test scores.
Professor Merritt suggests that law students do not require any incentives to work harder, since they are all already studying at full capacity. Some students presumably are, but there are many law students who can and should focus more on their studies. A roll call in most classes will reveal students whose attendance is well below 100 percent—so much so that the ABA now requires law schools to enforce minimum class attendance policies. When students do attend class, a visit to the back of a classroom and a glance at computer screens will reveal some students who are not giving their undivided attention. Cold calling will reveal students who have not done the required reading—although they do appear to be well informed about the latest sports and celebrity news. Some students have family or employment obligations that understandably limit the amount of time they can devote to their studies. But in the evening, while some student who are less constrained are studying, a stroll past the local bar will reveal others who are spending their time on less academic pursuits.
Shortly after graduation, some students who did not show up for class enough, did not pay attention enough, did not prepare for class enough, did not review after class enough, and did not seek out their professors when they were confused will find that they have not passed the bar exam and will not be permitted to practice law until they learn how to work hard and study. Others will find, rather less dramatically, that what they did not work hard enough to learn in law school could have made them more valuable to their employers.
Law schools can observe LSAT scores and undergraduate GPAs, but they cannot directly observe students’ work ethic and drive to succeed. Just as law students want to attend the best law schools, law schools want to educate the best students who have the motivation to become leaders in law, business, and government.
Given the goal of attracting and retaining the best students, rewarding motivation and ability seems like a reasonable policy. Anecdotes notwithstanding, the evidence suggests that most college and law students understand the terms of conditional scholarships well.
May 04, 2015
Many critics have attacked law schools for offering merit scholarships that can only be retained if students meet minimum GPA requirements. Jeremy Telman has a fascinating new post analyzing these scholarships in light of common practices in higher education and the peer-reviewed social science literature. It’s a powerful counterpoint to a previously unanswered critique of law school ethics.
Professor Telman notes that similar conditional scholarships are widely used by undergraduate institutions, and even some state government programs. Undergraduates behave as if they understand how conditional scholarships work, which suggests that most law students, who are older, wiser, and more sophisticated, probably understand the terms of these agreements as well.
Moreover, minimum GPA requirements can motivate students to study harder, pay closer attention, and learn more. This seems particularly likely in the context of the first year of law school where mandatory grading curves and required curriculums remove the opportunity to shop for “easy A’s”. (Professor Telman does, however, express concern about inadequate performance feedback to law students until the final exams at the end of their first semester).
Professor Telman notes that law schools may struggle to predict at the time of admission which students will be the most successful. Conditional scholarships help institutions gather additional information about students’ abilities and work ethic and ensure that limited merit scholarship resources go to the students who are most deserving. Students who are deemed undeserving and lose their scholarships retain the option of transferring to another institution for their remaining years of law school.
Professor Telman doesn't object to additional disclosure about the percent of students retaining their scholarships, but he doubts it would have made much of a difference in prospective law students' matriculation decisions.
April 30, 2015
An interesting, and not implausible, list. The JD comes in 6th, though most of the other options are unlikely to be pursued by an undergraduate humanities major--one reason, among others, why we have probably hit bottom in terms of the applicant pool and will probably see a slight uptick in the next couple of years.
April 29, 2015
A number of critics have argued against extrapolation from Professor Merritt’s study of the Ohio legal market to the national legal market. In her response, Professor Merritt makes some good points, and also several key points with which I disagree.
Professor Merritt suggests that an important contribution of her study is providing up-to-date information about national legal employment through the prism of Ohio. However, there is no shortage of up-to-date data that can provide a more accurate picture of national trends than a study specifically focused on Ohio.* The primary value of Professor Merritt’s study is as an isolated snapshot of a single cohort in Ohio at a particular point in time. Without additional information, it is hard to know how much, if at all, Professor Merritt’s findings should be generalized to other legal markets or other time periods.
There is no reason to believe that the single Ohio cohort tracked by Professor Merritt will better predict outcomes for those currently enrolling in law school than a national cohort. The single Ohio cohort will likely be less predictive than a long-term national average across multiple cohorts. Indeed, as Professor Merritt acknowledges, her study is not a study of going to law school in Ohio because of selection issues from law graduates leaving for larger markets, coming to Ohio from other markets, and from non-bar passage. **
Year-to-year changes in employment, earnings, and economic growth can vary widely from state to state. Absent evidence of a history of correlated economic activity, a single state should not be used as a proxy for the U.S. as a whole or for other states.
There is no reason to believe that the trajectory of Ohio’s legal market from year to year will closely track national trends, particularly when the national legal market is heavily concentrated elsewhere. Washington D.C. and the top 5 states by size of legal market*** collectively account for more than half of the national legal market.
If Professor Merritt wishes to use Ohio as a proxy for the rest of the U.S., then she should supply evidence that Ohio tracks national trends, and she should compare Ohio to Ohio at different points in time and Ohio to the U.S. at the same point in time.
Second, Professor Merritt suggests that focusing on Ohio is just as reasonable as focusing on New York or California. New York and California collectively constitute 28 percent of the national legal market.*** Ohio constitutes 2.5 percent of the national legal market. Moreover, the New York legal market is unusually large relative to the New York economy, while Ohio has a legal market that is small relative to its economy.
Third, Professor Merritt suggests that Ohio can be made nationally representative by deflating salaries elsewhere by cost of living differences. Cost of living differences are not the reason corporations—who can send legal work anywhere— pay a premium for lawyers in the major legal markets such as New York, D.C., Los Angeles, Boston and Houston. Rather, corporate clients believe that differences in quality of work justify higher billing rates for important matters. New York, D.C. and other high-paying markets are importers of top legal talent from across the country.
Differences in costs of living are not random, but rather reflect real differences in quality. Cost of living indexes often focus on quantitative rather than qualitative factors. For example, a restaurant meal in Manhattan may cost more than a restaurant meal in Buffalo, but the quality of the experience in the restaurant in Manhattan will on average be higher because the high prices restaurants in Manhattan can charge will attract the most talented restaurateurs. Similarly, there may be differences in the quality of healthcare, legal services, education, policing, parks and recreation, environmental safety, transit, housing and other factors. Money attracts talent. Some amenities or opportunities may only be available in particular locations, and people are willing to pay for proximity to consumption, employment, and social opportunities.
Many costs are not local, but rather national. These include automobiles, items ordered online, higher education at major universities, and investments (stocks, bonds, etc.). For law school graduates—who will typically be able to earn far more than they consume in a given year—it is financially better to work where both income and costs are proportionately higher because this will maximize the dollar value of savings. Law graduates can always retire to a lower-cost location later in life if they wish.
One quantitative measure for differences in quality of life is differences in life expectancy.**** High cost, high income, high infrastructure states like New York, Connecticut, and Massachusetts generally rank well on this measure, while lower cost, lower income states rank less well. This pattern can also be seen internationally and individually—higher income and higher life expectancy are correlated.*****
There will indeed be some lucky individuals who find low-cost locales both more attractive and less expensive, and some unlucky individuals who find high cost locales unworthy of the price. Costs of living reflect the aggregation by the market of many individual preferences, not any particular person’s idiosyncratic views. Nevertheless, local prices can contain important information about quality of life that we should not assume away.
* There are numerous sources of up-to-date (2013 or even 2014) national information, including data from:
- the U.S. Department of Labor Bureau of Labor Statistics (BLS)
- the U.S. Census Bureau’s
- American Community Survey (ACS),
- Current Population Survey (CPS), and
- Survey of Income and Program Participation (SIPP),
- the National Association for Law Placement (NALP), and
- from the American Bar Association (ABA)
NALP and ABA data are for the most recent graduating class shortly after graduation. SIPP earnings data includes earnings as recently as 2013, but only through the class of 2008. ACS and CPS have young lawyers and young professional degree holders, but cannot specifically identify young law degree holders. The Department of Education also has information on student loan default rates for recent cohorts. Default rats remain much lower for former law students than for most other borrowers.
Another valuable source of information is After the JD III. Professor Merritt notes that response rates for higher income individuals may be higher in After the JD, but the After the JD researchers, like the U.S. Census, weight their sample to take into account differences in response rates.
**The selection bias issues may be more severe than Merritt has acknowledged. Looking at Ohio State’s 509 report for 2011, there were 24 students who took the NY bar vs. 136 who took the Ohio bar—a substantial percentage of the class taking a bar in a non-adjacent state. The New York bar takers had much higher bar passage rates (11% above the state average for N.Y. vs. 1.3% above the state average for Ohio), which is consistent with positive selection out of state. In any given year, roughly 25 to 50 percent of Ohio State law school graduates who are employed 9 or 10 months after graduation are employed outside of Ohio. For Case Western graduates, employment seems to be even less Ohio-centered than Ohio State.
*** Size of the legal market calculated using ACS data, multiplying number of lawyers by average total personal income per lawyer to get aggregate pay to all lawyers. In other words, the measure is a dollar count, not a body count.
**** It is probably preferable to consider life expectancy within race (life expectancy varies by race, and racial demographics vary by geography).
***** After controlling for GDP per capita, societies with less income dispersion tend to have higher life expectancy. Another issue is selection effects vs. causation. For example, those with higher life expectancy to begin with may choose to pursue additional education and therefore have the opportunity to live in high cost, high income states.
April 27, 2015
New York Times relies on unrepresentative anecdotes and flawed study to provide slanted coverage of legal education (Michael Simkovic)
Just when you thought The New York Times was rounding the corner and starting to report responsibly about legal education based on hard data and serious labor economics studies, their reporting reverts to the unfortunate form it has taken for much of the last 5 years*—relying on unrepresentative anecdotes and citing fundamentally flawed working papers to paint legal education in a negative light.
Responsible press coverage would have put law graduate outcomes in context by noting that:
(1) law graduates continue to do better in terms of employment (both overall and full time) and earnings than similar bachelor’s degree holders, even in an economy that has generally been challenging for young workers
(2) law students, even from some of the lowest ranked and most widely criticized law schools, continue to have much lower student loan default rates than the national average across institutions according to standardized measurements reported by the Department of Education
(3) law graduate earnings and employment rates typically increase as they gain experience
(4) Data from After the JD shows that law graduates continue to pay down their student loans and approximately half of graduates from the class of 2001 paid them off completely within 12 years of graduation
Instead, The New York Times compares law graduate outcomes today to law graduate outcomes when the economy was booming. But not all law graduates. The Times focuses on law graduates who have been unusually unsuccessful in the job market or have unusually large amounts of debt. For example, The New York Times focused on a Columbia law school graduate working as an LSAT tutor** as if that were a typical outcome for graduates of elite law schools. But according to the National Law Journal, two-thirds of recent Columbia graduates were employed at NLJ 250 law firms (very high paying, very attractive jobs),*** and the overwhelming majority of recent Columbia graduates appear to work in attractive positions. (Columbia outcomes are much better than most, but the negative outcomes discussed in The New York Times are substantially below average for law graduates as a whole).
In Timing Law School, Frank McIntyre’s and I analyze long term outcomes for those who graduated into previous recessions, using nationally representative data and well-established econometric methods. Our results suggest that law graduates continue to derive substantial benefits from their law degrees even when graduating into a recession. The recent recession does not appear to be an exception. (See also here and here). This analysis is not mentioned in the recent The New York Times article, even though it was cited in The New York Times less than a month ago (and alluded to in The Washington Post even more recently).
The implication of The New York Times’ story “Burdened With Debt, Law School Graduates Struggle in Job Market” is that there is some law specific problem, when the reality is that the recession continues to negatively affect all young and inexperienced workers and law graduates continue to do better than most. Law school improves young workers’ chances of finding attractive employment opportunities and reduces the risk of defaulting on debt. The benefits of law school exceed the costs for the overwhelming majority of law school graduates.
The New York Times relies heavily on a deeply flawed working paper by Professor Deborah Merritt of Ohio State. Problems with this study were already explained by Professor Brian Galle:
“My problem is that instead DJM wants to offer us a dynamic analysis, comparing 2014 to 2011, and arguing that the resulting differential tells us that there has been a "structural shift" in the market for lawyers. It might be that the data exist somewhere to conduct that kind of analysis, but if so they aren't in the paper. Nearly all the analysis in the paper is built on the tend line between DJM's 2014 Ohio results and national-average survey results from NALP.
Let me say that again. Almost everything DJM says is built on a mathematical comparison between two different pools whose data were constructed using different methods. I would not blame you if now stopped reading."
In other words, it is difficult to tell whether any differences identified by Professor Merritt are:
(1) Due to differences between Ohio and the U.S. as a whole
(2) Due to differences in methodology between Merritt, NALP, and After the JD
(3) Actually due to differences between 2011 and 2014 for the same group
After Professor Galle’s devastating critique, journalists should have been extremely skeptical of Merritt’s methodology and her conclusions. Professor Merritt’s response to Galle’s critique, in the comments below his post, is not reassuring:
“Bottom line for me is that the comparison in law firm employment (62.1% for the Class of 2000 three years after graduation, 40.5% for the lawyers in my population) seems too stark to stem solely from different populations or different methods—particularly because other data show a more modest decline in law firm employment over time. But this is definitely an area in which we need much, much more research.”
Judging from this response and the quotes in The New York Times, Merritt appears to be doubling down on her inapposite comparisons rather than checking how much of her conclusions are due to potentially fatal methodological problems. What Professor Merritt should have done is replicate her 2014 Ohio-only methodology in 2000/2001 or 2010/2011, compared the results for Ohio only at different points in time, and limited her claims to an analysis of the Ohio legal employment market.
There are additional problems with Professor Merritt’s study (or at least the March 11 version that I reviewed).****
- Ohio is not a representative legal employment market, but rather a relatively low paying one where lawyers comprise a relatively small proportion of the workforce.
- A disproportionate share of the 8 or 9 law schools in Ohio (9 if you include Northern Kentucky) are low ranked or unranked, and this presumably is reflected in their employment outcomes.
- Merritt’s sample is subject to selection bias because of movement of the most capable law graduates out of Ohio and into higher paying legal markets. Ohio law graduates who do not take the Ohio bar after obtaining jobs in Chicago, New York, Washington D.C., or other leading markets will not show up in Merritt’s sample.
- Whereas Merritt concludes that law graduate outcomes have not improved, the data may simply reflect the fact that Ohio is a less robust employment market than the U.S. as a whole.
- Merritt’s analysis of employment categories does not take into account increases in earnings within employment categories. After the JD and follow-ups suggests that these within-category gains are substantial, as does overall increases in earnings from Census data.
- Merritt makes a biased assumption that anyone she could not reach is unemployed instead of gathering additional information about non-respondents and weighting the results to take into account response bias. Law schools may have been more aggressive in tracking down non-respondents than Professor Merritt was.
For the benefit of those who are curious, I am making my full 8 page critique of Professor Merritt's working paper available here, but please keep in mind that it was written in mid March and Professor Merritt may have addressed some of these issues in more recent versions of her paper. If that is the case, I trust that she’ll highlight any changes or improvements in a blog post response.
* A few weeks ago I asked a research assistant (a third year law student) to search for stories in The New York Times and Wall Street Journal about law school. Depending on whether the story would have made my research assistant more likely or less likely to want to go to law school when he was considering it or would have had no effect, he coded the stories as positive, negative, or neutral. According to my research assistant, The New York Times reported 7 negative stories to 1 positive story in 2011 and 5 negative stories to 1 positive story in 2012. In 2013, 2014, and 2015, The New York Times coverage was relatively balanced. In aggregate over the five-year period The New York Times reported about 2 negative stories for every 1 positive story. The Wall Street Journal’s coverage was even more slanted—about 3.75 negative stories for every positive story—and remained heavily biased toward negative stories throughout the five-year period.
** Professor Stephen Diamond notes the LSAT tutor’s relatively high hourly wage, more lucrative opportunities the tutor claims he turned down, and how the tutor describes his own work ethic.
*** For the class of 2010, the figure at Columbia was roughly 52 percent 9 months after graduation, but activity in the lateral recruitment market suggests things may be looking up.
**** The comments that follow summarize a lengthy (8 page) critique I sent to Professor Merritt privately in mid March after reviewing the March 11 draft of her paper. I have not had a chance to review Professor Merritt’s latest draft, and Professor Merritt may have responded to some of these issues in a revision.
April 27, 2015 in Advice for Academic Job Seekers, Guest Blogger: Michael Simkovic, Law in Cyberspace, Legal Profession, Of Academic Interest, Professional Advice, Science, Student Advice, Web/Tech, Weblogs | Permalink
April 20, 2015
April 11, 2015
Deborah Merritt and Kyle McEntee conflated “response rates” with nonresponse bias and response bias. After I brought this error to light, Professor Merritt explained that she and Mr. McEntee were not confused about basic statistical terminology, but rather were being intentionally vague in their critique to be more polite* to the law schools.
Professor Merritt also changed the topic of conversation from Georgetown’s employment statistics—which had been mentioned in The New York Times and discussed by me, Professor Merritt, and Kyle McEntee—to the employment statistics of the institution where I teach.**
What Professor Merritt meant to say is that law schools have not been properly weighting their data to take into account nonresponse bias. This is an interesting critique. However, proper weights and adjustments to data should take into account all forms of nonresponse bias and response bias, not just the issue of over-representation of large law firms in NALP salary data raised by Professor Merritt.
While such over-representation would have an effect on the mean, it is unclear how much impact, if any, it would have on reported medians—the measure of central tendency used by The New York Times and critiqued by Mr. McEntee.
Other biases such as systematic under-reporting of incomes by highly educated individuals,*** under-reporting of bonuses and outside income, and the like should be taken into account.**** To the extent that these biases cut in opposite directions, they can offset each other. It’s possible that in aggregate the data are unbiased, or that the bias is much smaller than examination of a single bias would suggest.
Moreover, focusing on first year salaries as indicative of the value of a lifetime investment is itself a bias. As The Economic Value of a Law Degree, showed, incomes tend to rise rapidly among law graduates. They do not appreciably decrease, either, until the fourth decade of employment.
If Professor Merritt’s view is that differences between NALP, ABA, and U.S. Census Bureau data collection and reporting conventions make law school-collected data more difficult to compare to other data sources and make law school data less useful, then I am glad to see Professor Merritt coming around to a point I have made repeatedly.
I have gone further and suggested that perhaps the Census Bureau and other government agencies should be collecting all data for graduate degree programs to ensure the accuracy and comparability of data across programs and avoid wasting resources on duplicative data collection efforts.
This could also help avoid an undue amount of focus on short-term outcomes, which can be misleading in light of the rapid growth of law graduate earnings as they gain experience. The inappropriate focus on the short term can be misleading if students are not aware of the growth trajectory and how it compares to the growth trajectory of likely earnings without a law degree.
** This tactic, bringing up the employment statistics of the institution where those whom she disagrees with teach, is something of a habit for Professor Merritt. See her response Anders Walker at St. Louis).
*** Law graduates outside of the big firms are highly educated, high-income individuals compared to most of the rest of individuals in the United States. That is the benchmark used by researchers when they identified the reporting biases in census data that lead to under-reporting of incomes.
**** The risk of under-reporting income in law may be particularly high because of opportunities for tax evasion for those who run small businesses or have income outside of their salary.
UPDATE (4/14/2015): I just confirmed with NALP that their starting salary data does not include end of year bonuses.
April 10, 2015
Did law schools behave unethically by providing employment and earnings information without simultaneously reporting survey response rates? Or is this standard practice?
The answer is that not reporting response rates is standard practice in communication with most audiences. For most users of employment and earnings data, response rates are a technical detail that is not relevant or interesting. The U.S. Government and other data providers routinely report earnings and employment figures separate from survey response rates.*
Sometimes, too much information can be distracting.** It’s often best to keep communication simple and focus only on the most important details.
Nonresponse is not the same thing as nonresponse bias. Law school critics do not seem to understand this distinction. A problem only arises if the individuals who respond are systematically different from those who do not respond along the dimensions being measured. Weighting and imputation can often alleviate these problems. The critics’ claims about the existence, direction, and magnitude of biases in the survey data are unsubstantiated.
High non-response rates to questions about income are not a sign of something amiss, but rather are normal and expected. The U.S. Census Bureau routinely finds that questions about income have lower response rates (higher allocation rates) than other questions.
Law school critics claim that law school graduates who do not respond to questions about income are likely to have lower incomes than those who do respond. This claim is not consistent with the evidence. To the contrary, high-income individuals often value privacy and are reluctant to share details about their finances.***
Another potential problem is “response bias”, in which individuals respond to survey questions in a way that is systematically different from the underlying value being measured. For example, some individuals may under report or over-report their incomes.
The best way to determine whether or not we have nonresponse bias or response bias problems is to gather additional information about non-responders and responders.
Researchers have compared income reported to Census surveys with administrative earnings data from the Social Security Administration and Internal Revenue Service. They find that highly educated, high-income individuals systematically under-report their incomes, while less educated, lower income individuals over-report. (Assuming the administrative data is more accurate than the survey data).
Part of the problem seems to be that bonuses are underreported, and bonuses can be substantial. Another problem seems to be that high-income workers sometimes report their take-home pay (after tax withholding and deductions for benefits) rather than their gross pay.
Other studies have also found that response bias and nonresponse bias lead to underestimation of earnings and employment figures.
In other words, there may indeed be biases in law school earnings data, but if there is, it is likely in the opposite direction of the one the law school critics have claimed.
Of course, the presence of such biases in law school data would not necessarily be a problem if the same biases exist in data on employment and earnings for alternatives to law school. After all, earnings and employment data is only useful when compared to a likely alternative to law school.
As with gross employment data, the critics are yet again claiming that an uncontroversial and nearly universal data reporting practice, regularly used by the United States Government, is somehow scandalous when done by law schools.
The only thing the law school critics have demonstrated is their unfamiliarity with basic statistical concepts that are central to their views.
* Reporting earnings and employment estimates without response rates in communication intended for a general audience—and even some fairly technically sophisticated audiences—is standard practice for U.S. government agencies such as the U.S. Census Bureau and the U.S. Department of Labor, Bureau of Labor Statistics. A few examples below:
- Earnings and unemployment by education level
- Unemployment rates
- Employment population ratio
- Tabular summaries from
** Information on response rates is available for researchers working with microdata to develop their own estimates, and for those who want to scour the technical and methodological documentation. But response rates aren’t of much interest to most audiences.
*** After the JD researchers noted that young law graduates working in large urban markets—presumably a relatively high-income group—were particularly reluctant to respond to the survey. From After the JD III:
“Responses . . . varied by urban and rural or regional status, law school rank, and practice setting. By Wave 2, in the adjusted sample, the significant difference between respondents and nonrespondents continued to be by geographic areas, meaning those from larger legal markets (i.e. New York City) were less likely to respond to the survey. By Wave 3, now over 12 years out into practice, nonrespondents and respondents did not seem to differ significantly in these selected characteristics.”
In the first wave of the study, non-respondents were also more likely to be male and black. All in all, it may be hard to say what the overall direction of any nonresponse bias might be with respect to incomes. A fairly reasonable assumption might be that the responders and non-responders are reasonably close with respect to income, at least within job categories.