April 10, 2015
Did law schools behave unethically by providing employment and earnings information without simultaneously reporting survey response rates? Or is this standard practice?
The answer is that not reporting response rates is standard practice in communication with most audiences. For most users of employment and earnings data, response rates are a technical detail that is not relevant or interesting. The U.S. Government and other data providers routinely report earnings and employment figures separate from survey response rates.*
Sometimes, too much information can be distracting.** It’s often best to keep communication simple and focus only on the most important details.
Nonresponse is not the same thing as nonresponse bias. Law school critics do not seem to understand this distinction. A problem only arises if the individuals who respond are systematically different from those who do not respond along the dimensions being measured. Weighting and imputation can often alleviate these problems. The critics’ claims about the existence, direction, and magnitude of biases in the survey data are unsubstantiated.
High non-response rates to questions about income are not a sign of something amiss, but rather are normal and expected. The U.S. Census Bureau routinely finds that questions about income have lower response rates (higher allocation rates) than other questions.
Law school critics claim that law school graduates who do not respond to questions about income are likely to have lower incomes than those who do respond. This claim is not consistent with the evidence. To the contrary, high-income individuals often value privacy and are reluctant to share details about their finances.***
Another potential problem is “response bias”, in which individuals respond to survey questions in a way that is systematically different from the underlying value being measured. For example, some individuals may under report or over-report their incomes.
The best way to determine whether or not we have nonresponse bias or response bias problems is to gather additional information about non-responders and responders.
Researchers have compared income reported to Census surveys with administrative earnings data from the Social Security Administration and Internal Revenue Service. They find that highly educated, high-income individuals systematically under-report their incomes, while less educated, lower income individuals over-report. (Assuming the administrative data is more accurate than the survey data).
Part of the problem seems to be that bonuses are underreported, and bonuses can be substantial. Another problem seems to be that high-income workers sometimes report their take-home pay (after tax withholding and deductions for benefits) rather than their gross pay.
Other studies have also found that response bias and nonresponse bias lead to underestimation of earnings and employment figures.
In other words, there may indeed be biases in law school earnings data, but if there is, it is likely in the opposite direction of the one the law school critics have claimed.
Of course, the presence of such biases in law school data would not necessarily be a problem if the same biases exist in data on employment and earnings for alternatives to law school. After all, earnings and employment data is only useful when compared to a likely alternative to law school.
As with gross employment data, the critics are yet again claiming that an uncontroversial and nearly universal data reporting practice, regularly used by the United States Government, is somehow scandalous when done by law schools.
The only thing the law school critics have demonstrated is their unfamiliarity with basic statistical concepts that are central to their views.
* Reporting earnings and employment estimates without response rates in communication intended for a general audience—and even some fairly technically sophisticated audiences—is standard practice for U.S. government agencies such as the U.S. Census Bureau and the U.S. Department of Labor, Bureau of Labor Statistics. A few examples below:
- Earnings and unemployment by education level
- Unemployment rates
- Employment population ratio
- Tabular summaries from
** Information on response rates is available for researchers working with microdata to develop their own estimates, and for those who want to scour the technical and methodological documentation. But response rates aren’t of much interest to most audiences.
*** After the JD researchers noted that young law graduates working in large urban markets—presumably a relatively high-income group—were particularly reluctant to respond to the survey. From After the JD III:
“Responses . . . varied by urban and rural or regional status, law school rank, and practice setting. By Wave 2, in the adjusted sample, the significant difference between respondents and nonrespondents continued to be by geographic areas, meaning those from larger legal markets (i.e. New York City) were less likely to respond to the survey. By Wave 3, now over 12 years out into practice, nonrespondents and respondents did not seem to differ significantly in these selected characteristics.”
In the first wave of the study, non-respondents were also more likely to be male and black. All in all, it may be hard to say what the overall direction of any nonresponse bias might be with respect to incomes. A fairly reasonable assumption might be that the responders and non-responders are reasonably close with respect to income, at least within job categories.
January 30, 2015
January 16, 2015
January 05, 2015
Cardozo's Susan Crawford, Miami's Mary Anne Franks and Columbia's Tim Wu named 2014 "Heroes of the Internet"...
December 10, 2014
September 24, 2014
May 23, 2014
April 16, 2014
...that things aren't as awful as the various charlatans and other law-school haters claim, and, predictably (given the social psychology), the charlatans and haters go crazy. I won't link to the hysterical reactions (they are easy enough to find with Google), but they boil down to one complaint: Chemerinsky & Menkel-Meadow cited NALP data without treating it as bogus (e.g., that JD Advantage jobs are really jobs [actually many of them are, but never mind]). That's true, they linked to the NALP data, but they didn't spend the rest of their piece debunking that data based on speculation, skepticism, and occasionally other actual evidence. This has certainly been a standing problem in the debate about American legal education, as when serious data analysis showed that legal education was a sound economic investment for the vast majority of students, and critics refused to believe that was true, though without any contrary evidence or analysis. So we can all agree that we should be more careful about how we present data and its import.
That being said, my main disagreement with Chemerinsky & Menkel-Meadow is about the necessity of three years of legal education, as I've said before: two years could work, and work very well for many students. In reality, the biggest obstacle to reducing costs in legal education, however, is unnoted in their op-ed: it remains the lax tenure standards and the unwillingness of universities to terminate tenured faculty for cause, i.e., when they manifestly do not do their job.
Imagine, for example, a law school that pays a six figure salary (closing in on 200K) to someone with almost no legal experience and an M.A. in literature who teaches the same couple of substantive courses year in and year out, courses in which he has no experience, whose teaching evaluations are consistently below average, who hasn't written any serious legal scholarship in years, who is regarded as a joke by his colleagues at his own school and in the academy at large, and who mostly spends his time insulting, defaming, and blackmailing colleagues who do their jobs. It endangers the institution of tenure when universities do not initiate proceedings to terminate malevolent charlatans like this. Many law schools, as we've noted before, are offering financial inducements to "buy out" senior faculty, most of whom are not charlatans. Real cost reduction, however, will require universities to move against the charlatans and the de facto retired in their midst, even those who have tried to insulate themselves from termination for cause by setting up frivolous retaliation claims.
UPDATE: More thoughts on reforming legal education from Michael Madison (Pitt).
March 26, 2014
In typically irresponsible fashion, ATL yesterday posted factually inaccurate rumors about Denver (which they are slowly correcting). Here is what a tenured colleague at Denver wrote to me:
The truth is that we are reducing our tenure and tenure-track faculty by 10 over multiple years. This is consistent with a long-term plan to shrink the size of the school that began in 2007, prior to the economic downturn. At that point we had 380 students. Our ultimate goal was and is approximately 250 students. The school needs *at most* one person to retire or take a buyout this year to meet our budget for 2014-2015. In subsequent years the faculty who will be offered the option of buyouts will be exclusively tenured faculty who have held their positions for a minimum number of years. The buyouts will NOT include tenure-track faculty who are not yet tenured. That is, the ATL story is simply wrong when it says that untenured tenure-track faculty are being asked to leave. None has been asked to do so. In fact, Denver Law has recommended 4 tenure-track faculty for tenure this year. While tenure is not official until the summer, it is common knowledge that the Dean has recently assured those four faculty members that the planned buyouts will not affect their tenure process. Finally, ATL's unattributed claim that the Denver Law faculty is "quite displeased" with direction of the school is simply false. Of course there are outliers in every institution, but the overall faculty climate is collegial and the vast majority of faculty are pleased with Dean Katz's leadership during a difficult time for all law schools. Of course, it is never ideal for ATL to report facts that are patently false, particularly with respect to untenured faculty, and one would hope that they care enough about their credibility to print a correction.
I've heard the same about Dean Katz from other faculty at Denver as well. (I hope Blog Emperor Caron will learn a lesson from this incident, namely, not to reprint nonsense from ATL without independent verification.)