April 22, 2015
April 21, 2015
At the faculty lounge, Professor Bernard Burk of the University of North Carolina echoes questions raised earlier by Professor Merritt of Ohio State about whether it is unethical or misleading for law schools to report employment using the international standard definition of employment. I have discussed these issues extensively before.*
Employment statistics are primarily useful for purposes of comparing alternatives. Comparison requires standard measurements. Standardization is efficient because it reduces the number of definitions that must be learned to use data. The standard definition of employment is meaningful and useful because, notwithstanding preferences for particular kinds of work, a job of some kind is generally preferable to no job at all. This does not mean that employment is the only measurement one should consider, but rather that it is a useful measurement.
Because international standards exist, it is not necessary to explain to a college graduate what a centimeter means when describing the length of an object. Similarly it is not necessary to explain to college graduates contemplating law school what employment means when using the international standard definition of employment.**
College educated individuals who are unfamiliar with standard terminology can easily look up or inquire about the relevant definitions, and once they have learned, can begin to understand a world of data. The standard definitions of employment and unemployment can be quickly discovered through intuitive internet searches. (see searches for unemployment and employment definitions) These definitions are neither obscure nor technically challenging.
In addition, many law schools disclose bar passage rates that are lower than their employment rates. It seems doubtful that many college educated adults contemplating law school—in particular, the subset basing their decisions on outcome data such as employment and bar passage rates—would assume that every law graduate who is employed shortly after graduation is working as a lawyer when many of those graduates cannot legally practice law.
Critiquing international standardized measurements as inherently immoral is not without precedent.
According to Martin Gardner, during the 1800s, a U.S. group attacked the French metric system as atheistic and immoral.
“The president of the Ohio group, a civil engineer who prided himself on having an arm exactly one cubit in length, had this to say . . . : "We believe our work to be of God; we are actuated by no selfish or mercenary motive. We depreciate personal antagonisms of every kind, but we proclaim a ceaseless antagonism to that great evil, the French Metric System. . .The jests of the ignorant and the ridicule of the prejudiced, fall harmless upon us and deserve no notice. . . It is the Battle of the Standards. May our banner be ever upheld in the cause of Truth, Freedom, and Universal Brotherhood, founded upon a just weight and a just measure, which alone are acceptable to the Lord." “
“A later issue printed the words and music of a song, the fourth verse of which ran:
Then down with every "metric" scheme
Taught by the foreign school,
We'll worship still our Father's God!
And keep our Father's "rule"!
A perfect inch, a perfect pint,
The Anglo's honest pound,
Shall hold their place upon the earth,
Till time's last trump shall sound!”
Many thoughtful people believe the U.S.’s non-standard approach to measurement undermines U.S. competitiveness in science, math, engineering, and industry. Time is wasted learning and converting to and from a redundant and inefficient measurement system. This entails opportunity cost and leads to unnecessary and avoidable errors.
Law schools, the American Bar Association, and the National Association for Law Placement would be better served by using standard definitions for labor market measurements when standard definitions are available and widely in use elsewhere, or at least labeling non-standard definitions with names that will not be readily confused with standard definitions.
The ABA currently requires law schools to describe individuals as “Unemployed” who under standard definitions would be defined as either “Not in Labor Force” or “Unemployed.” In other words, “unemployment” as reported under ABA definitions will be higher than unemployment under the standard and most widely used government definition. A number of people have been confused by this, incorrectly claiming that “unemployment” for law graduates is unusually high in comparison to everyone else. In fact, under consistent measurements, the fraction of recent law graduates who are employed is higher than the overall proportion of the population that is employed. (Law graduates also do relatively well on the percent employed full-time).
I agree with Professor Burk that additional information about occupational categories could be useful to some users of data. However, I do not agree that presenting standard summary statistics is inherently misleading or unethical, particularly for the sophisticated audience using the data —college educated, internet savvy adults.
April 20, 2015
April 18, 2015
April 16, 2015
April 15, 2015
April 13, 2015
Stephen Diamond (Santa Clara) has the details. He won't be missed, it's fair to say.
April 11, 2015
Deborah Merritt and Kyle McEntee conflated “response rates” with nonresponse bias and response bias. After I brought this error to light, Professor Merritt explained that she and Mr. McEntee were not confused about basic statistical terminology, but rather were being intentionally vague in their critique to be more polite* to the law schools.
Professor Merritt also changed the topic of conversation from Georgetown’s employment statistics—which had been mentioned in The New York Times and discussed by me, Professor Merritt, and Kyle McEntee—to the employment statistics of the institution where I teach.**
What Professor Merritt meant to say is that law schools have not been properly weighting their data to take into account nonresponse bias. This is an interesting critique. However, proper weights and adjustments to data should take into account all forms of nonresponse bias and response bias, not just the issue of over-representation of large law firms in NALP salary data raised by Professor Merritt.
While such over-representation would have an effect on the mean, it is unclear how much impact, if any, it would have on reported medians—the measure of central tendency used by The New York Times and critiqued by Mr. McEntee.
Other biases such as systematic under-reporting of incomes by highly educated individuals,*** under-reporting of bonuses and outside income, and the like should be taken into account.**** To the extent that these biases cut in opposite directions, they can offset each other. It’s possible that in aggregate the data are unbiased, or that the bias is much smaller than examination of a single bias would suggest.
Moreover, focusing on first year salaries as indicative of the value of a lifetime investment is itself a bias. As The Economic Value of a Law Degree, showed, incomes tend to rise rapidly among law graduates. They do not appreciably decrease, either, until the fourth decade of employment.
If Professor Merritt’s view is that differences between NALP, ABA, and U.S. Census Bureau data collection and reporting conventions make law school-collected data more difficult to compare to other data sources and make law school data less useful, then I am glad to see Professor Merritt coming around to a point I have made repeatedly.
I have gone further and suggested that perhaps the Census Bureau and other government agencies should be collecting all data for graduate degree programs to ensure the accuracy and comparability of data across programs and avoid wasting resources on duplicative data collection efforts.
This could also help avoid an undue amount of focus on short-term outcomes, which can be misleading in light of the rapid growth of law graduate earnings as they gain experience. The inappropriate focus on the short term can be misleading if students are not aware of the growth trajectory and how it compares to the growth trajectory of likely earnings without a law degree.
** This tactic, bringing up the employment statistics of the institution where those whom she disagrees with teach, is something of a habit for Professor Merritt. See her response Anders Walker at St. Louis).
*** Law graduates outside of the big firms are highly educated, high-income individuals compared to most of the rest of individuals in the United States. That is the benchmark used by researchers when they identified the reporting biases in census data that lead to under-reporting of incomes.
**** The risk of under-reporting income in law may be particularly high because of opportunities for tax evasion for those who run small businesses or have income outside of their salary.
UPDATE (4/14/2015): I just confirmed with NALP that their starting salary data does not include end of year bonuses.