Offsetting Biases (Michael Simkovic)

Deborah Merritt and Kyle McEntee conflated “response rates” with nonresponse bias and response bias.  After I brought this error to light, Professor Merritt explained that she and Mr. McEntee were not confused about basic statistical terminology, but rather were being intentionally vague in their critique to be more polite* to the law schools.

Professor Merritt also changed the topic of conversation from Georgetown’s employment statistics—which had been mentioned in The New York Times and discussed by me, Professor Merritt, and Kyle McEntee—to the employment statistics of the institution where I teach.**  

What Professor Merritt meant to say is that law schools have not been properly weighting their data to take into account nonresponse bias.  This is an interesting critique.  However, proper weights and adjustments to data should take into account all forms of nonresponse bias and response bias, not just the issue of over-representation of large law firms in NALP salary data raised by Professor Merritt.

While such over-representation would have an effect on the mean, it is unclear how much impact, if any, it would have on reported medians—the measure of central tendency used by The New York Times and critiqued by Mr. McEntee.

Other biases such as systematic under-reporting of incomes by highly educated individuals,*** under-reporting of bonuses and outside income, and the like should be taken into account.****   To the extent that these biases cut in opposite directions, they can offset each other.  It’s possible that in aggregate the data are unbiased, or that the bias is much smaller than examination of a single bias would suggest.  

Moreover, focusing on first year salaries as indicative of the value of a lifetime investment is itself a bias. As The Economic Value of a Law Degree, showed, incomes tend to rise rapidly among law graduates. They do not appreciably decrease, either, until the fourth decade of employment.

 

If Professor Merritt’s view is that differences between NALP, ABA, and U.S. Census Bureau data collection and reporting conventions make law school-collected data more difficult to compare to other data sources and make law school data less useful, then I am glad to see Professor Merritt coming around to a point I have made repeatedly.

I have gone further and suggested that perhaps the Census Bureau and other government agencies should be collecting all data for graduate degree programs to ensure the accuracy and comparability of data across programs and avoid wasting resources on duplicative data collection efforts.

This could also help avoid an undue amount of focus on short-term outcomes, which can be misleading in light of the rapid growth of law graduate earnings as they gain experience.  The inappropriate focus on the short term can be misleading if students are not aware of the growth trajectory and how it compares to the growth trajectory of likely earnings without a law degree.

*    Readers of Professor Merritt’s blog posts will be familiar with Professor Merritt’s general level of politeness.   In her latest, Professor Merritt describes me as “clueless.”

**   This tactic, bringing up the employment statistics of the institution where those whom she disagrees with teach, is something of a habit for Professor Merritt.  See her response Anders Walker at St. Louis).

***  Law graduates outside of the big firms are highly educated, high-income individuals compared to most of the rest of individuals in the United States.  That is the benchmark used by researchers when they identified the reporting biases in census data that lead to under-reporting of incomes.

 **** The risk of under-reporting income in law may be particularly high because of opportunities for tax evasion for those who run small businesses or have income outside of their salary.

 

UPDATE (4/14/2015):  I just confirmed with NALP that their starting salary data does not include end of year bonuses.