March 26, 2015

The Absence of Evidence for Structural Change: Recent Entry Level Outcomes (Michael Simkovic)

Some have claimed that deteriorating outcomes for recent law school graduates are a sign of permanent structural change in the legal industry and that these changes are reducing the value of legal education. 

There are two important problems with this claim.  First, the same changes are taking place across the labor market, and are not a law-specific problem.  Indeed, the law degree has maintained its value relative to a bachelor’s degree.  Second, entry-level employment and starting salaries are known to be volatile and cyclical, so large swings aren’t a sign of much of anything other than business as usual in a recession or boom.

The Economic Value of a Law Degree lacked data on those who graduated after 2008 because of limitations of the Survey of Income and Program Participation (SIPPP).  Timing Law School supplements this data with additional information from the American Community Survey (ACS).  Using ACS we look at young lawyers and young professional degree holders excluding those in medical occupations—two proxies for law graduates, one under inclusive, the other over inclusive.   Both of these proxies (along with SIPP data) suggest that recent law graduates have maintained a large advantage relative to similar bachelor’s degree holders.  The ACS data is presented below.

  ACS Lawyers


ACS medical ex-professional

What many in the press and some law professors mistook for a law-specific crisis was in fact a widely known phenomenon in labor economics—employment and salaries for inexperienced workers are more volatile and sensitive to economic cycles than employment and salaries for those with more experience.



(The data above are for unemployment, but the picture is much the same for earnings.  See also here).

Predicting structural change on the basis of established cyclical patterns is analogous to drawing conclusions about permanent climate change on the basis of temperature changes between summer and winter.  Occasionally, the person making the prediction might get lucky and turn out to be right, but the evidence is weak, the analysis fails to test more plausible rival hypotheses, and the conclusion of permanent change is little more than a wild guess. 

Climate Scientists are more careful than this.  They use a back testing approach similar to the one Frank McIntyre and I use in Timing Law School. Back testing suggests that the prediction methods used to support the structural change hypothesis are baseless, at least with respect to such changes degrading the value of legal education.

Structural change can mean different things to different people.  By structural change, some people may simply mean that subjectively, the practice of law feels different than it used to, not that law graduates are getting any less value for the money.  Or they may mean more generally that the kind of work law graduates do is different, even if not relatively less well compensated.  This softer, humanistic view of structural change may have merit, although once again, it may also reflect broader trends in the economy rather than law-specific issues.

Over the last two weeks I’ve discussed the case for structural change—Bureau of Labor Statistics projections, entry-level outcomes, etc.—and found little support for the hypothesis that the value of a law degree has permanently declined.

Next week I’ll discuss another pillar of the structural change argument—growth rates in the “legal services” industry.

Posted by Michael Simkovic on March 26, 2015 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Professional Advice, Science, Student Advice, Weblogs | Permalink

March 25, 2015

Number of Law Graduates Does Not Predict Law Graduate Outcomes (Michael Simkovic)

Many legal educators believe that shrinking class sizes will help the students they do admit find higher paid work more easily and boost the value of legal education.  They reason that if the supply of law graduates shrinks, then the market price law graduates can command should increase. 

According to another hypothesis, now popular in the press, a decline in the number of law school applicants reflects the wisdom of the crowds.   Students now realize that a law degree simply isn’t worth it, and smaller class sizes reflect a consensus prediction of worse outcomes for law graduates in the future. 

Frank McIntyre and I investigated whether changes in law cohort size predict earnings premiums.  Historically, they have not.  Not for recent graduates, and not for law graduates overall.  Nor have changes in cohort size predicted much of anything about the entry-level measures used by the National Association for Law Placement (NALP)—starting salary, initial employment, initial law firm employment.

How can both of these theories be wrong?  One possibility is that they are both right, but the two effects offset each other.  This seems unlikely however.  If neither macroeconomic data  nor Bureau of Labor Statistics (BLS) employment projections can predict law employment conditions at graduation, then how likely is it that recent college graduates with less information and less expertise could make a better prediction?

A more likely possibility is that there are other factors at play that prevent any strong predictions about the relationship between cohort size and outcomes / value added.  For example, law schools may become less selective as cohort size shrinks and more selective as it increases.   In addition, the resources available to law schools, and therefore the quality of education and training they are able to provide, may also change with cohort size.  Since physical facilities expenses are not particularly variable in the ordinary course, most budgetary adjustments at law schools presumably take place with respect to personnel.

Anecdotally, many law schools appear to be managing the recent decline in enrollments by shrinking their faculties and administrations and using remaining personnel to teach classes and perform functions outside of their areas of expertise.  Reduced specialization and a lack of economies of scale could affect the quality of service provided to students, offsetting any benefits to students from less competition at graduation.

Previous research in labor economics has found that resources per student are an important predictor of value added by college education, and that the use of adjuncts can lead to worse outcomes for students. (See here for a review)

Much of this is speculative—we do not yet understand why changes in cohort size do not predict law graduate outcomes, only that they do not predict outcomes.  Given the historical data, it is probably not advisable to read too much into what the decline in law school enrollment means for students who will graduate over the next few years.

Instead, we should focus on the long-term historical data and the value of a law degree across economic cycles and enrollment levels.

Posted by Michael Simkovic on March 25, 2015 in Advice for Academic Job Seekers, Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Professional Advice, Science, Student Advice, Weblogs | Permalink

March 23, 2015

BLS Employment Projections Are Unreliable (Michael Simkovic)

Labor economists have long cautioned against the misuse of Bureau of Labor Statistics (BLS) employment projections.

In 2004, Michael Horrigan at the BLS explained that the BLS projections should not be used to value education or to attempt to predict shortages or surpluses of educated labor.  Instead, the value of education should be measured based on earnings premiums—the measure used in The Economic Value of a Law Degree and Timing Law School

Horrigan wrote:

The general problem with addressing the question whether the U.S. labor market will have a shortage of workers in specific occupations over the next 10 years is the difficulty of projecting, for each detailed occupation, the dynamic labor market responses to shortage conditions. . . .  Since the late 1970s, average premiums paid by the labor markets to those with higher levels of education have increased.


It is the growing distance, on average, between those with more education, compared with those with less, that speaks to a general preference on the part of employers to hire those with skills associated with higher levels of education.

The BLS takes the same position in its FAQ.  The BLS does not project labor shortages or surpluses.

In 2006, Richard Freeman back-tested the BLS projections and found that “the projections of future demands for skills lack the reliability to guide policies on skill development.”

The BLS employment projections are not only unreliable.  Comparing occupation-specific employment projections to number of graduates in related fields systematically underestimates the value of higher education.

In 2011 David Neumark, Hans Johnson, & Marisol Cuellar Mejia wrote:

If there are positive returns to education levels above those indicated as the skill requirement for an occupation in the BLS data – and especially if these wage premia are similar to those in other occupations – then relying on the BLS skill requirements likely substantially understates projected skill demands.


For nearly every occupational grouping, wage returns are higher for more highly-educated workers even if the BLS says such high levels of education are not necessary. For example . . . for management occupations, the estimated coefficients for Master’s, professional, and doctoral degrees are all above the estimated coefficient for a Bachelor’s degree, which is the BLS required level. . . ..


If the BLS numbers are correct, we might expect to see higher unemployment and greater underemployment of more highly-educated workers in the United States.  As noted earlier, we do not find evidence of this kind of underemployment based on earnings data. Similarly, labor force participation rates are higher and unemployment rates are lower for more highly educated workers.

(note—above I link to and quote the 2013 published article; Neumark et al. said much the same thing in their 2011 working paper).

Neumark et. al. also noted that recent BLS projections appeared to be much too low for managerial and legal services occupations.

Starting around 2012 many law professors and pundits argued that the number of job openings for lawyers projected by the BLS relative to the number of expected law graduates suggested that too many students were attending law school and that they would not get much value out of their degrees.

In 2012, Deborah Merritt wrote:

The Bureau [of Labor Statistic]’s occupational employment projections . . . answer the very question that many law school applicants want to know: How many new lawyers will the economy be able to absorb this decade?


The Bureau currently estimates that the economy will create 218,800 job openings for lawyers and judicial law clerks during the decade stretching from 2010 through 2020. That number, unfortunately, falls far short of the number of aspiring lawyers that law schools are graduating.


The oversupply of entry-level lawyers deprives many graduates of any opportunity to practice law. At the same time, the lawyer surplus constrains entry-level salaries.

Merritt notes the possibility that law might be a versatile degree with value outside of legal practice.

In 2012, Paul Campos wrote:

Further evidence that law degrees are unlikely to become more valuable going forward can be found in the projections of the Bureau for Labor Statistics (BLS) . . . [which suggest many more law graduates than job openings].”

 In 2013, Brian Tamanaha wrote:

The U.S. Bureau of Labor Statistics estimates about 22,000 lawyer openings annually through 2020 (counting departures and newly created jobs). Yet law schools yearly turn out more than 40,000 graduates. This bleak job market coexists with astronomically high tuition.

Several pundits and journalists also started comparing BLS projections and job openings to make much the same argument.

In 2013, unaware of the problems with job openings projections, I (Simkovic) suggested that projections might be used to make adjustments to more objective historical baselines for risk-based student loan pricing

On the chance that BLS projections that perform poorly in other contexts perform well in the legal education context, Frank McIntyre and I analyzed the extent to which BLS projections predict law graduate outcomes (earnings premiums).  The answer is: no better than random chance.  

As in other areas, BLS employment projections are not reliable or meaningful for predicting earnings premiums and are therefore not useful for valuing legal education.

But what about the number of job openings for lawyers?  Can BLS projections at least predict that reasonably well?

The answer appears to be “no.”  Recent analysis by the BLS suggests that their projection methods—the ones we analyze in Timing Law School—have systematically under-predicted job openings for lawyers.  

The BLS is considering changing its methodology to one that it believes will be more accurate.   Ted Seto has discussed this issue extensively. 

It is unclear at this point if the new job opening projections method will predict earnings premiums better than the old ones.  In any case, that was never their intended purpose, and it would be safer to predict earnings premiums and value education based on historical earnings premiums.

It remains likely that many law school graduates will not practice law.  Such has been the case in the past, and such is the case in other fields.   Many engineering, math and science graduates do not work as engineers, mathematicians or scientists in their fields of study.  Most fields of study do not have a one-to-one correspondence with a particular occupation, but are more broadly useful in the labor market, and law is no exception.  In spite of many individuals working outside their degree fields, higher education typically has been, and likely will remain, an investment with positive returns.

To best way to tell whether there is too much or too little investment in education is to consider relative returns that take into account risks and variability in employment.  Are the returns to education higher or lower than returns that can be had elsewhere with similar levels of risk?  The returns to education are generally much higher, and risk does not appear to explain this difference adequately.  The high relative returns to education suggest underinvestment in education.  

(A word of caution—BLS employment projections should not be confused with the BLS Occupational Employment Statistics (OES); the OES remains a useful and important, albeit imperfect, source of data). 

Posted by Michael Simkovic on March 23, 2015 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Professional Advice, Science, Student Advice, Television, Weblogs | Permalink

March 19, 2015

The Perils of Prediction (Michael Simkovic)

How can we test predictions about the future when we don’t yet have data showing what will happen in the future?  One answer is hindcasting.  You already believe in hindcasting if you believe in the science behind global warming (see also here and here).  

“Hindcasting” (or “backtesting”) is using historical data to test prediction methods and it is widely used in finance, engineering, and climate science.  The basic idea is that a prediction method can be reduced to a set of rules or mathematical formulas.  Historical data from the more distant past can be fed into these rules and formulas, and the resulting predictions about the “future” (relative to the distant past that provided the data) will also be predictions about the past (relative to the period in which the researcher conducts the backtest).   

Since data about the “future” is now available, predictions generated by the prediction method can be compared to what actually happened.  A prediction method does not have to be correct all of the time to be useful; if a prediction method performs a bit better than random chance, it might still be useful in many contexts, especially in investment management.  If it performs better than the next best prediction method, then it is still useful even if it is imperfect.  But if a prediction method does not perform any better than random chance, it is discredited and discarded.

 Using this hindcasting approach, Frank McIntyre and I test popular prediction methods used by various pundits and professors to try to predict whether now is a good or bad time to go to law school. (See Timing Law School)  As in our previous research, our primary outcome variable of interest is law earnings premiums—the earnings of law school graduates relative to the earnings of similar bachelor’s degree holders.  This is the relevant measure, because it goes to the value added by law school, and can be compared to the cost of attendance.

The peer-reviewed labor economics literature finds that a law degree has been a lucrative investment for the overwhelming majority of law school graduates compared to entering the labor market with just a bachelor’s degree.  Nevertheless, questions persist about whether now is an unusually good or bad time to start law school.

According to one popular hypothesis, now is an unusually bad time to go to law school because employment outcomes for recent graduates 9 months after graduation have deteriorated.   These graduates, it is argued, will not have the same career success as law school graduates in the past.  Moreover, deterioration in outcomes for those who graduated last year predicts poor outcomes three or four years in the future and beyond for those who are entering law school now.

According to another popular hypothesis, now is an unusually good time to go to law school because so few people are doing it.  When these small cohorts of law students eventually graduate, they will all be more likely to find a high paying job than the larger cohorts of the past.  A variation on this argument is that now is still a bad time to go to law school in spite of falling enrollments because the number of law school graduates will still be greater than the number of BLS projected job openings for lawyers.  (For a discussion of newer BLS projection methods showing more job openings, see here)

Our analysis includes graduates from 1964 through 2008 and earnings data from 1984 to 2013.  This period captures numerous economic booms and recessions.  As in The Economic Value of a Law Degree, our main source of data is the U.S. Census Bureau’s Survey of Income and Program Participation.  We were able to backfill the data to include older versions of the survey and capture more years of macroeconomic variation thanks to grant funding from the Access Group, Inc., and LSAC.  (Because the older data has some limitations, those who are interested in the value of a law degree rather than the size of cohort effects should still consult our 2014 article). 

None of the prediction methods we tested perform better than random chance.  Cohort size is not predictive.  Cohort size relative to BLS projections is equally useless.  Although those who graduate in a boom when unemployment is low do indeed have higher earnings premiums in their first few years after graduation than those who graduate when unemployment is low or moderate, the effect fades after the first four years.   More importantly, it is not possible to predict whether unemployment will be high or low four years in the future based on currently available data.  Even those who are unlucky enough to graduate into a weak economy still generally benefit substantially from their law degrees.

Delaying law school to attempt to “time the market” is an imprudent strategy.  It does not improve one’s chances of graduating into a favorable economic climate.  It entails substantial opportunity costs in the form of fewer years of higher, post-law-school earnings.  The cost of every year of delay averages tens of thousands of dollars.  Popular prediction methods for market timing are not only scientifically baseless; they also appear to be financially toxic to prospective students who take them seriously.  

The best guide to the future continues to be the long-term historical data.  Short-term fluctuations around these averages are not readily predictable.  Instead of trying to predict the unpredictable, it may be more prudent to focus on helping students manage these risks, for example through insurance programs similar to Income-Based Repayment of student loans. (See also here)

But what about more recent graduates?  How much can we say about those who graduated after 2008, and is this time different?  How can we explain our results in light of previous research on cohort effects focused on bachelor’s degree holders?

For answers to some of these questions, look for our next blog post. 

Posted by Michael Simkovic on March 19, 2015 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Professional Advice, Science, Weblogs | Permalink

October 30, 2013

Present Value and Cash Flows

Several critics of the Economic Value of a Law Degree have made mathematical errors or misunderstood the contents of the study.  One example relates to a fundamental financial concept, net present value.  The net present value is the value today of cash flows or payments that will be given or received in the future. 

The psychological and financial costs to the recipient of delay in payment are already incorporated into present value—present value is the equivalent of an immediate lump sum payment with no delay.

The difference between present value and nominal future value can be large.  For example, the value of a single $1,000,000 payment forty years from now is just over $97,200 today (assuming a 6 percent nominal discount rate).  In other words, receiving $1,000,000 in forty years is financially and psychologically the same as receiving $97,200 today.

In The Economic Value of a Law Degree, Frank McIntyre and I describe the law degree earnings premium—the difference in earnings between law degree holders and similar bachelor’s degree holders—on both an annual basis and, for the lifetime value, in present value terms.  In other words, we show what a lifetime of higher earnings is worth immediately, as of the start of law school, not spread out over the course of a lifetime. 

The pre-tax, pre-tuition present value of a lifetime of higher earnings is approximately $1,000,000 at the mean and $600,000 at the median.   This includes the opportunity costs of lower earnings while in school, and the cost of interest payments on student loans.

The law graduate will not get to keep the full present value.  Approximately 30 percent will go to the government as income and payroll tax revenue, and some of the remainder will go to the law school to pay for the cost of the legal education. 

One critic, Steven Harper, took an estimate of the after-tax, after tuition net present value at the median ($330,000) and erroneously claimed that this amount of money would be spread out over a 40-year career.   Dividing by 40 years and again by 12 months, Harper claimed that the law graduate would receive “at most a lifetime average of $687 a month.”  (Or $8,175 per year).

In other words, Harper conflated present value with future values and miscalculated the private return on a legal education.   If cash flows were level during the 40 years after law school, it would take more than $25,000 per year in after tax, after debt-service payment, nominal dollars to equal a present value of $330,000 as of the start of law school.  In 2012 inflation-adjusted dollars, it would require about $16,000 per year.   Harper is off by a factor of about 2 or 3.

In practice, cash flows will not be level—they will be lower in the initial years and rise through middle age.  The present value calculation already incorporates the cost of lower cash flows in the initial years.  To the extent that cash flows in the initial yeasrs are a concern, some students may use debt repayment options with lower payments in the initial years. The costs of these programs are already incorporated into our present value calculations.

Posted by Michael Simkovic on October 30, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Science, Weblogs | Permalink

October 29, 2013

The Economic Value of a Law Degree: Means, Medians, Modes (Michael Simkovic)

The three averages—means, medians, and modes—are basic mathematical concepts.  Nevertheless, they seem to have generated an inordinate amount of confusion among some critics of the Economic Value of a Law Degree (see herehere, and here).

Some of the critics have emphasized modes and medians while downplaying the importance of means.   Steven Harper, for example, has claimed that the mean is a “meaningless” statistic and we should instead focus on the medians and modes while ignoring the mean.

To understand his error, imagine two lecture halls, each with 100 seats.  Underneath each of those seats is a suitcase full of cash.  The individuals sitting in those seats will get to keep whatever cash they find when they open the suitcase.

In Lecture Hall A, every suitcase contains $600,000.  $600,000 is the mean, median, and mode value.

In Lecture Hall B, 60 of the suitcases contain $600,000, but the remaining 40 suitcases each contain $1,600,000.  The median and mode is exactly the same as in Lecture Hall A--$600,000.  But the mean is much higher in Lecture Hall B—it is $1,000,000 instead of $600,000.

If you didn’t know how much money would be in your suitcase, but you could choose between sitting in Lecture Hall A and Lecture Hall B, which room would you choose?

You would be wise to choose Lecture Hall B.  But the only reason to choose Lecture Hall B is because the mean (average) is higher in Lecture Hall B.  The median and mode in both lecture halls is identical.

Now imagine a slightly different fact pattern.  In Lecture Hall A there are three suitcases each containing $1.6 million, while the remaining 97 suitcases contain amounts that are close to $600,000 (i.e., a range from $599,950 to $600,050), with none of these 97 suitcases containing the exact same dollar value.  The mode value in Lecture Hall A is $1.6 million, while the median is $600,000 and the mean is $630,000.

Lecture Hall B is the same as in the previous fact pattern—60 suitcases contain $600,000 while 40 suitcases contain $1.6 million.  The mode value in Lecture Hall B is $600,000, while the mean is $1,000,000.

In other words, the mode is higher in Lecture Hall A, but the mean is higher in Lecture Hall B.  The medians are identical.    

Which room would you choose to sit in?

Once again, you would be wise to choose Lecture Hall B.  This suggests that you believe that means (averages) are more important than modes.

The money at the top (or the bottom) matters.  Means provide useful information that is not available from medians alone, and that is not reflected in modes.  That’s why we provide both means and medians, as well as 75th percentile and 25th percentile values in Economic Value of a Law Degree.  

(Posted by Michael Simkovic)

Posted by Michael Simkovic on October 29, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Science, Weblogs | Permalink

August 05, 2013

Sample size, standard errors, and confidence intervals

At law school café (reposted on Tax Prof) Deborah Merritt asks several questions about The Economic Value of a Law Degree related to sample size and uncertainty.    We thank Professor Merritt for her comments and hope they helped clarify the annual results for those who were having trouble interpreting Figures 5 and 6. In the paper we are careful to display the large confidence intervals for Figure 6, which looks at young law graduates over time, and we avoid drawing any strong conclusions from them.  Also, as we'll discuss below, one can readily reject that Figure 5's ups and downs are just noise. 

This post includes brief discussions of some of the interesting points raised. 

 The estimates in the paper don't depend on cyclical law school premia

We want to be clear that our underlying results do not rely on cyclicality.  SIPP annual estimates do not show a recent post-recession decline in the overall law graduate earnings premium that needs to be explained.   The recent decline in earnings for law graduates in our sample is matched by a decline in earnings for bachelor’s degree holders, and the law graduates retained their relative advantage.  But as one can see in Figure 6, the small sample for young lawyers makes it hard to be sure about the recent outcomes for that group in isolation.  Whether the premium cycles up and down or stays flat, over a lifetime every law grad will see many such transitions over their life, averaging out over time.


Is our overall sample size big enough?

Yes, our sample size is more than sufficient to support our conclusions on lifetime earnings.  The standard errors in Tables 1 to 4 reflect the degree of uncertainty about our estimates, which pool data over many years to increase precision. The standard errors are very small relative to our law degree earnings premium coefficient estimates, and our results are statistically significant well beyond conventional levels of statistical significance.  Deborah Merritt's discussion is focused specifically on what we can say about how the premium has changed over time (Figures 5 and 6).  As one can see in Figure 5, any changes in that premium have been fairly small relative to its size.  


How strong is the specific evidence from SIPP for cyclicality of earnings premiums?

Consistent with cyclicality, there is evidence of fluctuations of the earnings premium (measured on a percentage basis) in the 1996-2011 period.   Prompted by Deborah Merritt's concerns, we went ahead and added the joint test statistics to the figures in question.  We can reject the hypothesis that the law degree earnings premium was the same in all years from 1996-2011 (p<0.001).  In other words, fluctuations in the point estimate in Figure 5 are not all simply random noise.  Further, we don’t see evidence of a notable long term upward or downward trend.  Indeed, despite the occasional fluctuations we think the most noticeable feature of the law school premium recently is its stability.

Several previous studies have found evidence of fluctuations in law degree holder earnings premiums and starting salaries.  We cite many of these studies in the paper.  It would be a bad idea to extrapolate gloom or boom from a downward or upward trend in earnings using the last few years of data.  Trends, even when present, can stop or reverse themselves through dynamic labor market responses or exogenous shocks.  A sustained 85 percent decline in the lifetime earnings premium would be required for our main result--that a law degree is a value-creating investment for most law graduates--to no longer hold true.  Such a steep decline seems unlikely.

Though not crucial to our inqiury into lifetime earnings, it would be interesting to know if the premium rises and falls with the business cycle.  Prompted by the interest in this question, we did some exploratory analysis of data from the much larger, but less precise, American Community Survey which also seems to be consistent with fairly stable earnings premiums for recent cohorts of law graduates, but more research on the question will be useful, especially as passing time provides us more data.


How should we understand confidence intervals and point estimates?

Professor Merritt’s description of confidence intervals may seem to suggest that the true population parameter is equally likely to fall close to the point estimate as it is at the outer edges at the top or bottom of the confidence interval.  

This interpretation would be incorrect.  The probability density is highest at the center of the confidence interval, near the point estimate, and lowest at the outer edges of the confidence interval.   The point estimate is the best estimate of the population parameter.

Professor Merritt’s description also doesn't discuss the relationship between different point estimates, looking instead only at the confidence interval for each point estimate individually.  In a nutshell, two estimates may have overlapping confidence intervals and still be statistically separable.


How strong is the evidence for a bi-modal distribution of earnings? 

We don’t think the evidence for a bimodal distribution of lifetime earnings for law graduates is very compelling.  Recent full time starting salaries from NALP are not the same thing as lifetime earnings because:

Because earnings across people are close to log normally distributed it is typical to see a few people making a lot more than most people. 


Would bimodality cast doubt on the results of our analysis?

Bimodality does not really call for change to our approach, even if present.  As the sample gets larger the sampling distribution is asymptotically normal, so standard errors on our key results should be consistent.  Regression techniques are consistent regardless of the underlying distribution, but for those concerned about a thick right tail, we'd suggest they concentrate on the results in Tables 1 and 2 that use a log transformation—reducing such concerns.  Bi-modality in the earnings distribution would also not change how we did our quantile regressions.   Quantile regressions estimate the earnings premium at different points in the distribution independent of the shape of the overall distribution.  

Posted by Michael Simkovic on August 5, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Science, Weblogs | Permalink

August 01, 2013

The Economic Value of a Law Degree: Correcting Misconceptions


Ability sorting and selection

In The Economic Value of a Law Degree, Frank McIntyre and I estimate the increase in annual and lifetime earnings that is attributable to a law degree.  To do so, we compare those with law degrees to similar individuals with less education.

Because those who matriculate at law schools may be different from the average bachelor’s degree holder, we compare law degree holders to a group of similar bachelor’s degree holders.  

There is a misperception—apparently started by Brian Tamanaha (here and here) and repeated by others—that we simply compare law degree holders to all bachelor’s degree holders, or that we compare the 25th percentile of law degree holders to the 25th percentile of all bachelor’s degree holders.  This is not true.

At a high level, what we essentially did was to create two subgroups of bachelor’s degree holders—all bachelor’s degree holders, and a subset of bachelor’s degree holders who look like the law degree holders with respect to many observable characteristics that predict earnings—demographics, academic achievement, parental socio-economic status, measures of motivation and values.  It is this second group of bachelor’s degree holders that we compare to the law degree holders.

To check for ability sorting and selection, we use statistical techniques including:


The observable characteristics (pretreatment covariates) that we focus on as controls in the Survey of Income and Program Participation include:


These controls bring down our earnings premium estimates by around 10 percent at the mean and around 8 percent at the 25th percentile. 

Controls and no controls


In other words, the data and statistical techniques that we use suggest that the kinds of people who go to law school would probably earn about 10 percent more than the average bachelor’s degree holder even if they hadn’t gone to law school.  But the law school earnings premium is much greater than that, and the earnings premiums we report are after controls for ability sorting.

We do an additional check for ability sorting using another data set called the National Education Longitudinal Study (NELS).  NELS follows a cohort from 8th grade through their late 20s, and includes additional pretreatment control variables that are not available in SIPP.

Controls that are available in NELS include:

The results of the analysis using NELS are very similar to the results of the analysis in SIPP.  The bachelor’s degree holders who go on to law school would probably earn about 10 percent more than the average bachelor’s degree holder, even if they had not gone to law school.

Because this level of ability sorting was already taken into account in our SIPP analysis, we do not believe that any further adjustment to our SIPP results would be justified based on the analysis in NELS.  Because different measures of ability that predict earnings are often correlated with each other, adding more and more control variables that measure essentially the same thing often won’t substantially change the estimate of the earnings premium.

Thus we found very little to suggest that law graduates’ above average undergraduate academic performance translates into higher earnings other than what we had already accounted for.  This may be surprising to people for two reasons.  First, law degree holder undergraduate academic performance is better but not fantastically better than the typical BA.  Second, that above average performance does not actually translate into much of a boost to earnings.   It turns out higher undergraduate grades, for example, do not show a strong correlation with later earnings.  We find that this is especially true, by the way, in the majors preferred by law students in the humanities and social sciences.

 College Majors


Eric Rasmusen has an interesting blog post qualitatively describing the "typical" law student.

There are several other issues related to selection on unobservables and offsetting biases that are worth mentioning.

Annual vs. Lifetime and regression to the median:

Annual earnings tend to be much more varied than longer-term lifetime earnings.  For one example, job losses or transitions can cause a sharp drop in one year, but tend to be resolved by the next year.  People going through such temporary rough spots show up low in the earnings distribution.  So the 25th percentile of one year earnings is much lower than the 25th percentile over average lifetime earnings.

Reporting Bias:

When reporting earnings, people tend to not report periods of unemployment and such.  The SIPP returns to interview people every four months, so this is not as much of a problem as it could be, but it means that low income people tend to over-report their income relative to those higher up.  This typically will bias down estimates of how much more one group earns than another.

Specific Ability:

People tend to pick the career they will succeed at.  Thus those who are bad at some jobs but good at jobs available to law degree holders will gravitate towards law.  But, in fact, had they not gone in to law they might end up doing very badly.  This has several effects – it means that we will tend to underestimate the value of law school to those who choose law because that is their particular advantage but at the same time we may be overestimating it for those who are not choosing law.  It is hard to know for sure if this is a large effect or not.  It is very difficult to nail down statistically.

The 25th Percentile:

When we look at the 25th percentile earnings lawyer we use quantile regression to make these ability adjustments to the data before comparing them to the 25th percentile earnings BA, thus we’re correcting for ability as much as possible.  Though not reported in the paper we find the ability gap (that we adjust for in our lifetime value estimates) between BA and law grads is about eight percentage points at the 25th percentile.  This is completely in line with what we found at the mean both in the SIPP and in our more refined estimates from the NELS survey.  It is possible that the gap is larger (or smaller) at the bottom than our data show, so that would be a great place for future research, but we think this is the best currently available estimate, especially given issues (1) and (2) biasing the premium down.


Occupation and the versatile law degree

A very large fraction of law degree holders do not end up practicing law.  For some, this is a disappointment and for others it is a preferred outcome.  We include all these people in our estimates of the value of a law degree.  That is because the question we are interested in answering is the value of the law degree, not the earnings of the subset of individuals who practice law.   Controlling for occupation would have been methodologically improper because occupation is an outcome variable, not a pretreatment covariate.

As MIT labor economist Joshua Angrist and LSE labor economist Jörn-Steffen Pischke explain in Mostly Harmless Econometrics:

Some variables are bad controls and should not be included in a regression model even when their inclusion might be expected to change the short regression coefficients.   Bad controls are variables that are themselves outcome variables . . . That is, bad controls might just as well be dependent variables too. The essence of the bad control problem is a version of selection bias . . .

To illustrate, suppose we are interested in the effects of a college degree on earnings and that people can work in one of two occupations, white collar and blue collar. A college degree clearly opens the door to higher-paying white collar jobs.  Should occupation therefore be seen as an omitted variable in a regression of wages on schooling?  After all, occupation is highly correlated with both education and pay.  Perhaps it’s best to look at the effect of college on wages for those within an occupation, say white collar only.   

The problem with this argument is that once we acknowledge the fact that college affects occupation, comparisons of wages by college degree status within an occupation are no longer apples-to-apples, even if college degree completion is randomly assigned . . . [because of selection bias].

We would do better to control only for variables that are not themselves caused by education. 

In a recent article, David Neumark and co-authors also include a helpful explanation of the problems with controlling for occupation and “underemployment”, or relying on BLS occupational earnings projections when trying to measure education earnings premiums:

For nearly every occupational grouping, wage returns are higher for more highly-educated workers even if the BLS says such high levels of education are not necessary. For example . . . for management occupations, the estimated coefficients for Master’s, professional, and doctoral degrees are all above the estimated coefficient for a Bachelor’s degree, which is the BLS required level. . . ..

If the BLS numbers are correct, we might expect to see higher unemployment and greater underemployment of more highly-educated workers in the United States.  As noted earlier, we do not find evidence of this kind of underemployment based on earnings data. Similarly, labor force participation rates are higher and unemployment rates are lower for more highly educated workers.”

Even economists at the BLS emphasize that educational earnings premiums, and not BLS employment projections, are the key measure of the value of education:

The general problem with addressing the question whether the U.S. labor market will have a shortage of workers in specific occupations over the next 10 years is the difficulty of projecting, for each detailed occupation, the dynamic labor market responses to shortage conditions. . . .

Since the late 1970s, average premiums paid by the labor markets to those with higher levels of education have increased.

It is the growing distance, on average, between those with more education, compared with those with less, that speaks to a general preference on the part of employers to hire those with skills associated with higher levels of education.


Long term versus short term

We value a law degree based on the present value of a lifetime of increased earnings. The valuation literature is unambiguous about the correct time period to value the cash flows generated by an asset:  the entire life of the asset.  The delay and higher risks of cash flows in the distant future are already taken into account through the application of a discount rate and the present value formula.

Our approach, using the typical span of a working life and discounting back to present value, is the correct one for the majority of potential law students who obtain their degrees relatively early, in their 20s or 30s.  A much shorter time period would only be appropriate for individuals who complete their law degrees later in life, closer to retirement, or who anticipated working only a few years during their lifetimes.

In a recent post post, Brian Tamanaha suggests that the difference between his approach and ours is that he focused on the short-term value of a law degree while we focused on the long-term value of a law degree.    

Michael Froomkin wonders if law degree holders will experience a cash crunch early in their careers when their incomes are lower and debt levels are higher. 

It is unlikely that a debt financed law degree would create a cash crunch.  Young bachelor’s degree holders also have lower incomes early in their careers.  The earnings premium associated with the law degree will typically exceed required debt service payments on law school debt, particularly in light of the availability of extended repayment, deferment, forbearance, and income based repayment plans.  Graduate degrees can readily be financed entirely with federal student loans.

The costs of delayed repayment (i.e., higher interest) are already taken into account in our present value calculation, because we discount back at the weighted average interest rate on law school debt.  We’re pretty conservative in this respect: we ignore the (likely) possibility that students will prepay their highest interest rate debts first.  Indeed, After the JD II found evidence of rapid pre-payment of law school debt.

 After the JD prepayment


Our results suggest that most young law degree holders most of the time likely have more positive cash flow—even after debt service payments—than they would likely have had with only a bachelor’s degree.

Because the economic value of a given level of education can generally be maximized by completing that level of education early—and thereby maximizing the number of years of subsequent work with the benefit of higher wages from the education earnings premium—delaying graduate school to try to time the market is a high-cost strategy.   And timing the market three or four years in advance is difficult. 

We recommend long-term historical data on lifetime earnings premiums as a guide rather than short-term fluctuations in starting salaries.  Indeed, starting salaries tell us very little—earnings premiums are what matters, and there is no evidence that premiums have compressed, even for the young. 


Youngster Cycles



In a supplemental exploratory analysis using ACS data, we find some evidence that post 2008 cohorts of individuals who are probably young law degree holders (professional degree holders excluding those in medical practice) continue to have the same earnings advantage over bachelor’s as they had prior to 2008.

Ben Barros has done some interesting work comparing outcomes 9 months after graduation to subsequent outcomes for recent graduates of Widener Law School.


The broader labor market

Tamanaha argues that law continues to be depressed while the rest of the labor market has recovered.   The data does not support this view.  As can be seen from the chart below, the broader employment population ratio remains below 2007 levels across levels of education, and the more educated continue to be more likely to work than those with less education.




Present value and opportunity costs

Many of our critics have made mistakes relating to net present value, opportunity costs, and direct costs of a law degree.  Some general guidelines are provided below.

  1. Everything has to be discounted back to the start of law school
  2. Costs can't be something that is already taken into account through opportunity cost of lower in school earnings
  3. Costs have to be something that the law student would only incur for law school and not matched by any other comparable expense if the student were a working BA; the cost has to be something that is a necessary expense to attend law school
  4. The cost can't provide consumption benefits that justify the greater expense
  5. The cost has to be what the student actually spends, and not hypothetically what a student might have spent if the student had paid full price

For example, since living expenses would be paid out of higher earnings if law students were working, we have already taken cost of living into account.  

Since many students receive scholarships and grants, full-sticker tuition should not be used as a base-case.

Our estimates of in-school earnings are based on data from the SIPP and other Census Bureau Surveys.  As we note in footnote 101:

Footnote 101: We assume that law students earn $5,000 in their first year, $7,000 in their second year and $12,000 in their third year with part time and summer work, for a total of $24,000 during law school. SIPP data suggests typical three-year in-school earnings between $21,800 (median) and $48,000 (mean) for fulltime graduate and professional school students. Census data suggests substantial work hours among fulltime graduate and professional students See Jessica Davis, U.S. CENSUS BUREAU, SCHOOL ENROLLMENT AND WORK STATUS: 2011 (Oct. 2012).”


Thanks and Goodbye

It’s been a fun couple of weeks.  We’d like to thank Brian Leiter, Brian Tamanaha, and others for the wonderful opportunity they’ve given us to explain our research to a wider audience.  And I’d like to thank Frank McIntyre for his contributions to this post and previous posts.  This will hopefully be our last post about The Economic Value of a Law Degree, at least for a little while.

Posted by Michael Simkovic on August 1, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Science, Weblogs | Permalink

July 29, 2013

Brian Tamanaha’s Straw Men (Part 4): We would have to be off by 85 percent for our basic conclusion to be incorrect


Brian Tamanaha’s latest blog post has been hailed by Stephen Diamond and others as a major concession.  But in a comment below the post, Tamanaha explains to one of his followers:

 “I believe the doubts I raised about the study in my previous three posts have not been answered satisfactorily.”

We therefore continue our response to Tamanaha’s first three posts before addressing Tamanaha’s fourth post.

BT Claim 4:  Historical economic data tells us nothing about the future

"It is exeedingly rare to find reliably predictive 'historical norms' in the social sciences because social life is too complex and circumstances are constantly changing . . . S&M have produced a narrow, partial, time-bound study that has zero predictive relevance for anyone thinking about attending law school today." A proper study "may require data over several centuries."  

Response: We would have to be off by 85 percent for our basic conclusion to be incorrect

In finance, valuation entails using historical data to establish a baseline scenario.  This baseline is generally viewed as the center of a distribution of possible future outcomes.  The baseline can be modified to construct upside and downside scenarios to get a sense of what could happen if the future is better or worse than the past.  Scenario analysis can help understand how robust the findings are--that is, how much the future would need to deviate from the past to change the basic directional conclusion of the valuation analysis.  For the extreme downside, this is sometimes called "break-even analysis."

For general background focused on the corporate context, I recommend Tim Koller, Marc Goedhart & David Wessel, McKinsey, Valuation: Measuring and Managing the Value of Companies (4th Edition), and Brealey, Myers & Allen, Principles of Corporate Finance.

We estimate the present value of a law degree at the median as $610,000 as of the start of law school.  This figure is pre-tax and pre-tuition, but includes opportunity costs and financing costs. 

In other words, some combination of the student and the federal government could pay up to $610,000 for the law degree and break even.  The government might contribute to the cost through debt forgiveness through Income Based Repayment, or through some other method.

As we note in the paper, ABA data suggest that the typical tuition cost for law school, less scholarships and grants, is roughly around $30,000 per year.  Spread over 3 years, and assuming tuition rises 6 percent per year nominal (i.e., at our discount rate), this comes to $90,000 in present value terms as of the start of law school.

For law school to cease to be a value-creating investment for the majority of law students, the present value of the lifetime earnings premium would have to fall to below $90,000—a drop of 85 percent.

At the “25th percentile” (more like the 15th because of regression to the median), toward the bottom of the distribution, the law school earnings premium is $350,000.  Assuming tuition (less scholarships and grants) remains at $30,000, the 25th percentile premium would need to fall by 74 percent for a law degree to no longer be value-creating proposition toward the bottom of the distribution.  At the mean, we’d have to be off by 91 percent.

These would be extreme deviations from the pattern seen in 1996-2011.

Posted by Michael Simkovic on July 29, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Science, Weblogs | Permalink

July 28, 2013

Repetitive (and avoidable) mistakes

At the American Lawyer, Matt Leichter repeats many misrepresentations of our research that originally appeared in the tabloid Above the Law, even after Above the Law posted corrections and after we refuted many of these misrepresentations.  He also refers to anonymous comments attacking our research from people who did not read it.  

Steve Harper makes many of the same mistakes, and throws in a few disparaging remarks to boot.

Slide 20 Blog Post 1


Slide 17 Blog Post 1

We've already responded to many of these same misrepresentations of our research from Above the Law, Brian Tamanaha, and Paul Campos. Simple fact-checking, either by reading the article or by checking our blog posts, could have prevented these errors.

Hopefully the editors at the American Lawyer will promptly post corrections and have a serious discussion with Mr. Leichter and Mr. Harper about the differences between critiquing research on the merits and misrepresenting the contents of that research--and impugning the integrity of its authors--in a nationally distributed publication.

Posted by Michael Simkovic on July 28, 2013 in Guest Blogger: Michael Simkovic, Legal Profession, Science, Weblogs | Permalink