Monday, August 5, 2013

Sample size, standard errors, and confidence intervals

At law school café (reposted on Tax Prof) Deborah Merritt asks several questions about The Economic Value of a Law Degree related to sample size and uncertainty.    We thank Professor Merritt for her comments and hope they helped clarify the annual results for those who were having trouble interpreting Figures 5 and 6. In the paper we are careful to display the large confidence intervals for Figure 6, which looks at young law graduates over time, and we avoid drawing any strong conclusions from them.  Also, as we'll discuss below, one can readily reject that Figure 5's ups and downs are just noise. 

This post includes brief discussions of some of the interesting points raised. 

 The estimates in the paper don't depend on cyclical law school premia

We want to be clear that our underlying results do not rely on cyclicality.  SIPP annual estimates do not show a recent post-recession decline in the overall law graduate earnings premium that needs to be explained.   The recent decline in earnings for law graduates in our sample is matched by a decline in earnings for bachelor’s degree holders, and the law graduates retained their relative advantage.  But as one can see in Figure 6, the small sample for young lawyers makes it hard to be sure about the recent outcomes for that group in isolation.  Whether the premium cycles up and down or stays flat, over a lifetime every law grad will see many such transitions over their life, averaging out over time.


Is our overall sample size big enough?

Yes, our sample size is more than sufficient to support our conclusions on lifetime earnings.  The standard errors in Tables 1 to 4 reflect the degree of uncertainty about our estimates, which pool data over many years to increase precision. The standard errors are very small relative to our law degree earnings premium coefficient estimates, and our results are statistically significant well beyond conventional levels of statistical significance.  Deborah Merritt's discussion is focused specifically on what we can say about how the premium has changed over time (Figures 5 and 6).  As one can see in Figure 5, any changes in that premium have been fairly small relative to its size.  


How strong is the specific evidence from SIPP for cyclicality of earnings premiums?

Consistent with cyclicality, there is evidence of fluctuations of the earnings premium (measured on a percentage basis) in the 1996-2011 period.   Prompted by Deborah Merritt's concerns, we went ahead and added the joint test statistics to the figures in question.  We can reject the hypothesis that the law degree earnings premium was the same in all years from 1996-2011 (p<0.001).  In other words, fluctuations in the point estimate in Figure 5 are not all simply random noise.  Further, we don’t see evidence of a notable long term upward or downward trend.  Indeed, despite the occasional fluctuations we think the most noticeable feature of the law school premium recently is its stability.

Several previous studies have found evidence of fluctuations in law degree holder earnings premiums and starting salaries.  We cite many of these studies in the paper.  It would be a bad idea to extrapolate gloom or boom from a downward or upward trend in earnings using the last few years of data.  Trends, even when present, can stop or reverse themselves through dynamic labor market responses or exogenous shocks.  A sustained 85 percent decline in the lifetime earnings premium would be required for our main result--that a law degree is a value-creating investment for most law graduates--to no longer hold true.  Such a steep decline seems unlikely.

Though not crucial to our inqiury into lifetime earnings, it would be interesting to know if the premium rises and falls with the business cycle.  Prompted by the interest in this question, we did some exploratory analysis of data from the much larger, but less precise, American Community Survey which also seems to be consistent with fairly stable earnings premiums for recent cohorts of law graduates, but more research on the question will be useful, especially as passing time provides us more data.


How should we understand confidence intervals and point estimates?

Professor Merritt’s description of confidence intervals may seem to suggest that the true population parameter is equally likely to fall close to the point estimate as it is at the outer edges at the top or bottom of the confidence interval.  

This interpretation would be incorrect.  The probability density is highest at the center of the confidence interval, near the point estimate, and lowest at the outer edges of the confidence interval.   The point estimate is the best estimate of the population parameter.

Professor Merritt’s description also doesn't discuss the relationship between different point estimates, looking instead only at the confidence interval for each point estimate individually.  In a nutshell, two estimates may have overlapping confidence intervals and still be statistically separable.


How strong is the evidence for a bi-modal distribution of earnings? 

We don’t think the evidence for a bimodal distribution of lifetime earnings for law graduates is very compelling.  Recent full time starting salaries from NALP are not the same thing as lifetime earnings because:

  • Full time salary excludes those who are working less than full time
  • Salaries exclude bonuses, which may be more variable than earnings
  • Starting salaries tend to be fairly lockstep compared to later earnings
  • After the JD II suggests faster growth of earnings (on a percentage basis) for graduates of lower ranked schools who have lower average initial earnings, which suggests convergence of earnings over time

Because earnings across people are close to log normally distributed it is typical to see a few people making a lot more than most people. 


Would bimodality cast doubt on the results of our analysis?

Bimodality does not really call for change to our approach, even if present.  As the sample gets larger the sampling distribution is asymptotically normal, so standard errors on our key results should be consistent.  Regression techniques are consistent regardless of the underlying distribution, but for those concerned about a thick right tail, we'd suggest they concentrate on the results in Tables 1 and 2 that use a log transformation—reducing such concerns.  Bi-modality in the earnings distribution would also not change how we did our quantile regressions.   Quantile regressions estimate the earnings premium at different points in the distribution independent of the shape of the overall distribution.

Guest Blogger: Michael Simkovic, Legal Profession, Of Academic Interest, Science, Weblogs | Permalink