The Perils of Prediction (Michael Simkovic)

How can we test predictions about the future when we don’t yet have data showing what will happen in the future?  One answer is hindcasting.  You already believe in hindcasting if you believe in the science behind global warming (see also here and here).  

“Hindcasting” (or “backtesting”) is using historical data to test prediction methods and it is widely used in finance, engineering, and climate science.  The basic idea is that a prediction method can be reduced to a set of rules or mathematical formulas.  Historical data from the more distant past can be fed into these rules and formulas, and the resulting predictions about the “future” (relative to the distant past that provided the data) will also be predictions about the past (relative to the period in which the researcher conducts the backtest).   

Since data about the “future” is now available, predictions generated by the prediction method can be compared to what actually happened.  A prediction method does not have to be correct all of the time to be useful; if a prediction method performs a bit better than random chance, it might still be useful in many contexts, especially in investment management.  If it performs better than the next best prediction method, then it is still useful even if it is imperfect.  But if a prediction method does not perform any better than random chance, it is discredited and discarded.

 Using this hindcasting approach, Frank McIntyre and I test popular prediction methods used by various pundits and professors to try to predict whether now is a good or bad time to go to law school. (See Timing Law School)  As in our previous research, our primary outcome variable of interest is law earnings premiums—the earnings of law school graduates relative to the earnings of similar bachelor’s degree holders.  This is the relevant measure, because it goes to the value added by law school, and can be compared to the cost of attendance.

The peer-reviewed labor economics literature finds that a law degree has been a lucrative investment for the overwhelming majority of law school graduates compared to entering the labor market with just a bachelor’s degree.  Nevertheless, questions persist about whether now is an unusually good or bad time to start law school.

According to one popular hypothesis, now is an unusually bad time to go to law school because employment outcomes for recent graduates 9 months after graduation have deteriorated.   These graduates, it is argued, will not have the same career success as law school graduates in the past.  Moreover, deterioration in outcomes for those who graduated last year predicts poor outcomes three or four years in the future and beyond for those who are entering law school now.

According to another popular hypothesis, now is an unusually good time to go to law school because so few people are doing it.  When these small cohorts of law students eventually graduate, they will all be more likely to find a high paying job than the larger cohorts of the past.  A variation on this argument is that now is still a bad time to go to law school in spite of falling enrollments because the number of law school graduates will still be greater than the number of BLS projected job openings for lawyers.  (For a discussion of newer BLS projection methods showing more job openings, see here)

Our analysis includes graduates from 1964 through 2008 and earnings data from 1984 to 2013.  This period captures numerous economic booms and recessions.  As in The Economic Value of a Law Degree, our main source of data is the U.S. Census Bureau’s Survey of Income and Program Participation.  We were able to backfill the data to include older versions of the survey and capture more years of macroeconomic variation thanks to grant funding from Access Group, Inc., and LSAC.  (Because the older data has some limitations, those who are interested in the value of a law degree rather than the size of cohort effects should still consult our 2014 article). 

None of the prediction methods we tested perform better than random chance.  Cohort size is not predictive.  Cohort size relative to BLS projections is equally useless.  Although those who graduate in a boom when unemployment is low do indeed have higher earnings premiums in their first few years after graduation than those who graduate when unemployment is low or moderate, the effect fades after the first four years.   More importantly, it is not possible to predict whether unemployment will be high or low four years in the future based on currently available data.  Even those who are unlucky enough to graduate into a weak economy still generally benefit substantially from their law degrees.

Delaying law school to attempt to “time the market” is an imprudent strategy.  It does not improve one’s chances of graduating into a favorable economic climate.  It entails substantial opportunity costs in the form of fewer years of higher, post-law-school earnings.  The cost of every year of delay averages tens of thousands of dollars.  Popular prediction methods for market timing are not only scientifically baseless; they also appear to be financially toxic to prospective students who take them seriously.  

The best guide to the future continues to be the long-term historical data.  Short-term fluctuations around these averages are not readily predictable.  Instead of trying to predict the unpredictable, it may be more prudent to focus on helping students manage these risks, for example through insurance programs similar to Income-Based Repayment of student loans. (See also here)

But what about more recent graduates?  How much can we say about those who graduated after 2008, and is this time different?  How can we explain our results in light of previous research on cohort effects focused on bachelor’s degree holders?

For answers to some of these questions, look for our next blog post.