National Jurist Now Back-Pedalling on its Thomas Cooleyesque Law School Rankings

Since it's obvious--and should have been obvious prior to anyone pointing it out--that using Rate My Professors as part of a ranking of academic institutions is preposterous on its face, National Jurist is now doing a partial mea culpa, though so far, it hardly goes far enough.  Mr. Crittenden, the editor, actually had the audacity to do a video interview 48 hours ago in which he defended the use of Rate My Professors, stating, falsely, that studies had found it to be a valid measure.  There are no studies that validate the use of Rate My Professors the way NJ has used it.  None.  Zero.

A study at the University of Maine, involving more than 400 ratings by undergraduates, found that their scores on Rate My Professors generally matched internal Maine evaluations in two categories:  overall course evaluation and easiness/difficulty of the course.  But NJ used the results for the teaching quality categories, not these other categories.  The Maine study found no correlation with teaching quality, and they also found that the correlations that did exist in the other categories broke down outside the highest rated courses. 

Since law students barely use Rate My Professors, and since in some cases identified, the Rate My Professors score was based on non-law faculty and just a a few dozen responses (not 400!), it's clear that the data has no validity.  None.  It's worse than garbage. 

Mr. Crittenden even told an assistant dean here that if they dropped the fraudulent Rate My Professors data, Chicago would have gone from something like 50 to the top 5.   Since the data they used for Chicago included less than two dozen evaluations for actual law faculty, and twice as many evaluations for non-law faculty (including the lowest scores), this is a remarkable admission.   But this is almost surely just the tip of the iceberg!  As I noted earlier this week, Rate My Professors is driving all the variance among schools with very similar employment and other results.  (I learn from Mr. Crittenden's partial mea culpa that Princeton Review gives a very different portrait, but I know nothing about the methods by which PR compiles its data.)

Although National Jurist is now actually reviewing the "accuracy" of the Rate My Professors data, the question is why didn't they do that before publishing the ranking?  NJ says "we believe that the voice of students is essential."  But that's not in dispute.  The point is that Rate My Professors is not "the voice of students," and a magazine with any integrity, as opposed to an interest in generating hype, never would have utilized such an absurd source.  (I'm not even sure what it means to check the accuracy of Rate My Professors data:  anyone with Internet access on the planet earth can fill out a Rate My Professors survey, how could that be meaningful?)

As I indicated in an earllier post, I spoke with Mr. Crittenden just a few weeks ago about what would be required to do a sensible alternative to U.S. News.  He never let on that he and his staff had already concocted this nonsense ranking, and that he was going to use my quotes in an article accompanying this ranking, whose existence he had not disclosed to me.  Mr. Crittenden is clearly not an honest man or journalist.  He can only redeem his reputation by repudiating this whole fiasco.

NJ was always a bit of a joke, a magazine so unneeded that it is given away free in piles at every law school in the country (most copies end up in the trash).  But now they've done something perniciously stupid, which doesn't help students, and groundlessly defames dozens of law faculties.  I hope it marks the end of a pointless magazine.

ADDENDUM:  Professor Scott Bauries (Kentucky) writes:

I am enjoying your skewering of the obviously ridiculous effort of the National Jurist to create a new ranking system.  One further flaw with the use of Rate My Professors occurs to me that, to my knowledge, has not yet been reported (perhaps because it is obvious).  Other than by logging the IP address of the rater’s computer, Rate My Professors does not appear to ask its users for any personally identifying information or proof that a rater ever took a class from the professor rated.  This means that literally anyone who knows a professor’s name and institution can “rate” that professor. 

Therefore, if a school were of a mind to game this set of rankings, then it would be as easy as (1) pulling up the site; (2) searching for the school’s own professors; and (3) giving each one as many fraudulently high ratings, from as many unique IP addresses, as possible.  The site does ask for the course number and textbook, but these would be easy to communicate to false raters.  From the perspective of the unethical school, this sort of fraud would be perfect because it would be completely undetectable by outside observers (at least without a discovery request).  Given all of the efforts (both yours and those of the Law School Transparency folks and others) to eliminate opportunities to game the US News system, this flaw in the National Journal’s system should be particularly troubling.