Problems with the SSRN Rankings

There was a bit of chatter among law profs about the new SSRN download rankings of law schools and individual faculty awhile back.  (Free registration required for access.)  I noted some of the limitations of this measure at the time.  (My colleague Bernie Black and Cincinnati's Paul Caron [emperor of Law Prof Blog land!] discuss SSRN rankings here.)  Now an anonymous commenter on one of Professor Morriss's recent postings on rankings at the Volokh site has given a particularly good summary of the problems with the SSRN rankings; some excerpts:

1. Economics professors, corporate scholars, and IP scholars use SSRN much more often than other legal professors. Their papers dominate the most downloaded papers lists. While these are interesting fields, they are extremely overrepresented in the ratings.

2. Blogs make a big difference: With no offense given to the fine writers at volokh, of course, who would no doubt be well received without their blog, a mention on one of the more popular blogs can have a very substantial impact on downloads. Take a look at the SSRN page for that "Perfect Crime" article in Georgetown that Volokh posted about a few months ago. It's a fun article, but should it really have far more downloads than anything else Georgetown is publishing this year?

3. Specialty articles make too big of a difference: Articles that rate law schools or law faculties, for example, get a huge number of downloads; typically more than 10-15 good articles at well-rregarded journals get combined. Do these articles have that much more of an academic impact?

4. A few big name scholars drastically skew things: There are a handful of scholars that get more reads than entire other schools. These scholars deserve recognition, but do they actually have more scholarly impact then entire faculties?

5. Older scholars are underrepresented: At least at my school, a number of older scholars are still quite prolific but don't use SSRN unless forced.

6. There's no standard for what to upload: Some professors only upload articles they are still working on and would like feedback on. Others upload articles they wrote 6 years ago, along with their various book chapters.

7. Downloads just aren't the same as actual citations: Setting aside whether even citations mean anything important about scholarship, a citation, at the very least, means that another scholar respected an article enough to reference it in his or her own work.  [Ed.-I doubt citations even mean this.] Downloads don't mean the same thing. An abstract could sell a paper as having an entirely new schema for analyzing an area of law, then the paper could turn out to be garbage. But, SSRN would already reflect the paper as influential. Heck, the attached paper could be a blank .PDF with a catchy title, and it would do well.

SSRN downloads are certainly a very good measure of...who is getting downloaded on SSRN a lot!  But more than that:  some of the problems noted above may wash out across whole faculties, and the resulting lists of individuals and schools does bear, in many instances, a stronger relationship to faculty quality than, say, U.S. News.  And one consequence of creating the SSRN rankings will be that more faculty will be inspired to put their papers on SSRN--and that's a good thing, independent of any ranking.  The important story about SSRN--and this is related to our discussion yesterday about student-edited law reviews--is that it may, in fact, be on the verge of supplanting law review publication altogether.