World University Rankings by the Times Higher Education Supplement (THES) (London)

THES has published their second ranking of leading universities around the world here (you can register for a 2-week free trial period to see the rankings).  In format it looks like a U.S. News excercise, but (and this is key!) the focus is (largely) on the research and graduate quality of institutions.  60% of the overall rank is, happily, based on research/faculty quality:  40% based on an international reputation survey of academics; 20% on citations as measured by Thomson ISI's database (which is skewed towards medicine and the hard sciences).  That's the good news! 

Unfortunately, being journalists, the THES folks apparently have no real idea what they're doing, so mixed in with these research-oriented criteria are incommensurable and unrelated factors:  an evaluation of graduates by employers (10%); the percentage of international faculty and students (5% each); and faculty/student ratio (20%).   These factors aren't uninteresting, but how they should be weighted relative to the research strengths of an institution is utterly mysterious.

THES is clearly still figuring out what it's doing, and some of the mistakes are rather dramatic.  In 2004, for example, the University of Texas at Austin was ranked 15th in the world by THES, and in the top ten in the US, a result that was, shall we say,  a bit surprising.  Scrutinizing the data, it became clear to me what had happened:  Texas was ranked in the top ten in citations per faculty member, an astonishing result for a university without a medical school (research in medicine drives citation measures), but one that would make good sense if the University of Texas Southwestern Medical School at Dallas (one of the real medical powerhouses in the world, with more Nobel Laureates on its faculty than any other medical school in the US) were counted as part of the Austin results, which is obviously what had happened.  This year, THES apparently figured out that Austin and Dallas are two different campuses, and UT Austin's citation per faculty member dropped dramatically--as did its overall rank (from 15th in the world, to 26th, and to 12th in the US--more realistic results). 

Although THES claims to include only schools with undergraduates, they rank the University of California, San Francisco 17th in the world, though UCSF is exclusively devoted to graduate education in the health fields (that it fares so well is one indication of how skewed the results are to universities with strengths in medicine and biological sciences).  Duke jumped from outside the top 50 last year into the top 15, a change that can only mean there was some kind of mistake last year or this (probably last year--my guess is THES neglected to count the medical school). 

All this being said, the core THES data on research quality of institutions is interesting, and the effort is far superior to the absolutely bizarre "world university rankings" being put out by a Chinese university, which rewards institutions for its dead Nobel Laureates (if you used to be really good, that counts in your favor here!) and--because it has no peer review component and relies heavily on citation measures--is hugely skewed to schools strong in medicine, engineering, and the hard sciences (a first-rate "arts and humanities" school would barely register in the Chinese rankings).  On the other hand, THES is not sufficiently forthcoming about its methods and underlying data to permit a thorough evaluation of its results.

According to the 2005 Times Higher Education Supplement, here are the top 20 American research universities (excluding the medical school at UC San Francisco, for the reasons noted above); given the big impact of medicine on these rankings, I note which schools also have medical schools, and note, very roughly, its quality.  Bear in mind that this is the overall ranking based on the odd stew of criteria noted above:

1.  Harvard University (excellent medical school)

2.  Massachussetts Institute of Technology (no medical school)

3.  Stanford University (excellent medical school)

4.  University of California, Berkeley (no medical school)

5.  Yale University (strong medical school)

6.  California Institute of Technology (no medical school)

7.  Princeton University (no medical school)

8.  Duke University (excellent medical school)

9.  Cornell University (strong medical school)

10. University of Chicago (strong medical school)

11. Columbia University (strong medical school)

12. University of Texas, Austin (no medical school)

13. Johns Hopkins University (excellent medical school)

14. University of Pennsylvania (excellent medical school)

15. University of Michigan, Ann Arbor (strong medical school)

16. University of California, Los Angeles (strong medical school)

17. University ofCalifornia, San Diego (strong medical school)

18. Carnegie-Mellon University (no medical school)

19. Northwestern University (average medical school)

20. Boston University (average medical school)

Apart from the presence of BU and the absence of Wisconsin and Illinois (and putting aside quibbles about relative placements) this is not a surprising top 20 list.  (This isn't gratuitous BU-bashing, or Wisconsin/Illinois-praising:  I'm going on other data about research and faculty quality, such as the National Research Council rankings, membership in learned societies, and the like.  By these indicators, the BU performance is completely bizarre.)

More interesting, perhaps, are the results when we look just at the "peer review" scores, i.e., the evaluations by other academics around the world.  Because the surveys were international in character, and although THES claims to have balanced the response pools by geographic region and fields, one still suspects that fields which are truly international (like the sciences and medicine) fare better than those fields which are more geographically bounded (like the humanities and many of the social sciences).  Still, the results are a lot closer to the mark in depicting research strength than the amalgmated results, above, based on the stew of incommensurable criteria.  (The one exception, again, is Boston University:  I suspect this will turn out to be this year's "big mistake," probably attributable to some error in data compilation.  The U Mass result is also somewhat surprising.)  Here are the overall peer evaluation scores for the U.S.

1.  Harvard University (100)

2.  University of California, Berkeley (95)

3.  Massachussetts Institute of Technology (84)

4.  Stanford University (78)

5.  Yale University (71)

6.  Princeton University (69)

7.  Columbia University (56)

7.  Cornell University (56)

9.  University of California, Los Angeles (52)

9.  University of Chicago (52)

11. Johns Hopkins University (50)

12. California Institute of Technology (48)

13. University of Texas, Austin (47)

14. University of Michigan, Ann Arbor (46)

15. University of California, San Diego (43)

16. University of Pennsylvania (42)

17. Boston University (41)

18. University of Illinois, Urbana-Champaign (39)

19. University of Massachussetts, Amherst (38)

20.  Duke University (36)

20. New York University (36)

20. Purdue University (36)

20. University of Wisconsin, Madison (36)

All well and good, but where the peer review results get a bit more c urious is when we start looking at the specialty areas, broken down as "Arts & Humanities," "Social Science," "Science, "Technology," and "Biomedicine."  Unfortunately, there is no indication of what disciplines fall within the scope of each of these categories.

We get some idea of the importance of strength in medicine and the biological sciences when we look at the U.S. top 20 based on peer evaluation in "Biomedicine"; note that only Berkeley, MIT, Princeton, and Cal Tech on this list lack medical schools; UC San Francisco and Baylor are devoted exclusively to medicine and the life sciences.  Schools like Hopkins and Duke get their highest scores in this category, indicating the extent to which their overall rank is driven by their outstanding medical schools.

1.  Harvard University (100)

2.  Stanford University (81)

3.  Johns Hopkins University (77.7)

4.  University of California, Berkeley (69.2)

5.  Yale University (63.9)

6.  Massachussetts Institute of Technology (60.6)

7.  University of California, San Diego (59.1)

8.  University of California, San Francisco (54.9)

9.  Duke University (52.1)

10. Columbia University (50.6)

11. Cornell University (50.5)

12. University of California, Los Angeles (44.2)

13. Princeton University (42.7)

14. California Institute of Technology (41.5)

15. Baylor College of Medicine (39.6)

16. Boston University (38.7)

17. University of Michigan, Ann Arbor (37.7)

18. University of Washington, Seattle (36.5)

19. University of Chicago (35.9)

20. University of Pennsylvania (33.8)

The U.S. top 20 in "Science" (which presumably means physics, chemistry, mathematics, and astronomy, as well as those biological sciences unrelated to medicine) is a fairly plausible list:

1.  University of California, Berkeley (92.7)

2.  Harvard University (89.9)

3.  Massachussetts Institute of Technology (87.3)

4.  Princeton University (80.4)

5.  Stanford University (79.1)

6.  California Institute of Technology (72.4)

7.  Cornell University (64.3)

8.  Yale University (60.1)

9.  University of Chicago (58.3)

10. University of California, Los Angeles (49.3)

11. University of California, Santa Barbara (46.8)

11. University of Texas, Austin (46.8)

13. University of Illinois, Urbana-Champaign (45.9)

14. Columbia University (45.8)

15. University of Michigan, Ann Arbor (39.3)

16. Johns Hopkins University (39)

17. University of California, San Diego (36.7)

18. University of Pennsylvania (33.8)

19. University of Wisconsin, Madison (31)

20. Boston University (30.9)

The top 20 U.S. schools in the "Technology" areas (presumably meaning Engineering, but perhaps not only that) show the importance of this category for schools like Carnegie-Mellon, Purdue and Texas A&M, which get their highest scores here:

1.  Massachussetts Institute of Technology (100)

2.  University of California, Berkeley (98.7)

3.  Stanford University (84.9)

4.  California Institute of Technology (78)

5.  Carnegie-Mellon University (65.8)

6.  Georgia Institute of Technology (58.7)

7.  Harvard University (58.3)

8.  University of Illinois, Urbana-Champaign (54)

9.  University of Texas, Austin (53.4)

10. Cornell University (51.5)

11. Purdue University (51.2)

12. University of California, Los Angeles (50.6)

13. Princeton University (49.8)

14. University of Massachussetts, Amherst (46.2)

15. University of Michigan, Ann Arbor (42.5)

16. University of California, San Diego (40.9)

17. Yale University (40.5)

18. University of Wisconsin, Madison (39.2)

19. Texas A&M University (38.8)

20. Boston University (37)

Here are the top 20 U.S. universities in "Arts & Humanities" based on peer evaluation according to THES:

1.  Harvard University (100)

2.  University of California, Berkeley (77.8)

3.  Yale University (77.4)

4.  Princeton University (69.2)

5.  Columbia University (56.5)

6.  Massachussetts Institute of Technology (53.5)

7.  University of Texas, Austin (50.2)

8.  Georgetown University (45.4)

9.  University of Chicago (44.1)

10. University of Michigan, Ann Arbor (43.9)

11. University of California, Los Angeles (41.5)

11. University of Pennsylvania (41.5)

13. Brown University (40.9)

14. Johns Hopkins University (38.3)

15. University of Virginia (36.7)

16. New York University (36.5)

16. University of Utah (36.5)

18. Duke University (34.3)

19. Purdue University (32.5)

19. University of Illinois, Urbana-Champaign (32.5)

19. University of New Mexico (32.5)

19. University of Wisconsin, Madison (32.5)

The top five make perfect sense, but then the list becomes, shall we say, surprising.  One wonders what disciplines are being counted?  Linguistics and Philosophy at MIT are very strong, but is Linguistics being counted here, and should two fields be sufficient?  Classics, Philosophy, Latin American History are strong at UT Austin, but is that enough to make Texas better than Chicago in "Arts & Humanities"?  Georgetown is a complete surprise.  Perhaps the emphasis is being laid on "Arts," which would give an advantage to a school like Texas with reputable programs across the arts (from Art History to Film to Music) in contrast to schools like Chicago which don't offer programs in all these fields.  In any case, it is a surprising top 20 list.  Note, too, that Cornell is nowhere to be found on this list, even though it has strong programs in most major Humanities fields.

Now take a look at the top twenty in "Social Science" according to THES's peer evaluations:

1.  Harvard University (100)

2.  University of California, Berkeley (85.3)

3.  Stanford University (80.1)

4.  Yale University (77.3)

5.  Massachussetts Institute of Technology (73.9)

6.  University of Chicago (73)

7.  Princeton University (68.9)

8.  Columbia University (63.3)

9.  Cornell University (57.1)

10. New York University (50.3)

11. University of Pennsylvania (48.6)

12. University of California, Los Angeles (47)

13. Boston University (46.1)

14. Carnegie-Mellon University (43)

15. University of Michigan, Ann Arbor (41.5)

16. Northwestern University (39.5)

17. University of Massachussetts, Amherst (39.4)

18. University of Texas, Austin (36.2)

19. Duke University (36)

19. Pennsylvania State University (36)

How can Michigan be 15th in the "Social Science" category, given its top five (by every other measure) departments in Political Science, Sociology, Psychology, and Anthropology?  Is its relative weakness in Economics dragging it down?  (The strong showing of MIT here suggests Economics is very important to the category.)  It's all a bit mysterious.  The results change a good bit if you then look at THES's citation per paper measure in Social Science; here are the top ten:

1.  University of Chicago (9.9)

2.  Carnegie-Mellon University (9.4)

2.  Harvard University (9.4)

4. Massachussetts Institute of Technology (8.9)

5.  Stanford University (8.5)

5.  University of California, San Diego (8.5)

7.  Northwestern University (8.1)

7.  University of Rochester (8.1)

9.  Duke University (8)

9.  Princeton University (8)

Michigan's position doesn't change much (it has 7.6 cites per paper), while Berkeley, with a "mere" 5.3 cites per paper (the same as UT Austin) doesn't even make the top twenty.

Only seven schools rank in the top 20 in all five areas (Harvard, Berkeley, Yale, MIT, UCLA, Princeton, and Michigan); seven additional schools rank in the top 20 in at least four areas (Stanford, Columbia, Cornell, Chicago, Penn, Texas, and, surprisingly again, BU).  Five schools are in the top 20 in at least three areas (Johns Hopkins, UC San Diego, Duke, Cal Tech, and Illinois).  Here is how these schools are evaluated in the specialty areas (they are listed alphabetically):

BOSTON UNIVERSITY

16 in Biomedicine

20 in Science

20 in Technology

Not in top 20 in Arts & Humanities

13 in Social Science

CALIFORNIA INSTITUTE OF TECHNOLOGY

14 in Biomedicine (despite not having a medical school)

6 in Science

4 in Technology

Not in top 20 in Arts & Humanities

Not in top 20 in Social Science

COLUMBIA UNIVERSITY

10 in Biomedicine

14 in Science

Not in top 20 in Technology

5 in Arts & Humanities

8 in Social Science

CORNELL UNIVERSITY

11 in Biomedicine

7 in Science

10 in Technology

Not in top 20 in Arts & Humanities (very suspicious result!)

9 in Social Science

DUKE UNIVERSITY

9 in Biomedicine

Not in top 20 in Science

Not in top 20 in Technology

18 in Arts & Humanities

19 in Social Science

HARVARD UNIVERSITY

1 in Biomedicine

2 in Science

7 in Technology

1 in Arts & Humanities

1 in Social Science

JOHNS HOPKINS UNIVERSITY

3 in Biomedicine

16 in Science

Not in top 20 in Technology

14 in Arts & Humanities

Not in top 20 in Social Science

MASSACHUSSETTS INSTITUTE OF TECHNOLOGY

6 in Biomedicine (despite not having a medical school)

3 in Science

1 in Technology

6 in Arts & Humanities

5 in Social Science

PRINCETON UNIVERSITY

13 in Biomedicine (despite not having a medical school)

4 in Science

13 in Technology

4 in Arts & Humanities

7 in Social Science

STANFORD UNIVERSITY

2 in Biomedicine

5 in Science

3 in Technology

Not in top 20 in Arts & Humanities

3 in Social Science

UNIVERSITY OF CALIFORNIA, BERKELEY

4 in Biomedicine (despite not having a medical school)

1 in Science

2 in Technology

2 in Arts & Humanities

2 in Social Science

UNIVERSITY OF CALIFORNIA, LOS ANGELES

12 in Biomedicine

10 in Science

12 in Technology

11 in Arts & Humanities

12 in Social Science

UNIVERSITY OF CALIFORNIA, SAN DIEGO

7 in Biomedicine

17 in Science

16 in Technology

Not in top 20 in Arts & Humanities

Not in top 20 in Social Science

UNIVERSITY OF CHICAGO

19 in Biomedicine

9 in Science

Not in top 20 in Technology

9 in Arts & Humanities

6 in Social Science

UNIVERSITY OF ILLINOIS, URBANA-CHAMPAIGN

Not in top 20 in Biomedicine

13 in Science

8 in Technology

19 in Arts & Humanities

Not in top 20 in Social Science

UNIVERSITY OF MICHIGAN, ANN ARBOR

17 in Biomedicine

15 in Science

15 in Technology

10 in Arts & Humanities

15 in Social Science

UNIVERSITY OF PENNSYLVANIA

20 in Biomedicine

18 in Science

Not in top 20 in Technology

11 in Arts & Humanities

11 in Social Science

UNIVERSITY OF TEXAS, AUSTIN

Not in top 20 in Biomedicine (no medical school at Austin campus)

11 in Science

9 in Technology

7 in Arts & Humanities

18 in Social Science

YALE UNIVERSITY

5 in Biomedicine

8 in Science

17 in Technology

3 in Arts & Humanities

4 in Social Science