Brian Leiter's Law School Reports

Brian Leiter
University of Chicago Law School

A Member of the Law Professor Blogs Network

Monday, June 4, 2007

Citation Study Methodology

A colleague at DePaul made the helpful suggestion that I should describe the methodology for the citation study for which I have been soliciting corrections to the draft faculty lists.  I shall utilize the same search methodology as in the July 2005 study.  The two differences will be that (1) we shall search every non-emeritus tenure-stream member of the academic faculty, in order to secure the per capita rate of citation; and (2) the searches will be confined to literature in the databases that has appeared from 2000 to the present.  (We will also need to employ a discount factor, since the data collection will take several days, during which time the database may increase in size.  We will use the citation total for Cass Sunstein as the measure of how much the database increases during the period when the data is being collected, since he is the most frequently cited legal scholar in the U.S.) 

I would be pleased if schools want to submit the results of self-studies as a check on our process.  The data will probably be collected in late June or in early July.

I would welcome constructive feedback aimed at improving the method; I have received lots of enormously helpful feedback over the years.  But please, if you are going to comment, try to do so in a way that is not transparently self-serving; I have already received too much of that, and it is tiresome.  Think of this from the standpoint of:  what is the most effective and efficient way to measure the scholarly impact of a law faculty, taking into account the limitations of the available databases?  Examples of questions on which I am especially interested in hearing people's views are:   (1) whether there are better databases than Westlaw's JLR database or whether there are good databases to supplement the JLR database; (2) whether most clinical faculty are now expected to produce scholarship as a major portion of their duties (this bears on the question of whether it is fair to include clinical faculty in the study--so far, I have only heard from clinical faculty seeking inclusion; I have yet to hear from any Deans or non-clinical faculty making the case that their clinical colleagues should be included).

Do not report that the Westlaw JLR database does not include some journals in which law professors publish.  Everyone knows that (it is certainly true in my own fields).  We are looking at per capita impact of entire faculties; unless there is some reason to think that the gaps in the database will produce systematic advantages or disadvantages, the fact that it does not include your favorite journal is neither here nor there.  (It does not include the journal I edit, Legal Theory, which may not be my favorite journal, but it's one I like!!!) 

Only signed, verifiable comments will stand a chance of being published.  (Even though I wrote this last time, some people submitted anonymous comments.  None of those appeared, needless to say.)  I will approve those comments which make a substantive point that is actually relevant to the issues noted above.  Irrational and self-serving rants, like Professor Neumann's, can go elsewhere.  (I have been fortunate to receive some quite informative e-mails from clinical faculty, and I would like to hear from more.)

http://leiterlawschool.typepad.com/leiter/2007/06/citation_study_.html

Rankings | Permalink

TrackBack URL for this entry:

http://www.typepad.com/services/trackback/6a00d8341c659b53ef00df351fcb6c8834

Listed below are links to weblogs that reference Citation Study Methodology:

Comments

I think if you supplement with the legal periodicals in JSTOR, you'll do better than with just JLR. And I would guess that there's probably some systematic bias, otherwise, away from interdisciplinary scholarship of various kinds. I'm not sure what that would do for any particular faculty, though.

Posted by: Margo Schlanger | Jun 4, 2007 5:57:41 PM

I welcome the opportunity to comment on the methodology of this study. I want to start, however, by saying what probably goes without saying, that scholarship alone is far from all that matters to law faculty, clinical or nonclinical, and that quantitative measures of scholarship are far from perfect measures of the value of what we write. That said, I agree that one important thing one can usefully assess about law faculties is the impact of their scholarship among other scholars, and that one meaningful way to do that is quantitatively, and I’m writing to offer reasons why the scholarship of clinical scholars should be included in this survey. (I think that the same approach would be appropriate for legal writing teachers, but I’m much more familiar with the work of clinical teachers, and I focus on them here. For what it’s worth, I personally have taught both clinically and nonclinically over the years.)

Professor Leiter suggested in a post on June 3 that “[s]ince clinical and legal writing faculty do not, typically, have the same obligations to produce scholarship that tenure stream academic faculty have, it seemed unfair in a study of per capita impact to include those faculty and then ‘evaluate’ them by reference to a criterion that isn't always apt for what they do.” This proposition seems to me to overstate the differences between clinical and nonclinical faculty. First, many clinicians are in fact members of the tenured faculty at their respective schools, and probably virtually every tenured clinician has been required to meet a scholarship requirement as part of earning tenure. Second, many other clinicians, who hold either clinical tenure or long-term contracts, have probably also earned that status by meeting requirements that included a scholarship criterion. Third, even clinicians who did not have to write in order to obtain job security may in fact be writing, and may be encouraged and supported in writing by their respective schools.

It would probably be unrealistic, however, to say that clinical faculty are typically engaged in traditional scholarship to the same extent as nonclinicians. I think it is fair to say that, in general, teaching is more important to the work and recognition of clinicians than it is to nonclinical faculty; indeed, the commitment to teaching is part of what clinicians are rightly proud of. This doesn’t mean that there aren’t very productive clinical scholars – there certainly are – and for that matter it doesn’t mean there aren’t classroom teachers who approach their teaching with the utmost dedication and intensity – again, there certainly are – but there is a difference in emphasis between clinicians and nonclinicians and it probably does mean that, on average, nonclinical faculty are more focused on producing traditional scholarship than clinical faculty are, and more likely to write more of it.

But there are two ways that this difference is misleading. The first is that while the average nonclinical faculty member may write more than the average clinical teacher, scholarship is nevertheless an important part of the professional life and responsibility of a great many clinicians. The sheer size of the clinical bibliography most recently revised by Professor J.P. (Sandy) Ogilvy of Catholic University reflects how vigorous clinical scholarship in fact is. Not considering citations to their scholarship in this survey will therefore necessarily omit the impact of a substantial amount of writing by people who count (and whose schools count) writing as one of the responsibilities of their work.

Second, the decision whether to require or encourage scholarship by clinicians is a policy decision that schools and their faculties make. It's certainly possible to debate the merits of such decisions, just as it's possible to debate the wisdom of many other choices (e.g., re course load) which schools make that have an impact on their scholarly productivity. But a school that opts to foster scholarship by clinicians is likely a school that believes such scholarship matters.

How does it matter? Perhaps it simply makes clinicians more scholarly, which might be a good thing for various reasons. Hopefully it contributes to the pursuit of truth, for example about the characteristic clinical topics of how to practice law ethically and effectively, and how to teach students to do so. But clinical scholarship may also matter because schools hope this scholarship will be read and that their reputation for fostering such scholarship will contribute to their institutional academic reputations.

I should add that I don't think this institutional strategy is hypothetical. To name just one example, the clinical scholars at UCLA have produced a body of work that is particularly extensive. A lot of people have read and/or taught from the clinical scholarship produced at UCLA. I take it that that school’s support for clinical scholarship is very much a product of the decisions taken by its faculty over the years, and that part of what makes UCLA an admired school is the contribution its clinical faculty have made to the understanding of clinical issues around the country. Several other schools among the top 49 listed by Professor Leiter also have notably scholarly clinical faculties, and may well have made this same kind of deliberate institutional choice and earned the same sort of institutional regard as a result. Others may have made the opposite choice. A survey of scholarship that didn’t count citations to the scholarship of clinicians would necessarily miss the impact of such choices, and deny those schools the opportunity to have the impact of this strategy measured.

In sum, it is probably true that a citation survey of only non-clinical faculty would measure the impact of the work of those who are, in general, the most likely to be the most active authors -- though not without some striking omissions and no doubt with the inclusion of a number of very unproductive classroom teachers. But such a survey would miss not only the work of a sizable number of people who are genuinely and productively committed to scholarship, by reason of professional obligation and personal inclination, but also the impact of the institutional decisions that have concentrated these people at some schools rather than others.

It seems fair to add that the task of separating clinical and nonclinical faculty is itself so difficult that undertaking it is bound to generate errors along the way. Many clinicians, of course, have titles that are identical to those of their nonclinical colleagues, and so they cannot be distinguished by title. Many also cannot be distinguished by contractual status, since they hold full, regular tenure. In addition, many clinicians do not teach only clinically. Suppose a clinical professor also teaches Civil Procedure, and so his or her teaching time is 1/4 clinical, 3/4 nonclinical. Would this professor count as a clinician? What if his or her teaching load was half clinical and half nonclinical? Or 3/4 nonclinical and 1/4 clinical? Or suppose that the professor in question is, initially, a nonclinical faculty member, but over time comes to spend a portion of his or her time teaching in a clinic? Which of these people (and I think there are a lot of people who fit one or another of these models) would count as clinicians, and why? I think that figuring out the right time-share definition of "clinician" won't be simple, but it’s important also to keep in mind that formulating the definition may well be easier than collecting the data with which to apply it.

One other anomaly bears mentioning: a failure to count citations to scholarship by clinicians will mean that citations to otherwise comparable works of scholarship, perhaps appearing in the pages of the same law review issue, will be counted, or omitted, depending solely on the identity of their author. An issue of the Journal of Legal Education, for example, might include articles on law school pedagogy by both clinical and nonclinical faculty – but only citations to those written by the latter group would be counted. Clinicians, it should be noted, write on a great many topics; a symposium on torts, for instance, might feature articles by clinicians and nonclinicians, but again only those written by nonclinicians would be accounted for. Or to take one more example, there may be instances where a clinician from one school and a nonclinician from another school co-author a piece (I’m not speaking hypothetically, since I know of a book that fits this description precisely); citations to the book, or article, would be credited to the nonclinician and his or her school, but not to the clinician and the school he or she taught at.

On all these grounds, it seems to me that the better course is to include clinicians who are on tenure or long-term-contract tracks, and find out what the overall scholarly impact of their schools -- including these faculty as well as their classroom counterparts -- may be.

Posted by: Stephen Ellmann | Jun 4, 2007 9:47:34 PM

I believe that, apart from fueling law porn (and who in the academy would want to deny anyone access to this?), one of the main purposes of the Leiter study is to give law school rankings the same kind of rigor that other university departments have in the NRC doctoral program rankings. For many law schools, the Leiter study is something that can be presented to a Provost, as law schools are not ranked in the NRC study that frequently holds sway with university administrators. The comparative information reflected in Leiter's study is thus a useful gauge of comparative performance, primarily of faculty in law in comparison to those in other fields at the same university, and additional information reflecting the idiosyncracies of any law school or institution can readily be added so long as the method is transparent.

I would encourage you to look at the new NRC rankings to determine how they include faculty for various campus departments, as I think that it has a very specific definition of who can be included and who cannot. In addition, if you do decide to include clinicians because they have long-term contracts, you might also consider including including legal writing instructors, academic administrators and librarians with the same kind of employment terms, unless clinical scholarship is somehow distinguishable in principle from legal writing and law library scholarship. And, of course, who you include in citation and other studies may also influence who you ask to rank programs in the subjective component of your ranking.

I suppose that many take the position that "legal education is different" and that law faculty cannot fairly be compared to other university programs, but to the extent we do something "different" in legal education, to accomodate clinical and other kinds of special professional training, unless this clearly meets some analog to the NRC definition of faculty I would propose that this be reflected in an additional study that could easily be added to an NRC-type study of law schools. For many university law schools, I think that having something that is comparable to the NRC study is very important to preserving the academic and scholarly integrity of law schools and to securing resources. If you define faculty too expansively for these purposes -- for example, by making it consistent with the ABA definition of faculty rather than the NRC definition of faculty, assuming that these diverge -- you will distort, and quite possibly dilute, that function of your study.

I would also encourage you to use ISI Web of Science (Social Science Citation Index) in addition to Westlaw's JLR. SSCI is a standard method for comparing programs in the social sciences, and it would be useful to know how academic law faculty who purport to be interdiciplinary stack up to their counterparts in other disciplines. I agree that JSTOR would be great, but the lag might make the post-2000 study quite incomplete; it might be better for a historical study of law faculty citations.

Posted by: Jim Rossi | Jun 5, 2007 6:47:05 AM

Brian, a small point of correction: In your post, you say "so far, I have only heard from clinical faculty seeking inclusion; I have yet to hear from any Deans or non-clinical faculty making the case that their clinical colleagues should be included."

As to UNC, this is erroneous; I am not a member of the clinical faculty and I made the case in a comment to your earlier post that UNC's clinicians on the tenure track (or tenured) should be included. In addition, Lolly Gasaway made the same case in the same comment thread, and she is our Associate Dean.

Posted by: Eric Muller | Jun 5, 2007 10:48:54 AM

Thanks for the correction; with the volume of postings and e-mails, I have lost track of who said what when.

What does concern me is that I have heard, in e-mail correspondence, from non-clinical faculty concerned about the inclusion of clinical faculty because it will depress the per capita score of the school's faculty in the citation study. Faculty who have this concern harbor no animosity that I can detect towards their clinical colleagues; I think they are genuinely worried because their clinical colleagues are not expected to be doing a lot of writing, and so their citation counts are bound to be lower. That is consistent, I fear, with Professor Ellmann's informative remarks, above.

Posted by: Brian Leiter | Jun 5, 2007 11:21:37 AM

Is there a way to factor in years in teaching, so that you end up with something like a list of "citations per years in teaching" for all faculty members? I'm thinking that some schools may be resting on old laurels, and others may be undervalued if they've done a lot of good hiring at the junior level. I don't know whether these factors vary among schools, or whether factoring in years in teaching would create other problems. Certainly within any particular faculty, it makes more sense to compare people by a "citations per years in teaching" number than just a raw "citations" number. So it might be a good idea across faculties as well. Just a thought.

Posted by: Suzanna Sherry | Jun 5, 2007 1:43:39 PM

[BL: I HAVE INTERSPERSED SOME REPLIES TO PROFESSOR NEUMANN IN CAPITAL LETTERS; HIS CONTRIBUTIONS ARE IDENTIFIED WITH "RN".]

RN: Publication and other deadlines prevent my writing a detailed critique of the proposed methodology for this project. I’ll identify here some of the problems here.

Here are some problems with the proposed methodology for this project, nearly all of which were evident from the very first description of it posted on the blog:

1. Unverified assumptions about the population being studied: An assumption in the methodology is that clinicians and legal writing teachers are not subject to scholarship requirements. That is true at some schools and might be true at Mr. Leiter’s school, but it is not true everywhere.

BL: IN FACT, THERE WAS NO ASSUMPTION THAT CLINICIANS AND LEGAL WRITING INSTRUCTORS WERE NOT EXPECTED TO PRODUCE SCHOLARSHIP; THERE WAS NO REASON TO IMPUTE THAT ASSUMPTION BASED ON THE INITIAL POSTING, AND EVEN LESS AFTER THE REPLY I POSTED TO YOU. THE ASSUMPTION, WHICH AS FAR AS I CAN SEE IS CORRECT, WAS THAT, AS PROFESSOR ELLMAN PUTS IT, IT IS "unrealistic....to say that clinical faculty are typically engaged in traditional scholarship to the same extent as nonclinicians. I think it is fair to say that, in general, teaching is more important to the work and recognition of clinicians than it is to nonclinical faculty; indeed, the commitment to teaching is part of what clinicians are rightly proud of." THE SITUATION APPEARS TO BE WORSE IN THE CASE OF LEGAL WRITING INSTRUCTORS--AS ONE OF THEM TOLD ME, 50% OF LAW SCHOOLS DO NOT EVEN "ENCOURAGE" LEGAL WRITING FACULTY TO PUBLISH, LET ALONE REQUIRE THEM TO DO SO. AS I WROTE IN AN EARLIER ITEM ON THE BLOG: "Since clinical and legal writing faculty do not, typically, have the same obligations to produce scholarship that tenure stream academic faculty have, it seemed unfair in a study of per capita impact to include those faculty and then "evaluate" them by reference to a criterion that isn't always apt for what they do." NOTHING YOU HAVE SAID IS RESPONSIVE TO THIS POINT. (SOME OF WHAT PROFESSOR ELLMAN SAID IS RESPONSIVE, I SHOULD ADD.)

RN: A quick look at the faculty lists appended to the original blog announcement immediately revealed that a significant number of skills teachers who are subject to scholarship requirements had been deleted. Social science research makes as few assumptions as possible and verifies them thoroughly before designing the rest of a methodology. One of the things that upsets social scientists is the ease with which those requirements are dispensed with in the empirical research that law faculties generate (esp. in law and economics).

BL: IT IS PROPER, IN SOCIAL AND IN REAL SCIENCE, TO MAKE ASSUMPTIONS WARRANTED BY EVIDENCE. THAT IS WHAT IS AT ISSUE HERE. PERHAPS YOU HAVE EVIDENCE THAT CLINICAL AND LEGAL WRITING FACULTY GENERALLY HAVE OBLIGATIONS TO PROUCE SCHOLARSHIP, COMPARABLE TO THEIR ACADEMIC COLLEAGUES? SO FAR, ALL THE EVIDENCE ON OFFER POINTS IN THE OPPOSITE DIRECTION.

RN: 2. Inadequate coding and inadequate verification of coding accuracy: The proposed methodology did a review of faculties, apparently on the basis of job titles, and depended on readers of the blog to fix mistakes. The two biggest problems with this are obvious. First, job titles are inconsistent from school to school and can be misleading.

BL: IN FACT, JOB TITLES ARE STRIKINGLY CONSISTENT ACROSS FACULTIES, WHICH YOU MUST SURELY KNOW HAVING WORKED WITH THE AALS DIRECTORY IN YOUR OWN RESEARCH. THERE ARE SOME DIFFERENCES, TO BE SURE; MORE ON THOSE IN A MOMENT.

RN: The only way to avoid this is to learn each school’s personnel rules and job title practices. That takes a lot of effort, but it is necessary to satisfy the standards of social science research.

BL: NONSENSE. IT IS WHOLLY REASONABLE TO EMPLOY GENERAL CATEGORIES, EVEN IF THEY SUFFER FROM UNDER- OR OVER-INCLUSION, DEPENDING ON WHAT IT IS ONE IS TRYING TO INVESTIGATE. MORE ON THAT IN A MOMENT.

RN: Empirical research takes a huge amount of effort to avoid generating questionable data. In a study that involved a similar investigation into job status and responsibilities, the coding, for every full-time law teaching job in the country took nearly two months of professorial time while on leave and undistracted by any other work. The methodology and the coding are described at 50 J. Leg. Educ. 313, 330-332.

BL: IT IS NICE OF YOU TO CITE YOUR OWN ARTICLE ON THE STATUS OF WOMEN IN LEGAL EDUCATION, BUT WITHOUT NOTING THAT ITS AMBITIONS WERE WHOLLY DIFFERENT, AND THUS PLACED A MUCH DIFFERENT DATA COLLECTION BURDEN ON YOU. YOU CLAIM, IN THE ARTICLE, THAT YOUR DATA SHOW THAT "everywhere in legal education the line between the conventional tenure track and the lesser forms of faculty employment has become a line of gender segregation." TO ESTABLISH THAT, YOU MUST, OF COURSE, ENGAGE IN FAR MORE FINE-GRAINED JOB CODING. THE AMBITION OF MY STUDY IS MUCH SIMPLER: TO ASSESS THE PER CAPITA SCHOLARLY IMPACT OF FACULTIES AS MEASURED BY CITATIONS. IT WILL ARTIFICIALLY DEFLATE THE SCORES IF WE INCLUDE LARGE NUMBERS OF FACULTY WHOSE PRIMARY DUTIES LIE ELSEWHERE. ALL THE EVIDENCE SUGGESTS THAT CLINICAL AND LEGAL WRITING FACULTY FALL INTO THE LATTER CATEGORY; THAT IS CONSISTENT WITH THE FACT THAT THERE ARE CLINICAL AND LEGAL WRITING FACULTY WHO PRODUCE SCHOLARSHIP.

RN: The second problem is inadequate verification of the accuracy of the coding. The proposed methodology for the blog’s project simply made the coding results available and asked any reader of the blog who knew better to suggest corrections. This assumed that the regular readers of the blog, in total, would know all the details of their faculties, including exactly who has scholarship responsibilities, and would have the time and inclination to find out the facts and report them on the blog. Social science standards are not normally satisfied when coding accuracy is left to happenstance in this way.

BL: THE QUESTION ASKED WAS NOT WHO HAD SCHOLARSHIP RESPONSIBILITIES: THE QUESTION ASKED WAS TO IDENTIFY FACULTY THAT WERE WRONGLY OR RIGHTLY INCLUDED, GIVEN THE CRITERIA SPECIFIED (WHICH DID NOT INCLUDE SCHOLARSHIP RESPONSIBILITIES, BUT RATHER THINGS THAT FACULTY TYPICALLY KNOW QUITE WELL, LIKE WHO IS RETIRED, WHO IS A LEGAL WRITING INSTRUCTOR AND SO ON). IN FACT, THIS METHOD HAS ALSO WORKED QUITE WELL, HAVING IDENTIFIED SEVERAL FACULTIES WHOSE TITLES DO NOT ACCURATELY TRACK THE CLINICAL/NON-CLINICAL DISTINCTION. IN THOSE CASES, WE WILL HAVE TO TURN TO LAW SCHOOL HOMEPAGES. YOU SIMPLY ASSUME, WITHOUT EVIDENCE, THAT THE ONLY CORRECTION FOR THE CODING WAS TO BE THE BLOG. (GIVEN THE BLOG'S RATHER LARGE READERSHIP, IT WAS, BY THE WAY, QUITE REASONABLE TO THINK THAT THIS EXERCISE WOULD GENERATE HELPFUL FEEDBACK, AS IT HAS.)

RN: 3. Reliance on a deeply flawed “methodology” used by a commercial enterprise to limit the field of study: The schools on the faculty list coincide pretty closely with the top schools as reported by US News.

BL: THIS IS FALSE, UNLESS ALL THE WORK IS BEING DONE BY "PRETTY CLOSELY" SO THAT ANY OVERLAP VINDICATES YOUR POINT. IN FACT, NEARLY HALF THE FACULTIES BEING CONSIDERED FOR THE TOP 30-35 IN SCHOLARLY IMPACT ARE NOT IN THE TOP 30-35 ACCORDING TO US NEWS. MOREOVER, AS I MADE CLEAR IN EARLIER POSTINGS--WHICH YOU OBVIOUSLY DID NOT READ WITH ANY CARE--IN DEVELOPING THE LIST OF 49 WE WERE RELYING ON PRIOR CITATION STUDIES, NOT US NEWS.

RN: Assumptions about the “quality” of the institutions and people being studied: The goal appears to be to study approximately the top three-dozen law schools. One assumption here is in the definition of top. Why is one law school considered better than another?

BL: I AM SURPRISED TO LEARN THAT IT IS CONTROVERSIAL IN SOME CIRCLES TO SUGGEST THAT FACULTIES IN HIGHER EDUCATION SHOULD BE EVALUATED BASED ON THEIR SCHOLARSHIP. THERE ARE MANY WAYS TO DO THAT; ONE COMMON ONE IS THE CITATION STUDY WHICH, AS YOU MUST SURELY KNOW, IS WIDELY USED IN THE SOCIAL AND NATURAL SCIENCES.

RN: Whatever ranking is being used to identify the superior schools has assumptions in it that make the study the empirical version of circular reasoning: the assumptions produce the results.

BL: THIS IS AGAIN SHEER NONSENSE. THE RESULTS ARE PRODUCED BY THE ACTUAL PER CAPITA SCHOLARLY IMPACT OF THE FACULTIES. THE RESULT IS A MEASURE OF WHICH LAW FACULTIES HAVE THE HIGHEST RATES OF PER CAPITA SCHOLARLY IMPACT. THE *RELEVANCE* OR *INTEREST* OF THOSE RESULTS WILL DEPEND ON WHETHER ONE THINKS SCHOLARLY IMPACT IS A VALID MEASURE OF SCHOLARLY QUALITY OF FACULTIES. IN THE JULY 2005 STUDY, TO WHICH I LINKED (DID YOU READ THAT?), I NOTE MANY CAVEATS ABOUT SUCH STUDIES.

RN: Another is an assumption that breakthrough research is produced primarily by people and at institutions that are ranked highly according to conventional notions. But breakthrough research and conventional notions are nearly mutually exclusive. The greatest insights in modern physics were published in a single year by a patent clerk who had trouble studying physics in school.

BL: ONCE AGAIN, YOU HAVE INVENTED THE ASSUMPTION, AND THEN ASCRIBED IT TO ME. THE ASSUMPTION IS NO PART OF THE STUDY OR ITS RESULTS.

I ASSUME YOU DO NOT INTEND TO BE ANALOGIZING EXCLUDED CLINICAL FACULTY TO ALBERT EINSTEIN, BUT PUTTING THAT TO ONE SIDE, IT IS *NOT* THE CASE THAT ANY MAJOR DEVELOPMENT IN LEGAL SCHOLARSHIP OVER THE LAST CENTURY--E.G., LEGAL REALISM, LAW & ECONOMICS, LAW & SOCIETY, LEGAL PROCESS, ETC.--EMERGED OUTSIDE THE TOP RANKS OF AMERICAN LAW SCHOOLS. SO EVEN IF MY STUDY DEPENDED ON THIS ASSUMPTION (IT DOES NOT), IT IS MORE LIKELY TO BE WARRANTED IN THE CASE OF LEGAL ACADEMIA. [I AM OMITTING ANOTHER OF PROFESSOR NEUMANN'S EXAMPLES INVOLVING RODIN, SINCE THERE ISN'T ENOUGH TIME IN THE DAY TO RESPOND TO EVERY NON-SEQUITUR.]

RN: Finally, in both the hard and soft sciences, researchers are generally grateful for criticism of a methodology before it is used. Someone who takes the effort to do that is doing the researcher a favor, and the researcher is grateful for assistance in avoiding the generation of questionable data. What happened here was that when I suggested --€” with a sternness that is not unusual in the sciences â-- that the methodology would not satisfy applicable standards, I was called “irrational,” “self-serving,” and “rant[ing]” with a “chip on the shoulder.” This is actually pretty tame stuff, and it gave me the impression that I might enjoy an evening with Mr. Leiter sharing hard liquor. But an appropriate response would have been to ask for details, some of which I provided above.

BL: YOU MAY RATIONALIZE YOUR INITIAL RUDENESS ANY WAY YOU WANT, BUT THE FACT REMAINS THAT YOUR INITIAL CRITICISMS MISSED THEIR MARK, BECAUSE YOU WERE ILL-INFORMED ABOUT THE METHODOLOGY OR WERE UNABLE TO RATIONALLY ASSESS THE PERTINENT ASSUMPTIONS AT WORK. WHY SHOULD ONE BE GRATEFUL FOR THAT? YOU HAVE MOSTLY WASTED MY TIME.

WE HAVE HEARD A GOOD DEAL OF POSTURING ABOUT "SOCIAL SCIENCE" HERE, BUT LET ME TELL YOU SOMETHING ABOUT MY MAIN FIELD IN ADDITION TO LAW: NAMELY, PHILOSOPHY. IN PHILOSOPHY, IMPORTANT SKILLS INCLUDE BEING ABLE TO (1) READ AND UNDERSTAND AN ARGUMENT; (2) CORRECTLY IDENTIFY ITS PREMISES AND ASSUMPTIONS; AND (3) CORRECTLY IDENTIFY LOGICAL, EVIDENTIAL AND INFERENTIAL WEAKNESSES OF THE PREMISES, ASSUMPTIONS, AND ARGUMENTS. IF YOU HAD DISPLAYED MORE OF THESE SKILLS IN ANY OF YOUR INTERVENTIONS, THEN SOME OF WHAT YOU HAD TO SAY MIGHT HAVE BEEN OF GREATER VALUE.

ALL THAT BEING SAID, YOUR ORIGINAL OUTBURST HAD ONE SALUTARY EFFECT: NAMELY, I HEARD FROM SOME CLINICAL FACULTY WHO HAD PERTINENT INFORMATION. SO I THANK YOU FOR THAT.

Posted by: Richard K. Neumann, Jr. | Jun 5, 2007 1:57:02 PM

I think a very small correction in your boolean search would elevate, however little (and I leave it entirely to you to determine whether it is worth the extra typing in the searches), the accuracy of your study.

Merely typing in "Brian /2 Leiter" includes not only references to your scholarship, but also the many many "thanks to x" references that are so commonplace in legal scholarship as well as self-citations. Perhaps these are appropriate citations for your study--I also leave that to you (although it does seem that some faculty are thanked in many many articles and, in addition, those who publish frequently often cite themselves thereby increasing their overall "impact" without noting whether anyone else reads their work).

I suggest doing your searches along these lines:

((brian /2 leiter) % au(leiter) % ((thank! grat! acknow!) /s leiter)

This is a westlaw search where "%" means "not"...Thanks.

Posted by: Doug Sylvester | Jun 5, 2007 7:36:27 PM

Prof. Leiter,

I posted this on the clinic listserve and offer it for your consideration.

Friends,

I agree with Steve Ellmann, a position that has always worked for me in the past.

I want to add a little finer grained context to this discussion by noting that Prof. Leiter's study only includes a subset of law schools that he reasonably identifies as likely to rank in the study. Although there are counterexamples, I think it fair to say that the most elite schools are the least likely to have a significant number of clinicians who receive writing incentives equivalent to their non-clinical colleagues. I also note that the list of 49 excludes 4 of the schools whose clinical programs rank in the top 10 on US News, 5 of the top 15 and 8 of the top 20.

These observations lead to one small point and one larger point. The smaller point is my impression, which is testable but untested, that including whatever scholarship might otherwise be excluded would probably tend to raise scores for some of the otherwise lower scoring schools on this methodology. My hypothesis is that those otherwise lower ranking schools are more likely to have writing clinicians, but I may be wrong about that, or about how much impact there would be from that likely small difference. My own view is that a more inclusive approach would better reflect scholarship at those schools, as no effort has been made to exclude other areas or kinds of scholarship that some might marginalize, but it would cut against the desirability of consistency of methodology across time for this study.

My larger point is that this discussion should remind us that we (clinicians) mean to be change agents and we are, but there is often discomfort and challenge in change. In one light, it is a great thing that the Clinical world uses a different metric in our evaluation of legal education. In another light, it can sting when we first read that we are being excluded from a study we might think would include us. We are, and want to be, both of the academy and outside of the academy.

I confess that I read Prof. Leiter quite regularly. I also confess that I was hoping to slip by unnoticed, as my title does not reveal my clinical leanings, and let my range of second and third rate scholarship (some clinical, some not) do what it can for my school's ranking. My own situation offers a case in point with respect to the difficulty of defining who is in and who is out.

Thanks.

Ian

Ian Weinstein
Professor of Law &
Director of Clinical Legal Education
Fordham University School of Law
voice (212) 636-7066
fax (212) 636-6923
iweinstein@law.fordham.edu

See what folks are saying in the clinical world, visit lawclinic.tv
http://www.lawclinic.tv/

Posted by: Ian Weinstein | Jun 6, 2007 11:06:44 AM

Is it possible to be self-serving, yet contribute to an improved methodology? Let me try. I think you should think about including citations to Restatements for faculty that were Reporters of those Restatements. The additional effort would not be significant and would include significant contributions to legal academic research.

Posted by: Michael Green | Jun 7, 2007 1:10:20 PM

This is in response to Doug Sylvester's suggestion about a retooled Boolean search. I can see why you might want to take away self-citations, but isn't there a risk that if you remove articles where you were thanked in the asterisk footnote one will also remove articles where you were cited elsewhere in the article. I find I often want the people I cite to read my drafts to make sure I'm fair to their ideas, etc. I don't see why citations to their work should be excluded, but I do understand why one might want to exclude the "thanks" sentences. Even that one is tricky too. Imagine someone says "I am indebted to Dan Markel for this point." It's not quite the same as a citation to one's scholarship, which is the goal of this study, but it might be functionally the same in terms of how another's ideas are influential on your article. Anyway, just something to think about.

Posted by: Dan Markel | Jun 8, 2007 6:30:08 AM

Just a quick reply to the technical point made by Dan Markel...you don't lose citations to your work. Articles where the ONLY reference to your name is in the "thanks" would be excluded. If you are thanked and, indeed, cited, then that reference will still be counted. The boolean restriction merely skips over all articles you yourself have authored and individual sentences where your name appears next to "thanks etc."

As to the larger question whether acknowledgements are citations, I think they are not...even the example provided (which would be counted by the way) doesn't seem like a citation to one's work.

Posted by: Doug Sylvester | Jun 8, 2007 11:17:05 AM

Doug, thanks for the clarification. I need to go back to bool-school.

Posted by: Dan Markel | Jun 12, 2007 1:22:40 PM

When I ran Sylvester's suggested search for Leiter, i.e., ((brian /2 leiter) % au(leiter) % ((thank! grat! acknow!) /s leiter)), it appeared to fail to pick up articles that both thanked and cited him. For example, the search did not pick up Linda Hamilton Krieger & Susan T. Fiske, Behavioral Realism in Employment Discrimination Law: Implicit Bias and Disparate Treatment, 96 Cal. L. Rev. 997 (2006), which both thanks and cites Leiter. (Note that I added a right parenthetical at the very end of Sylvester's suggested search because westlaw told me I had an unmatched left parenthetical. I don't think this is significant and had the same results when I took out some parentheticals and performed the more simple search: brian /2 leiter % au(leiter) % (thank! grat! acknow!) /s leiter.)

This is consistent with the "%" function meaning "but not" and not merely "not." It doesn't just ignore the thank you reference; rather it excludes the thanking article entirely regardless of what follows.

Posted by: Gregg Polsky | Jun 13, 2007 8:40:44 AM

Gregg and Dan appear to be correct, my apologies (not sure the "but not" or "not" explains it unless each is followed by an assumed, "not--this article" as opposed to this reference, but who cares).

However, it seems possible that the amount of "over-counting" that occurs by including mere acknowledgments is better than the small under-counting that may occur when excluding articles that both acknowledge and cite--but I have not done any systematic study (obviously!). In my own case, there were about 12 mere "thanks" that were counted by the "Doug! /2 Sylvester" search and only 1 that both thanked and cited. In any event, there seems no reason to include self-citations. I wonder whether a call to Westlaw (I'm not doing it!) could find a way to include any article that actually cites...

Posted by: Doug Sylvester | Jun 13, 2007 5:49:28 PM

Post a comment