Friday, April 5, 2019

A case of bad editorial practice at a journal


Professor Steven Davidoff Solomon (Berkeley) called to my attention a case of bad editorial practices, in  this instance, involving the Journal of Empirical Legal Studies, a distinguished journal in its field.   Briefly, JELS rejected an article of Professor Solomon's, sending along two referee reports; however, the editors revised one of the referee reports to make it sound less positive than it really was.  Professor Solomon discovered this because the referee had contacted him about his paper independently.   Professor Solomon submitted a "letter to the editor" of JELS about this matter, but JELS declined to break with its practice of not publishing such letters, so Professor Solomon supplied the letter to me (along with other documentation):  Download Letter to the Editor JELS.  The letter sets out the details of what transpired.

Journal editors are well within their rights to disregard the recommendations of referees or to disagree with their ultimate assessments.  Journal editors may also decide not to share referee reports with authors, or not to share them in full.  But what they should not do, out of respect for both their referees and authors, is unilaterally revise the content of a referee report to make it support their independent decision.  One hopes this is an anomalous incident.  I've opened comments here in case the editors or others wish to comment.  Comments must include a full name and a valid e-mail address, or they will not appear.

Of Academic Interest, Professional Advice | Permalink


I would suggest that Prof. M was out of line in contacting the author of the paper. I assume that JELS has a policy that the referee reports are unsigned (this is the norm in Economics and Finance). I note that I had something similar happen to me with a paper at JLS. The referee contacted me directly with a reason why my paper should be rejected, including a copy of his related working paper. When I approached the then editor consternation was obvious.
I always assumed, and properly so, that no author would know my identity when I refereed a paper. Which led to some interesting conversations with people whose papers I had panned and who then complained to me at a conference about the jerk of a referee they had on the paper. The system simply cannot work if authors can suss out who the referee is, or if referees identify themselves.

Posted by: Mark Weinstein | Apr 1, 2019 9:10:51 AM

I'm at a loss as to why the system can't work if the referee identifies himself. What difference does it make?

Posted by: Brian Leiter | Apr 1, 2019 9:34:08 AM

As described, this is just a simple little fraud: an intent to deceive SDS into believing a referee said something the referee did not say. The editor should resign.

To Mark’s point, the problem in this day and age is that the process is only single-blind. I agree with Brian that nothing in the system requires anonymity and certainly not one-sided anonymity.

Posted by: J.B. Heaton | Apr 1, 2019 10:59:38 AM

Having served as a referee for JELS in the past this is concerning. My experience was quite straightforward and the paper (which I liked) was published. But the whole point of peer review is to insure an independent perspective that can benefit the author even if the paper is not accepted.

Outside of law rejections are more frequent than acceptances and, in my experience, one learns a great deal from reviews even when, perhaps particularly when, a paper is rejected.

I prefer double blind review systems - and recent data about the exclusive nature of law review publishing suggests we ought to use peer review more widely. But this kind of incident does not help that cause.

In any case, even in a single blind system I don't see why it hurts for the referee to contact the author.

Posted by: Steve Diamond | Apr 1, 2019 11:18:45 AM

Here is the problem with referees identifying themselves to authors. It invites a negotiation between the author and the referee about what changes need to be made to the paper to make it worthy of publication. This can be a real problem if the author is junior to the referee. I know, personally, of one instance in which a junior faculty member at a well known Business School had a paper assigned by the editor to a senior colleague at that school (of course this should not have happened). Said senior colleague(?) approached junior and said, in effect, "I am the referee on this paper. I know how to make it good enough to publish and will tell you in return for co-authorship. Now, I understand that this happened even in what supposed to be a double blind system so think how much a system with single blind -- or even no blind -- could be manipulated.

Posted by: Mark Weinstein | Apr 1, 2019 3:00:50 PM

We are talking here about a referee voluntarily disclosing his identity to an author. No one is suggesting mandatory or involuntary disclosure. There is no harm in a referee voluntarily contacting the author.

Posted by: Brian | Apr 1, 2019 4:32:40 PM

Isn't the real issue here the fraud on SDS that occurred when the editor(s) decided to misrepresent what the referee said? Single- or double-blind, contact or no contact, this is really disgraceful. I've published my share of refereed articles in finance (tough place to publish) and the idea that what I saw from referees was doctored would have been infuriating. It sounds like an editor or more didn't want the paper, and that's fine. I've argued for publication as a referee and been overruled. But to pretend support in a referee report for that decision that has been falsified is certainly worthy of editorial resignation.

Posted by: J.B. Heaton | Apr 1, 2019 5:43:41 PM

Thanks for publishing this letter by Professor Solomon which demonstrates excellence in empirical studies by detecting and disproving the genuineness of the purported independent review.

Posted by: steve charnovitz | Apr 1, 2019 9:09:06 PM

I agree that the editor's actions were decidedly inappropriate. Certainly an editor can reject a paper that all the referees liked, or publish a paper that the referees did not like--that is why G-d made editors. However, I disagree with Brian's view that voluntary disclosure does not present a problem. If the system calls for double-blind then behooves referees not to identify themselves during the editorial process. I footnote in which the author thanks the "referee" whose name is then inserted by the editor in the print version is not a problem.

Posted by: Mark Weinstein | Apr 2, 2019 9:57:16 AM

I agree with Mark Weinstein. Referees should not reveal their identities during the review process. If there is another round of review, the reviewer may be reluctant to point out shortcomings in the paper if the review is no longer anonymous. There is also too much potential for collusion and corruption. In the example Mark provided above, the reviewer was leveraging their status as reviewer to try to gain coauthorship of the paper. There are many less egregious ways that a reviewer or author could corrupt the process if their identities are known to each other.

Posted by: Josh Fischman | Apr 2, 2019 2:22:54 PM

If the review process is over, there is no harm in a referee contacting an author to express their appreciation for the author's work. In this case, there was a positive benefit: it exposed, as J.B. Heaton put it, the "fraud" involved in the referee reports Professor Solomon was sent.

However, I am now declaring an end to this side debate! I welcome further comments about the main issues raised by Professor Solomon's experience.

Posted by: Brian | Apr 2, 2019 3:10:24 PM

The editors of JELS not only owe Dr Solomon an apology; they must afford him an opportunity to publish his work.

Posted by: F. E. Guerra-Pujol | Apr 4, 2019 1:20:30 PM

As a former editor, I certainly agree this is a bad practice. I never doctored a referee report. That said, I would never publish a paper that I didn't think warranted it, regardless of the referee reports (and the editors of JELS shouldn't have to either). Such a paper, however, should (in most cases) never go out to referees in the first place. If an editor knows she doesn't like a paper, she should simply desk reject it and save the author and the referees a great deal of time. In fact, because finding good referees is actually very difficult, it is very important for editors to desk reject a significant fraction of papers so that the small pool of good referees for a given paper are not overtaxed when you need them.

Posted by: abe wickelgren | Apr 5, 2019 6:16:47 AM

I've served as a referee for several European law journals, and their practice is that disclosing the referee's identity is the referee's choice... and that the reports (there is almost always more than one) go to the prospective author unaltered as attachments to the editor's cover letter stating the journal's overall evaluation, especially when reports don't entirely agree. This is consistent with my experiences in both the sciences and in literary fields. (I don't agree with Mr Wickelgren — unless a piece is rejected for failing on its face to meet guidelines, such as being a book chapter from a previously published/in-press book when the guidelines clearly state original and unpublished, it should go to the referee(s) even if the editor hates it.)

It appears that this journal is somewhat less rigorous on its own procedures. In short, this journal invites problems like this one (and, impliedly, that the editor feels free to pick and choose if there are multiple referees — arguably the worst of all worlds). This is a process problem, not just an individual-instance problem. But then, as someone with a STEM background who has participated in many study designs that led to publication in much-more-rigorous fora than this, "lack of rigor" seems all too common in "empirical legal studies," so I'm less surprised than disappointed.

Posted by: C.E. Petit | Apr 5, 2019 1:24:36 PM

Post a comment