don't you know who i am?

 [DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

elephant seal, throwing his weight around

when i started my first job as associate editor, i was worried that i would get a lot of complaints from disgruntled authors.  i wasn't afraid of the polite appeals based on substantive issues, i was worried about the complaints that appeal to the authors' status, the "don't you know who i am?" appeal.

 

i never did get that kind of response, at least not from authors. but i saw something worse - a pretty common attitude that we should be judging papers based, in part, on who wrote them.  socially sanctioned status bias. not so much at the journals i worked with, but in the world of journals more broadly. like the Nature editorial, on whether there should be author anonymity in peer review, that argued that "identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique)." the argument seems to be that some people should be given a chance to clear up their muddy explanations and others should not. or the editor who wrote in The Chronicle of Higher Education just a few days ago that "Editors rarely send work out to trusted reviewers if it comes from unproven authors using jazz-hands titles."  leaving aside the contentious issue of jazz-hand titles, when did we accept that it was ok to treat papers from 'unproven authors' differently?

 

once or twice, i've even been explicitly pressured to treat a manuscript differently because of the prestige of the authors or institutions they were from. that pressure didn't come from the authors, it came from someone at the journal.*  since then i've learned that this is pretty common - some journals and editors want to publish papers by famous people, because they think this will improve their impact factor (they're probably right).  in fact, some of the very top journals actively solicit manuscripts from the elite, and those are often given less scrutiny than unsolicited submissions.  

 

you probably already knew that, but i was naive and finding this out kind of blew my mind.** i thought maybe some arrogant people would think they deserve special treatment, but i didn't think the system would give it to them, and defend that practice with a straight face.  the Great Man (or Great Woman) culture is stronger than i thought.  there will probably always be this status bias for decisions like who gets to give keynotes, who gets media attention, and who is able to get money from private foundations.  but - and i know i'm too old to still be this naive -  we should eradicate this attitude from the peer review process, especially for publications (grants are more complicated because you're evaluating the person and the project, but i still think status bias is a huge problem there).  

 

the more i edit, the more i think we need to resist this Great Man culture.  who you are and what institution you're from should have no bearing on how your manuscript is handled during the review process.  there should certainly not be any explicit favoritism towards high status people, and we need to be vigilant against implicit status bias, too. 

 

for this reason (and others), i have become a big fan of blinding authors' identities during the review process.  i know it doesn't always work,*** but it's the right idea, and i can't think of any way in which it causes harm.  (the empirical evidence on blinding is completely inconclusive, so that suggests that, for now, we should decide on ethical grounds.  and the ethical considerations seem, to me, to squarely favor blinding.)

 

in fact, i'm such a fan of not knowing who the authors are when i review a paper that i have decided not to look at who the authors are when i first receive a manuscript as editor.  it's hard - it's tempting to look, and also i literally have to hold my hand up to my computer screen to hide the information from myself - but after doing this for several months, i'm convinced it has improved my editing.****

 

of course, if i decide to send a manuscript out to reviewers, at that point i need to look at who the authors are so i can make sure to avoid conflicts of interest.  however, when i am first reading a new submission and deciding whether to desk reject it or not, and when i am writing the desk rejection letter, i don't need to know who the authors are.  after writing the desk rejection letter but before sending it, i look at who the authors are, and very rarely, i find out i have a conflict of interest.  in that case, i assign it to an associate editor who does not have a conflict of interest, and let them make the decision.  otherwise i send the letter i wrote before i knew who the authors were. (i've occasionally added a sentence or two to the letter, based on my knowledge of the authors' past work, but have not changed my decision.)

 

this is not a policy at any journal i work with - no journal that i know of has a policy of blinding the editors to the authors' identities.  it's a choice i make - everyone has their own style of editing, what information they choose to look at in what order.  i choose not to know who the authors are for as long as possible.

 

this forces me to evaluate the work on its own merits.  i can't give someone the benefit of the doubt because i know they've done good work in the past, and i can't unconsciously penalize someone because i've never heard of them or their institution.  you would think that it would often be obvious who the authors are, or at least what institution they're from, but so far i've only been able to guess correctly a small fraction of the time (less than 5%).  maybe i suck at guessing, but even if someone was better at it than me, guessing is very different than knowing.  thinking that a manuscript was submitted by someone i think highly of, but not knowing for sure, makes me less likely to be influenced by that belief.

 

overall, it's kind of annoying and a lot less fun to read manuscripts without knowing who the authors are.  but there are several big advantages.  first, and most importantly, i think it's more fair to the authors.  second, it allows me to know, for sure, how i would have evaluated the manuscript if i didn't who the authors were.  it's happened to me a few times already that, upon revealing the authors' identities to myself, i was quite surprised, and had a very distinct feeling that i might have let my biases creep in if i had known their identities when reading the manuscript. (despite the fact that, of course, i try very hard not to let that influence me.  i know some social psychology.)  third, this practice makes me feel better about the necessary interconnections that naturally exist in any small field.  for example, as editors and reviewers, we often find ourselves at the same bar or dinner table as another researcher, and then the next day we receive their manuscript to handle or review.  by blinding myself to the authors' identities when i make the initial desk reject decision, i can rest easy when i socialize with researchers knowing that even if i do have to make a decision on their work the next day, it won't be personal.

 

a small tangent on conflicts of interest

another common argument against blinding authors' identities from reviewers is that this makes it harder for the reviewers to know if they have conflicts of interest. i would be more sympathetic to this concern if i saw evidence that reviewers (and editors) take these conflicts of interest seriously when they do know the authors' identities.  in my experience, people don't recuse themselves nearly as much as they should.  i know because i used to do it, too.  i believed i could be impartial (and my pattern of rejecting my friends' papers over and over again gives some plausibility to this claim*****).  but i now realize that this is not the question i should be asking myself.  the question i should be asking myself is, if the world knew that i was a reviewer or the editor for the manuscript, would they think that was unfair?  if the answer is yes, we should recuse ourselves.******

 

back to the main point

i've only rarely heard someone use the "don't you know who i am?" appeal - much more often, it's used in the third person.  as in, "we can't say no to [famous person]."  i guess i still haven't shaken the young idealist in me that fell in love with academia in part because it was supposed to reject such appeals to authority or fame.  i am not disillusioned - i think we often do a good job of evaluating research on its merits.  i also had the great privilege of having role models like Dave Kenny, Laura King, and Rich Lucas - people who taught me how to be a reviewer and editor, and who also exemplify how not to let success go to your head, who refuse to play the Great Man/Woman role.  we can all contribute to that culture by trying to combat - and protect ourselves from - our own biases. let's not confuse status with quality - the two may be correlated, but relying on that heuristic does a disservice to science.

 

* not that journal, another journal.

 

** frequent readers of this blog will notice that my mind gets blown pretty often.

 

*** you know what works even less? telling reviewers who the authors are.  if the argument against blinding authors' identities is that reviewers might guess who the authors are, how is that an argument for telling the reviewers who the authors are?

for other (in my view, pretty weak) arguments against blind review, see this: http://pps.sagepub.com/content/4/1/62.full.pdf+html.   

 

**** i also learned that if you work at cafes with your hand covering part of your computer screen, people will sit as far away from you as possible.

 

***** sorry alexa.

 

****** at SPPS the handling editor's name is now published with each manuscript, so this hypothetical is now a reality for the editors.  in fact, you'll notice that, in the july issue of SPPS, i was the handling editor on a paper with brian nosek as an author.  i assigned the manuscript to myself before brian and i really started collaborating on SIPS (and at the time i was not friends with him, except in the facebook sense*******), but even then i should've known better.  i'll try to do better in the future, and being identified on articles as handling editor will help keep me honest.  

 

******* he finally added me to his professional network on linkedin.

 

unproven authors.

 

Comments

[Wesbuc]

(the empirical evidence on blinding is completely inconclusive, so that suggests that, for now, we should decide on ethical grounds. and the ethical considerations seem, to me, to squarely favor blinding.)

Mostly agree but there are some ethical considerations on the other side. For example, take the review process in philosophy where anonymity is at least the stated norm. This has allowed a lot of mistreatment, unprofessionalism, etc, of authors that may not be as present in more open reviews. Like maybe if you had to write to an actual person norms of human decency would kick in at some point. Another potential harm might be to representation. Openness is probably the easiest way to ensure acceptable representation rates in journals, you just know who the authors are and can ensure a preestablished rate of publication among all the excellent papers vying for spots.

[David Funder]

Great commentary. I noticed you said you supported "blinding authors' identities during the review process." What about after the review process is over? It's never made sense to me that journals seem to have evolved (without every really deciding to make the change) towards concealing author identity from reviewers even after review is over. For more on this, see: https://funderstorms.wordpress.com/2014/06/09/why-i-decline-to-do-peer-reviews-part-two-eternally-masked-reviews/

[Simine Vazire]

@Wesbuc - yes, I think journals should track and pay attention to representation rates. I'm not sure this should factor into individual decisions, but a journal should be concerned if they notice disparate rates of acceptance etc. based on variables that should be orthogonal to quality of the paper (which of course is a tricky question in itself). Then the journal should try to adjust its process to try to identify the source of this bias and correct it.

@David Funder: hmm... i haven't thought about this much before. All your points seem sensible to me, and I admit that I didn't question this policy before. I'd have to think about it more to form a strong opinion but you've definitely got me thinking?

[Stephanie Preston]

THANK YOU.

[Lee Jussim]

1. Damn, Simine, this post just nails it. Are we nepotists or scientists? Social psychologists have spent decades railing against unjustified status biases (stereotypes, just world, blah blah blah), and, as you have pointed out, "We have met the enemy and they is us."

I have a chapter, maybe 20 years ago, where I pointed out that some of the best evidence for self-fulfilling prophecies comes from academia, because of exactly the malpractices you describe here.

2. One new point on signing reviews. Reviewers should not know who wrote the paper because we almost always, with rare exceptions, want the science judged on its merits, not on its authors' status.

But you convinced me a couple of years ago to sign my reviews. You were a reviewer of a paper of mine, criticized some of it, and because you signed it, I contacted your for explanation and details. You were happy to provide it -- and that was extremely helpful.

I have taken to signing my reviews ever since. I have now twice been contacted by authors in a similar vein, asking for advice and/or assistance. IDK how it will all turn out, but I did my best to provide it, and that makes scientific processes feel like the skeptical yet cooperative endeavor it should be.

Once I made the decision to sign, it also had the beneficial effect of leading me to read my reviews in an attempt to limit unnecessarily and gratuitously harsh comments.

Lee Jussim

[Roger Giner-Sorolla]

I've never heard the argument to blind from editors before but you do make an excellent point. JESP is a single-blind journal but your arguments make me want to at least discuss a change with the editorial team, and see if I can set things up to make blind editing easier (in EVISE it's not so easy as covering up with your thumb!)

If we are being game-theorists here I suppose it is in the interest of prestigious labs to unblind themselves de-facto by multiple self-citations, geographical details, etc. I wish I could say that people aren't that Machiavellian.

Conflict of interest is a difficult spectrum to slice; I draw the line at published collaborations but expect authors to bring up unpublished ones and romantic/family links. I'm not so sure about being friends; I mean, I'd like us all to be friends, and I'd like friends to be honest with each other, especially about work that could be improved. Some topics of research are bad enough with overlapping co-authorships; I mean, bad meaning good, because collaboration is great, but bad meaning bad when you have to find a reviewer without CoI.

As with so many other issues, the "active editor" policy makes friendship less of a problem. By this I mean that the editor assembles their decision from the facts brought up by the reviewers, not from their accept/reject/revise recommendations*. If you see a fatal flaw in a friend's work and you don't point it out, I don't know what to say, except at that point it's beyond self-deception. Anyway, I think attitude things like friendship, academic politics, etc. are going to show themselves more in the summary ratings than in the arguments.

* In fact, there's a case to do away with those recommendations altogether.**

** Now you've got me footnoting like David Foster Wallace!

[STeamTraen]

A limitation of withholding the authors' identity from the reviewers is that the reviewers, being human, will typically attempt to work out the authors' identities anyway. Although this process is inherently imperfect, it's quite easy to combine an 80% certainty (e.g., from looking at who is most cited in the references section) with 20% of good old-fashioned bias and process to review the article in the "knowledge" that the author is X, with whatever implications that might have for one's opinions of the quality of the work. Of course, this shouldn't happen, but as long as reviewers are human, it will, in too many cases.

[Matt_Craddock]

Earlier this year I attended a "Meet the Editors" session at a conference. The topic of double-blinding came up at some point.

One editor claimed that there was a big practical problem: in their experience, it was much harder to find reviewers when the authors identities were concealed. I don't know how true that is generally, but I can imagine that people are keener to review articles when they have heard of the authors. As you say above, somehow a little more fun when you're reviewing something by somebody whose work you've previously come and across and enjoyed (or otherwise). I was a little sympathetic towards the practical issues, but it's not really an argument against double-blinding in itself.

Another editor claimed that concealing the authors was a bad thing because author identities were actually informative. People expect work from names they know to be of a certain level of quality, that those names might produce interesting work. People might think that's bias, said the editor, but that works both ways. If you see something off in the paper, you might think hey, this author knows better than to do this, so give them a rougher ride than you might otherwise have done.

This second editor is basically admitting that the review process varies based on the identity of the authors, and claiming this is a *good* thing. I couldn't help but feel that whatever bias the editor was talking about typically works more in favour of the big name/lab than against: it's not hard to make the leap that the editor is more likely to send the article out for review in the first place if it's from a lab/name they know, and that something that doesn't sit right in the paper is glossed over because "this person/lab" knows what they're doing.

Incidentally, I disagreed with this particular editor on pretty much every topic, as I felt all the arguments they used in favour of existing processes were actually great arguments against them, which if nothing else shows opinions vary...

In any case - bias is a real problem. I don't know if double-blinding is *the* solution, but I'm inclined to side with you in thinking that it's worth trying, especially after reading about your experiences with it!

[Anonymous]

"the Great Man (or Great Woman) culture is stronger than i thought."

This possibly also goes for commenting. I recently posted anonymous comments on the APS site, concerning 2 presidential columns:

http://www.psychologicalscience.org/index.php/publications/observer/2016/sept-16/why-preregistration-makes-me-nervous.html

&

http://www.psychologicalscience.org/index.php/publications/observer/obsonline/preregistration-replication-and-nonexperimental-studies.html

I thought my anonymous comments were informational, reasonable, and fair, but they were deleted after a day or so. I wonder what reasons the APS had for doing so...

[Jenn]

totally agree with you, simine. but, given all the structural disparities that are also present in the field, it would be nice to have a way to flag relatively low status folks (e.g., new investigators without high-status co-authors/mentors), so as to adopt a more developmental role (as an editor, perhaps) with these folks who may have less experience/resources. not sure how practical this is, but it is the potential downside of a fully blind system.

[STeamTraen]

Genuine question: How do we square blinding of authors' identities with a culture of preprints? In many subfields, there's a good chance that the reviewers may already have read the manuscript at the preprint stage.

[simine vazire]

hi nick (STeamTraen) -
this is definitely something we need to figure out, and i suspect the answer will be different in different fields. right now in psychology, preprints are rare enough that i don't come across this much (but there have been exceptions). i would love for that to change (i.e., for preprints to become more common). even then, though, i'm not sure how often the editor or reviewers are likely to have come across the preprint and recognize it, or to go searching for it. maybe i'm naive and everyone will do that. in that case, i wonder if there's a way to hide the preprint authors' identities until the paper has been accepted (or maybe mask it in a way that still makes it possible for someone, e.g., members of a hiring committee, to confirm the authors' identities). or maybe there are other solutions i haven't thought of - lots of people have thought way more about these issues than i have.
for now, i find that it's pretty rare that i'm more than 50% sure who the authors are.
thanks for the comment!
simine