Enter your Email:
Preview | Powered by FeedBlitz

« your inner third grader | Main | i have found the solution and it is us »

Comments

Wesbuc

(the empirical evidence on blinding is completely inconclusive, so that suggests that, for now, we should decide on ethical grounds. and the ethical considerations seem, to me, to squarely favor blinding.)

Mostly agree but there are some ethical considerations on the other side. For example, take the review process in philosophy where anonymity is at least the stated norm. This has allowed a lot of mistreatment, unprofessionalism, etc, of authors that may not be as present in more open reviews. Like maybe if you had to write to an actual person norms of human decency would kick in at some point. Another potential harm might be to representation. Openness is probably the easiest way to ensure acceptable representation rates in journals, you just know who the authors are and can ensure a preestablished rate of publication among all the excellent papers vying for spots.

David Funder

Great commentary. I noticed you said you supported "blinding authors' identities during the review process." What about after the review process is over? It's never made sense to me that journals seem to have evolved (without every really deciding to make the change) towards concealing author identity from reviewers even after review is over. For more on this, see: https://funderstorms.wordpress.com/2014/06/09/why-i-decline-to-do-peer-reviews-part-two-eternally-masked-reviews/

Simine Vazire

@Wesbuc - yes, I think journals should track and pay attention to representation rates. I'm not sure this should factor into individual decisions, but a journal should be concerned if they notice disparate rates of acceptance etc. based on variables that should be orthogonal to quality of the paper (which of course is a tricky question in itself). Then the journal should try to adjust its process to try to identify the source of this bias and correct it.

@David Funder: hmm... i haven't thought about this much before. All your points seem sensible to me, and I admit that I didn't question this policy before. I'd have to think about it more to form a strong opinion but you've definitely got me thinking?

Stephanie Preston

THANK YOU.

Lee Jussim

1. Damn, Simine, this post just nails it. Are we nepotists or scientists? Social psychologists have spent decades railing against unjustified status biases (stereotypes, just world, blah blah blah), and, as you have pointed out, "We have met the enemy and they is us."

I have a chapter, maybe 20 years ago, where I pointed out that some of the best evidence for self-fulfilling prophecies comes from academia, because of exactly the malpractices you describe here.

2. One new point on signing reviews. Reviewers should not know who wrote the paper because we almost always, with rare exceptions, want the science judged on its merits, not on its authors' status.

But you convinced me a couple of years ago to sign my reviews. You were a reviewer of a paper of mine, criticized some of it, and because you signed it, I contacted your for explanation and details. You were happy to provide it -- and that was extremely helpful.

I have taken to signing my reviews ever since. I have now twice been contacted by authors in a similar vein, asking for advice and/or assistance. IDK how it will all turn out, but I did my best to provide it, and that makes scientific processes feel like the skeptical yet cooperative endeavor it should be.

Once I made the decision to sign, it also had the beneficial effect of leading me to read my reviews in an attempt to limit unnecessarily and gratuitously harsh comments.

Lee Jussim

Roger Giner-Sorolla

I've never heard the argument to blind from editors before but you do make an excellent point. JESP is a single-blind journal but your arguments make me want to at least discuss a change with the editorial team, and see if I can set things up to make blind editing easier (in EVISE it's not so easy as covering up with your thumb!)

If we are being game-theorists here I suppose it is in the interest of prestigious labs to unblind themselves de-facto by multiple self-citations, geographical details, etc. I wish I could say that people aren't that Machiavellian.

Conflict of interest is a difficult spectrum to slice; I draw the line at published collaborations but expect authors to bring up unpublished ones and romantic/family links. I'm not so sure about being friends; I mean, I'd like us all to be friends, and I'd like friends to be honest with each other, especially about work that could be improved. Some topics of research are bad enough with overlapping co-authorships; I mean, bad meaning good, because collaboration is great, but bad meaning bad when you have to find a reviewer without CoI.

As with so many other issues, the "active editor" policy makes friendship less of a problem. By this I mean that the editor assembles their decision from the facts brought up by the reviewers, not from their accept/reject/revise recommendations*. If you see a fatal flaw in a friend's work and you don't point it out, I don't know what to say, except at that point it's beyond self-deception. Anyway, I think attitude things like friendship, academic politics, etc. are going to show themselves more in the summary ratings than in the arguments.

* In fact, there's a case to do away with those recommendations altogether.**

** Now you've got me footnoting like David Foster Wallace!

STeamTraen

A limitation of withholding the authors' identity from the reviewers is that the reviewers, being human, will typically attempt to work out the authors' identities anyway. Although this process is inherently imperfect, it's quite easy to combine an 80% certainty (e.g., from looking at who is most cited in the references section) with 20% of good old-fashioned bias and process to review the article in the "knowledge" that the author is X, with whatever implications that might have for one's opinions of the quality of the work. Of course, this shouldn't happen, but as long as reviewers are human, it will, in too many cases.

Matt_Craddock

Earlier this year I attended a "Meet the Editors" session at a conference. The topic of double-blinding came up at some point.

One editor claimed that there was a big practical problem: in their experience, it was much harder to find reviewers when the authors identities were concealed. I don't know how true that is generally, but I can imagine that people are keener to review articles when they have heard of the authors. As you say above, somehow a little more fun when you're reviewing something by somebody whose work you've previously come and across and enjoyed (or otherwise). I was a little sympathetic towards the practical issues, but it's not really an argument against double-blinding in itself.

Another editor claimed that concealing the authors was a bad thing because author identities were actually informative. People expect work from names they know to be of a certain level of quality, that those names might produce interesting work. People might think that's bias, said the editor, but that works both ways. If you see something off in the paper, you might think hey, this author knows better than to do this, so give them a rougher ride than you might otherwise have done.

This second editor is basically admitting that the review process varies based on the identity of the authors, and claiming this is a *good* thing. I couldn't help but feel that whatever bias the editor was talking about typically works more in favour of the big name/lab than against: it's not hard to make the leap that the editor is more likely to send the article out for review in the first place if it's from a lab/name they know, and that something that doesn't sit right in the paper is glossed over because "this person/lab" knows what they're doing.

Incidentally, I disagreed with this particular editor on pretty much every topic, as I felt all the arguments they used in favour of existing processes were actually great arguments against them, which if nothing else shows opinions vary...

In any case - bias is a real problem. I don't know if double-blinding is *the* solution, but I'm inclined to side with you in thinking that it's worth trying, especially after reading about your experiences with it!

Anonymous

"the Great Man (or Great Woman) culture is stronger than i thought."

This possibly also goes for commenting. I recently posted anonymous comments on the APS site, concerning 2 presidential columns:

http://www.psychologicalscience.org/index.php/publications/observer/2016/sept-16/why-preregistration-makes-me-nervous.html

&

http://www.psychologicalscience.org/index.php/publications/observer/obsonline/preregistration-replication-and-nonexperimental-studies.html

I thought my anonymous comments were informational, reasonable, and fair, but they were deleted after a day or so. I wonder what reasons the APS had for doing so...

Jenn

totally agree with you, simine. but, given all the structural disparities that are also present in the field, it would be nice to have a way to flag relatively low status folks (e.g., new investigators without high-status co-authors/mentors), so as to adopt a more developmental role (as an editor, perhaps) with these folks who may have less experience/resources. not sure how practical this is, but it is the potential downside of a fully blind system.

STeamTraen

Genuine question: How do we square blinding of authors' identities with a culture of preprints? In many subfields, there's a good chance that the reviewers may already have read the manuscript at the preprint stage.

simine vazire

hi nick (STeamTraen) -
this is definitely something we need to figure out, and i suspect the answer will be different in different fields. right now in psychology, preprints are rare enough that i don't come across this much (but there have been exceptions). i would love for that to change (i.e., for preprints to become more common). even then, though, i'm not sure how often the editor or reviewers are likely to have come across the preprint and recognize it, or to go searching for it. maybe i'm naive and everyone will do that. in that case, i wonder if there's a way to hide the preprint authors' identities until the paper has been accepted (or maybe mask it in a way that still makes it possible for someone, e.g., members of a hiring committee, to confirm the authors' identities). or maybe there are other solutions i haven't thought of - lots of people have thought way more about these issues than i have.
for now, i find that it's pretty rare that i'm more than 50% sure who the authors are.
thanks for the comment!
simine

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)