flip yourself - part i

a bison who recently finished her editorial term.

my mom asked me a few months ago what i was going to do once i was no longer editor in chief of a journal.  she was worried about my well-being.  “you love being an editor,” she said.  when i told her i didn’t know, she said “could you just keep reading other people’s papers and sending them your comments?”

it’s not the first time my mom’s innocent suggestion, preposterous on its face, turned out to be the answer. in high school when i wanted to quit the basketball team but both of us still wanted me to have an after school sport, my mom suggested i join the wrestling team.  after laughing at her for a day or two, i realized it was the perfect solution.  there are perks to having unconventional parents.*

for the last few months i’ve been thinking about what i’ve learned from being an editor, what i loved about it, and what i didn’t love about it.  i loved the day to day work.  the intellectual challenge, and the challenge of using my power for good.  i am sure i made mistakes, and i apologize to authors, readers, and reviewers who were affected by those mistakes.  but my mom is right – i loved it, and i am going to miss it.  i applied for another editor in chief position but didn’t get it (my vision statement for my application is here).  but at the same time that i’ve enjoyed it and would have liked to do it more, a part of me felt conflicted.  the more i saw of journals, societies, and formal peer review, the less i believed in the current system.  i believed it was worth working within the system to make it better, and i still think this is a laudable approach and am grateful to those who continue to do this kind of work.  but i’m ready to flip.  yesterday, i was working for a traditional journal.  today, i’m flipping myself: i’ll spend some of the time i was spending as editor in chief to work for open, journal-independent peer review.

what does ‘flip yourself’ mean?

open science is about making the entire research process transparent.  one part of that is open access – making published articles open for everyone to read, without paying or having a subscription.  in the open access movement, ‘flipping’ a journal is when an editorial board walks away from an existing subscription journal whose papers are behind a paywall, and starts a new journal where all the papers are openly available.

the same principles of openness, transparency, and accountability should apply to the peer review part of the research process, too.  so I started to wonder – can we flip peer review?

we can, and we don’t even have to wait for a journal or society to flip to open review.  we can all flip ourselves, by spending part of our time doing open reviews.  tools like hypothes.is and plaudit let you post comments or evaluations on papers and preprints.  instead of, or in addition to, doing reviews for commercial publishers that can’t be read by others, we can volunteer our time as open reviewers to make the content and process of peer review open for everyone to see.  in my next blog post i’ll describe in greater detail what i am imagining, and what’s in it for science.  for today, i want to talk about how my experience as editor in chief of a traditional journal led me here, to flipping myself.

lessons from my time as EiC

first, i want to say that i am extremely grateful to have had the opportunity to edit a journal.  i am lucky to have been trusted with such an important responsibility.  i admire many of the editors i’ve worked with, and i admire the ones who are going out on a limb to change the incentive structures of science.  i have tremendous respect for those who are working within the system, and i’d still be doing it, too, if i could.  but since i’m not (at least not as editor in chief), i’m going to get some things off my chest that aren’t so great about the traditional journal system. 

these reflections are based on my subjective experience – i don’t have hard evidence to back up these impressions, so these stories should be taken for what they are, a data point filtered through the lens of my quirky mind.  i would love empirical data on these issues, but it’s hard to do empirical studies of the peer review process, because of #2. 

1. concentration of power

in thinking about what i would do on july 1, when my four-year term as editor in chief at SPPS ended, it became very vivid to me how arbitrary our evaluation system is.  we treat lines on CVs as if the journal name is some kind of objective seal of approval.  but all it means is that one editor, after consulting with a few reviewers, decided they liked your paper.  given what we know about the unreliability of peer review, this is a pretty terrible way to confer a reward that can make or break a career.

the arbitrariness is especially vivid to me today.  yesterday, if i liked your paper, that could mean a new line on your CV in a pretty-well-respected journal.  today, if i like your paper, that and $3.95 will get you a scoop of ice cream.  the problem isn’t that no one cares what i think today, the problem is that what editors think matters too much, especially given how we treat publications on CVs.  we should care what many people think, and we shouldn’t give so much weight to one person’s idiosyncratic preferences.

editors have a lot of power.  i think most of them want to use it for good, but 1) not all of them have good intentions (there are corrupt editors), 2) well-meaning editors may lack an understanding of how their actions affect the incentive structure, and 3) well-meaning and well-informed editors make mistakes.  this would be ok if there was a mechanism for correcting editors’ errors, but because editors control some of the most valuable rewards in the scientific community (journal acceptances), people are understandably reluctant to challenge them. 

as one example: think about the reactions to the Reproducibility Project: Psychology (OSC, 2015).  the articles targeted for replication were from three specific journals.  yet very little of the discussion i saw about these results asked any questions about what these three journals are doing to fix the problems that the RP:P revealed.  it’s been almost four years since the publication of the RP:P – where are the calls for accountability?  there are a few exceptions, and Steve Lindsay was proactive about implementing new policies at Psych Science, but for the most part there was little attention to how the specific journals sampled have reponded.  it’s hard to speak truth to power, especially when power is so concentrated in a few prestige journals. 

2. lack of transparency

which leads to the second problem – scientists couldn’t easily hold journals and editors accountable even if they had the self-destructive will to do it.  scientists can’t see which papers were rejected and why, they can’t even see the reviews for the accepted papers at most journals, they can’t evaluate whether the review process was thorough or fair.  so how would we know if there are systematic problems with a journal’s peer review?  i know of cases where it’s an open secret that a particular editor is terrible, because enough researchers in a small community have compared notes and noticed a pattern.  it shouldn’t work that way.  when you control some of the most valuable rewards in your professional world, you should be accountable for your decisions and your process.

some journals have given authors the choice to post their reviews publicly if the paper is accepted.  very few journals require this, and very, very few journals post the reviews even for rejected papers.  an editor once told me that i shouldn’t judge his journal for rejecting a replication study of its own papers, because i didn’t see the review process.  show me the review process, then.  otherwise, what you do and don’t publish is literally the only thing i have to go on.  i agree that journals don’t give readers enough information to put them in a good position to judge the journal – that seems to be part of the point.

3. bias

it’s hard to know what counts as bias in the peer review process, because many journals and editors are pretty open about favoring papers by famous authors.  some journals actively solicit submissions from eminent people, and they don’t always indicate which articles were solicited/invited.  some editors openly state (brag?) that they take the authors’ reputation into account when evaluating submissions.  are you an ‘unproven author’?  good luck with that. 

maybe journal peer review isn’t supposed to be fair. maybe journals should be allowed to weigh factors other than the value of the manuscript.  fine – then let’s stop evaluating job candidates and promotion cases on the basis of where articles were published.  we can’t simultaneously argue that journals shouldn’t be expected to provide a level playing field, and then turn around and pretend that counting journal publications is a fair evaluation system. 

every system is biased, so accusing traditional peer review of being biased is hardly original.  the problem is that 1) we often treat a journal publication as if it very closely tracks merit, and 2) the bias in the journal system is compounded by lack of transparency and the concentration of power.  not only do some researchers get a leg up, but we can’t examine the process to look for signs of bias, and even if we could, calling out a top journal or editor would be pretty masochistic.

conclusion

it’s been a joy to be an editor.  there were a few unpleasant experiences (a story for another day), but overall i found it to be comfortable – too comfortable.  editors, journals, and societies are protected from accountability by the power they hold to make or break people’s careers, by the fact that most of their process is kept secret, and by norms that tolerate, or even encourage, a system in which the rich get richer.  surely if we were going to design a system from scratch, we could do much better.

why did i stay in the system for so long if it has so many problems?  as long as they let me have a go, i was going to try to do what i could from within, and i encourage others to do the same if they want to and have the chance.  but did i spend some of my free time fantasizing about what i would do when that option was no longer available?  absolutely.**  in part ii, i’ll describe my plans. 

* there are also downsides.  it turns out taking your cat camping is not a good idea.
** thanks, mom.