bitter carrots*

 [DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

 

carrots

if you had told me five years ago that even one in twenty social/personality psych papers would provide links to their data and code, or to a pre-registration, i would have thought that would be huge progress.**   i've long been a fan of the nudges that encourage these kinds of practices (e.g., badges), and until recently i thought going as far as to require this kind of transparency (even with room for legitimate exceptions) was probably unrealistic - our field didn't seem ready for that.   i was sympathetic to the carrots-not-sticks approach.

 

but there's a problem with carrots-not-sticks.   we're asking researchers to eat the carrots, but some of the carrots are pretty bitter.   sometimes, when researchers are transparent, that brings information to light that undermines their claims, and readers don't buy the claims.   that's a necessary side effect of transparency.   and it means we can't in good faith tell researchers that transparency is always in their best interest and will be its own reward.   we can't lure people with carrots, and pretend all of the carrots are delicious and fun to eat.   sometimes carrots are hard to swallow.

 

i think it's time to admit that the main argument for transparency isn't self-interest - it's that transparency is just better for science.*** 

 

imagine the following scenarios:

 

scenario 1: you get a paper to review, and the authors have shared their data and code.   you look at the data and realize there is a coding error, or something else that makes the results uninterpretable (i.e., suggests the study needs to be re-run to fix the error).   you point this out in your review, the editor agrees and rejects the manuscript.

scenario 2: you get a paper to review, and the authors have shared a link to their pre-registration.   by comparing the manuscript and the pre-registration you realize that the analysis that the authors present as their planned analysis, and interpret the p-value for, is not actually the one they had specified a priori as their key planned analysis.   knowing this, you can tell that the claims in the paper are not supported by the evidence.   the editor agrees and rejects the manuscript.

 

scenarios 1 and 2 seem pretty straightforward.   but now consider scenarios 3 and 4:

 

scenario 3: you get a paper to review and the authors do not provide their data and code, but there is no sign of anything wrong.

 

scenario 4: you get a paper to review and the authors did not preregister their study, but they claim that the key result they present was their planned analysis, and interpret the p-value as if it was the only test they ran. 

 

what should you do in scenarios 3 and 4?

one option, and i think the way most of us have been operating, is to assume that the data have no anomalies, and the key analysis was indeed the one planned test that was run.   but is this fair to the authors in scenarios 1 and 2?   in scenarios 3 and 4, we're giving the authors the benefit of the doubt because they didn't let us verify their claims.   in scenarios 1 and 2 we're punishing them because they did let us verify their claims, and we learned that their claims were not justified.

 

but what else could we do in scenarios 3 and 4?   assume that their studies had the same flaws as the studies in scenarios 1 and 2?   that doesn't seem fair to the authors in scenarios 3 and 4.

 

when some authors choose to be transparent, we have no choice but to use the extra information they give us to assess the rigor of their study and the credibility of their claims.   but that also puts us in an impossible position with respect to the manuscripts in which authors are not transparent.   we can't assume these non-transparent studies have flaws, and we can't assume they don't.  

 

it seems to me the only fair thing to do is to make transparency the default.****   whenever possible, authors should be expected to share the data and code necessary to reproduce their results unless that's legally or ethically problematic.   and if authors claim that their key analysis was planned (i.e., if they're saying they're doing hypothesis testing and/or interpreting a p-value), we should ask that they document this plan (i.e., pre-register), or present their work as exploratory and their conclusions as less sure.   it's just not fair to let some authors say "trust me" when other authors are willing to say "check for yourself."

 

i know that's not the world we live in, and as long as transparency is not the default, we have to treat papers like those in scenarios 3 and 4 somewhere in the gray area between flawless and deeply flawed.   but my heart really goes out to the authors in scenarios like 1 and 2.   it would be completely rational for those authors to feel like they are paying too high a price for transparency.   (i've tried to make the case that there is a silver lining - their transparency makes the review process more fruitful for them, because it allows reviewers and editors to pinpoint specific ways they could improve their work which wouldn't have been possible without their openness.   but i'm sure that's not much consolation if they see papers like those in scenarios 3 and 4 getting published over theirs.)

 

my sense is that many people are getting on board with the credibility revolution, but only so long as all of the incentives are carrots, and not sticks.   as long as we can choose which carrots we want to go for, and not be punished if we don't eat our veggies, everyone is happy.   but that won't work in the long run.   it was perhaps a necessary step on the way to more widespread changes, but i think we need to start seriously considering making carrot-eating the default (also known as using sticks).   i can't think of how to make the current opt-in system we have fair.   if you can, i'd love to hear it.

 

* for more on problematic carrot-eating, see this by james heathers.

** in honor of the finding that two spaces after periods is the morally superior formatting, i am compensating for years of being bullied into one space by using three paces.   (yes, i'm aware of the shakiness of the evidence but i never let that get in the way of a good footnote.)   #iwantmyspacesback

*** i don't mean that everything should always be transparent, and i don't know anyone who believes that.   i mean that things that can legally and ethically be made transparent usually should be.  
****  this seems like a good time to remind readers that i do not set policy for any journals, and i am conscious of the difference between my personal fantasies and the realities of editorial responsibility (as tempting as it is to use my vast editorial powers to force everyone to put five spaces between sentences).*****   

***** editorial abuses of power is a topic for another blog post. #bringonthelawsuits

 

 

 

mmmm carrots