bitter carrots*

 [DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

 

carrots

if you had told me five years ago that even one in twenty social/personality psych papers would provide links to their data and code, or to a pre-registration, i would have thought that would be huge progress.**   i've long been a fan of the nudges that encourage these kinds of practices (e.g., badges), and until recently i thought going as far as to require this kind of transparency (even with room for legitimate exceptions) was probably unrealistic - our field didn't seem ready for that.   i was sympathetic to the carrots-not-sticks approach.

 

but there's a problem with carrots-not-sticks.   we're asking researchers to eat the carrots, but some of the carrots are pretty bitter.   sometimes, when researchers are transparent, that brings information to light that undermines their claims, and readers don't buy the claims.   that's a necessary side effect of transparency.   and it means we can't in good faith tell researchers that transparency is always in their best interest and will be its own reward.   we can't lure people with carrots, and pretend all of the carrots are delicious and fun to eat.   sometimes carrots are hard to swallow.

 

i think it's time to admit that the main argument for transparency isn't self-interest - it's that transparency is just better for science.*** 

 

imagine the following scenarios:

 

scenario 1: you get a paper to review, and the authors have shared their data and code.   you look at the data and realize there is a coding error, or something else that makes the results uninterpretable (i.e., suggests the study needs to be re-run to fix the error).   you point this out in your review, the editor agrees and rejects the manuscript.

scenario 2: you get a paper to review, and the authors have shared a link to their pre-registration.   by comparing the manuscript and the pre-registration you realize that the analysis that the authors present as their planned analysis, and interpret the p-value for, is not actually the one they had specified a priori as their key planned analysis.   knowing this, you can tell that the claims in the paper are not supported by the evidence.   the editor agrees and rejects the manuscript.

 

scenarios 1 and 2 seem pretty straightforward.   but now consider scenarios 3 and 4:

 

scenario 3: you get a paper to review and the authors do not provide their data and code, but there is no sign of anything wrong.

 

scenario 4: you get a paper to review and the authors did not preregister their study, but they claim that the key result they present was their planned analysis, and interpret the p-value as if it was the only test they ran. 

 

what should you do in scenarios 3 and 4?

one option, and i think the way most of us have been operating, is to assume that the data have no anomalies, and the key analysis was indeed the one planned test that was run.   but is this fair to the authors in scenarios 1 and 2?   in scenarios 3 and 4, we're giving the authors the benefit of the doubt because they didn't let us verify their claims.   in scenarios 1 and 2 we're punishing them because they did let us verify their claims, and we learned that their claims were not justified.

 

but what else could we do in scenarios 3 and 4?   assume that their studies had the same flaws as the studies in scenarios 1 and 2?   that doesn't seem fair to the authors in scenarios 3 and 4.

 

when some authors choose to be transparent, we have no choice but to use the extra information they give us to assess the rigor of their study and the credibility of their claims.   but that also puts us in an impossible position with respect to the manuscripts in which authors are not transparent.   we can't assume these non-transparent studies have flaws, and we can't assume they don't.  

 

it seems to me the only fair thing to do is to make transparency the default.****   whenever possible, authors should be expected to share the data and code necessary to reproduce their results unless that's legally or ethically problematic.   and if authors claim that their key analysis was planned (i.e., if they're saying they're doing hypothesis testing and/or interpreting a p-value), we should ask that they document this plan (i.e., pre-register), or present their work as exploratory and their conclusions as less sure.   it's just not fair to let some authors say "trust me" when other authors are willing to say "check for yourself."

 

i know that's not the world we live in, and as long as transparency is not the default, we have to treat papers like those in scenarios 3 and 4 somewhere in the gray area between flawless and deeply flawed.   but my heart really goes out to the authors in scenarios like 1 and 2.   it would be completely rational for those authors to feel like they are paying too high a price for transparency.   (i've tried to make the case that there is a silver lining - their transparency makes the review process more fruitful for them, because it allows reviewers and editors to pinpoint specific ways they could improve their work which wouldn't have been possible without their openness.   but i'm sure that's not much consolation if they see papers like those in scenarios 3 and 4 getting published over theirs.)

 

my sense is that many people are getting on board with the credibility revolution, but only so long as all of the incentives are carrots, and not sticks.   as long as we can choose which carrots we want to go for, and not be punished if we don't eat our veggies, everyone is happy.   but that won't work in the long run.   it was perhaps a necessary step on the way to more widespread changes, but i think we need to start seriously considering making carrot-eating the default (also known as using sticks).   i can't think of how to make the current opt-in system we have fair.   if you can, i'd love to hear it.

 

* for more on problematic carrot-eating, see this by james heathers.

** in honor of the finding that two spaces after periods is the morally superior formatting, i am compensating for years of being bullied into one space by using three paces.   (yes, i'm aware of the shakiness of the evidence but i never let that get in the way of a good footnote.)   #iwantmyspacesback

*** i don't mean that everything should always be transparent, and i don't know anyone who believes that.   i mean that things that can legally and ethically be made transparent usually should be.  
****  this seems like a good time to remind readers that i do not set policy for any journals, and i am conscious of the difference between my personal fantasies and the realities of editorial responsibility (as tempting as it is to use my vast editorial powers to force everyone to put five spaces between sentences).*****   

***** editorial abuses of power is a topic for another blog post. #bringonthelawsuits

 

 

 

mmmm carrots

Comments

[Brett Buttliere]

The carrots just need to be big enough.

Or and Also people are voting by their behavior. The people will not accept something that is forced upon them, if they don't see the value in it. Even speeding tickets -- people, in general, see the value in not speeding too much.

As I said back in 2014 - the solutions need to be so good and so obviously useful and apparent that people want to adopt them themselves -- nothing else will work. That is how innovation is adopted (e.g., Facebook, Cell Phones, Cars) rarely is an innovation put in place permanently where it makes life harder or more frustrating. It just doesn't last.

https://www.frontiersin.org/articles/10.3389/fncom.2014.00082/full

[Anonymous]

Regarding your 4 scenarios, i think the first two will not happen a lot. The last two will probably happen more, of which i find the 4th scenario the most troubling.

To me, however that one is easily solved: if after all that happened in the last decades researchers still think it's just fine to label certain analyses as "confirmatory" without giving the reader access to the pre-registration information via a 10-15 character link in the paper, i'd say they don't deserve the reader's trust. I sincerely hope "confirmatory analyses" will not simply become the new "we hypothesized that" (cf. John et al., 2012)

Anyway, i am more interested in your "let's make everything open" default and/or how to reward those researchers that are transparent and follow good practices. I think it's hard for journals to "demand" such a thing at this point in time, but perhaps there is an intermediate step/option that i have not heard (enough) about by influential voices (like yourself) in the open science/psychology community: the option of small groups of collaborating researchers.

I reason those researchers that want to perform and publish research with some higher standards (e.g. highly powered, pre-registered, replicating, interpreting null-results, optimally (re-) formulating and testing theories, etc.) are way more dependent on the “input” of their studies compared to those that don’t adhere to these higher standards, and are way more "vulnarable" in the possible current academic system.

I reason it could be very useful for those researchers working on a similar topic/phenomenon/theory and wanting to perform and publish research with some higher standards, if they would cooporate in small groups, e.g. via StudySwap and a format/idea i wrote down here:

http://andrewgelman.com/2017/12/17/stranger-than-fiction/#comment-628652

[Anonymous]

"I reason it could be very useful for those researchers working on a similar topic/phenomenon/theory and wanting to perform and publish research with some higher standards, if they would cooporate in small groups, e.g. via StudySwap and a format/idea i wrote down here:

http://andrewgelman.com/2017/12/17/stranger-than-fiction/#comment-628652"

Ow, and this format, and StudySwap, also doesn't need several "associate directors" and "committees" like the Psychological Science Accelerator apparently does (like it's a giant big firm or something).

Just one of many possible benefits compared to, for instance, the Psychological Science Accelerator. See:
http://andrewgelman.com/2018/05/08/what-killed-alchemy/#comment-728195

[Carl]

"but what else could we do in scenarios 3 and 4? assume that their studies had the same flaws as the studies in scenarios 1 and 2? that doesn't seem fair to the authors in scenarios 3 and 4...we can't assume these non-transparent studies have flaws, and we can't assume they don't. "

With a base rate of non-reproducible p-hacking over 50%, aren't small sample studies without pre-registration or other anti-hacking assurances usually flawed along those lines?

The evidential value of the generic study (with uncertain p-hacking levels) to a research consumer is far less than for the transparent study, so why not review it accordingly?

[Anonymous]

"The evidential value of the generic study (with uncertain p-hacking levels) to a research consumer is far less than for the transparent study, so why not review it accordingly?"

I agree: transparency should be judged differently, and be rewarded. However, i fear most journals really don't care about that. If i am not mistaken the current editor-in-chief of the journal "Psychological Science" has the following to say about pre-registration:

"Just because a study is preregistered does not, in my view, mean that its results warrant submission, let alone publication."

&

"Just because a study was preregistered does not mean that the work was worth doing or informative. It is quite easy to preregister an ill-conceived study."

Source: http://andrewgelman.com/2018/04/15/fixing-reproducibility-crisis-openness-increasing-sample-size-preregistration-not-enuf/#comment-712159

So, there you have it perhaps: for some editors/journals transparency and evidential value may come in 2nd, 3rd, 4th, or 5th place...