another $*%#! blog post about repligate*

because i can't very well have a blog about scientific integrity and not blog about this incident, some thoughts:

1. there are assholes on both 'sides.'**  of everything.  there are assholes everywhere. there is probably one under your chair right now.

2. most people on both sides are not assholes.  in this case, none of the main characters are assholes.  the record shows that those with significant roles in the special issue (original authors, replicators, editors) have tried very hard to be fair and balanced.  this is impressive.  i admire all of them for their dedication to the scientific process.  they all got more than they bargained for, and we owe them our thanks. thank you.
   

3. this incident is a kind of projective test***.  almost**** any reaction is justifiable - there is a lot to react to. 

4. i cannot help but see this incident through the lens of self-knowledge. this is an opportunity for self-reflection.  it is tempting to go with our knee-jerk reactions, but what is more challenging, and will be more fruitful, is to force ourselves to acknowledge that the 'other side' has valid points.  here are what i think are the most important points we on the side of All Things That Are Good and True***** would do well to consider.

a. in this case, the original authors should have been guaranteed the chance to write a commentary from the start.  i understand the reasons why they weren't, but i think this was a mistake.  i am not saying this should always be the case going forward, but it should have been the case here.

 

b. people care about their results.  they believe in them and are passionate about the ideas behind them.  when we talk about the possibility that those results or ideas are wrong, we should not take this lightly.  it is not a small thing to the people who have invested a lot of time and energy in the research. (note: i am not talking about the johnson et al. paper - they did not take things lightly).

 

c.  there is currently not much daylight between saying an original finding was a false positive and accusing the original researchers of having p-hacked.  in the current climate, they are almost the same thing (though of course they shouldn't be).  and even though we all p-hacked******, it is very different to single out one finding for having been p-hacked than to say we all did it.  there is no way around singling out individual findings if we are to pursue replications, and eventually we will need to find ways to make this process less personal, but for now, calling false positive on a finding is personal, and it is essentially the same as insinuating p-hacking.   (again, not talking about the johnson et al. paper, they did not conclude that the original finding was a false positive).

 

5. these observations lead to some difficult questions that i don't have the answers to:

a.  should people be passionate about their results?  does the fact that we are personally invested in our results make us less scientific?  is it plausible to ask people to be passionate about their questions but not their answers?

b.  what should we do if we accumulate strong evidence that an original finding was a false positive?  is there any way to publicly come to this conclusion without damaging the original authors' reputation? 

right now it is easy, and correct, to say that false positives are part of life and shouldn't harm anyone's reputation.  and of course even if we eliminate all p-hacking, qrps, publication bias, and world hunger, there will still be completely innocent false positives.  but as our techniques for detecting p-hacking get more and more sophisticated, and p-hacking becomes less and less forgivable, things are going to get dicier in terms of what counts as a personal attack, what should harm one's reputation, etc.  it's naive to think that there are easy, black and white answers to these questions.

 

personally, i favor some kind of amnesty for all papers published up until some date in the near past or future (like, for example, today.  6.3.14 - the day p-hacking became history*******). i don't mean amnesty from replication attempts, but amnesty from blame, or something like that.  if we agree not to hold people accountable for normal p-hacking behavior in the past, maybe this will make things less treacherous for authors and replicators.  if we agree to hold people accountable going forward, this would require making sure that all new researchers (and old ones) are educated about the new standards. 

i also favor putting a lot of energy into de-stigmatizing past less-than-ideal behavior, and encouraging future good behavior.  this means working to change the incentive structure.  a lot.  every day.  e.g., every time you review a paper, think about what kinds of practices you are rewarding/punishing.  it also means making heroes out of people who engage in adverserial collaborations to seek the truth about their own published work - e.g., this.

 

6.  dan gilbert should apologize.

* before this was a meme on twitter, it was a phrase used by john doris in his forthcoming book, which is where i stole it from.  there are many other good things to steal there: Doris, J. M. (forthcoming). Talking to Our Selves: Reflection, Skepticism, and Agency. Oxford: Oxford University Press.
** trust me, i'm an expert on assholes.
*** comparing anything to a projective test is usually an insult in my book.  i do not mean to demean this incident by comparing it to a form of assessment that mostly lacks any validity.  sorry incident.
****  see #6.
***** too soon?
****** i p-hacked.  sometimes when i am tired or hungry, i still catch myself p-hacking.  this is why i sleep nine hours a day and always keep tostitos in my office.

******* also my brother's birthday.  navid, i hope you like null findings.