your inner third grader

 [DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]

it felt like a confessional.  

'sometimes, we say we predicted things that we didn't actually predict.'  i paused, embarrassed.  'i know.'

'i'm sorry,' she said, 'but that sounds like something even a third grader would know is wrong.'

'i know.'

i tried not to make excuses, but to explain how this happened.  how an entire field convinced itself that HARKing (Hypothesizing After the Results are Known, Kerr, 1998) is ok.

we all do it, and we all know we do it, i explained, so we basically know to take 'predictions' with a grain of salt.  what we didn't know is how much this corrupts the process of science, inflating the rate of false positives. contrary to what Bem wrote in his classic paper on how to write an empirical article, science does care whether you predicted your results.  certainly, p-values care.  

but perhaps more importantly, it's just wrong to say you predicted something when you didn't.  any third grader could tell you that.  it's true outside of science, and it's especially true in science - we have a special duty to be honest about our process. 

the fact that we could forget this (and i include myself, 100%) blows my mind.  it serves as a signal to me about how much we need to re-examine our practices.  just because everyone* thinks a research practice is perfectly fine doesn't mean it is.

Kerr's 1998 article has an astounding number of good ideas about how we got into this mess and how to get out of it.  we need to destigmatize the presentation of exploratory findings as exploratory.  we need to encourage replications.  we need to clearly distinguish between confirmatory and exploratory research.**  we need to remember our ethical commitments as scientists.  i know that last one sounds super idealistic, but i am ashamed that i convinced myself that 'telling a good story' was more important than being honest.  i know, the situation was powerful, but still.***

after my confession, this person asked me: 'why is psychology so messed up?'

that's when i knew i was going to ruin her day.  it isn't just psychology, i told her.  outcome switching (the medical equivalent of HARKing) is rampant in clinical trials,**** and probably in many sciences. psychology is probably not exceptional, except maybe in our efforts to face our problems, and do better.

i'm sure there are other ways in which we've abandoned the ideals we all held about scientists when we were third graders - the ideals we teach our undergrads.  let's listen to our inner third graders. some things we do are hard to defend, and it's tempting to say 'don't worry, that's just how science is done.' but that should be a red flag.  science means never having to say 'trust me.'

* everyone except norb kerr.

** i know it's a continuum, but because of things like HARKing, we can't have nice things for a while. i think that for now, until we've gotten over our habit of playing fast and loose with the word 'prediction', confirmatory = pre-registered. or, as i overheard at SIPS: let's put the 'pre' back in prediction.

*** i hope the situationists will let me into their club, cause i'm definitely with them on this one.  the problem - and the solution - is not individual researchers but the system in which they operate. change the system (journals, hiring committees, award committees, etc.) and individual researchers will change. then again, norb kerr was subject to the same situational pressures, so there's that.  damnit, i liked that club.

**** ok, so pre-registration isn't going to help much if people are just going to lie about that, too. but at least with pre-registration, everyone can see that you HARKed. and since pre-registration is relatively new to psychology, we can start with a clean slate and firmly establish the norm that lying about one's pre-registration should be considered fraud.*****

***** this whole post is starting to sound pretty preachy, despite the fact that i drank a generous shot of whiskey before writing it.  i predict that whiskey is not causally linked to decreased self-righteousness.  

Comments

[Roger Giner-Sorolla]

I call this the "Uncle Fred" ethical standard. "Huh? You only report the studies that work??"

[Alex Garinther]

Hey, a few of us got together and talked -- you're allowed in.
-the situationists