it's the end of the world as we know it... and i feel fine

what, me worry?

several people have told me recently that they are incredibly depressed about the news from psychology (and medicine, and political science, and economics, and biology...).  which has made me wonder, why am i completely fine?

let's be real. one reason i'm not upset is because i have tenure (and a job that i love).  my heart goes out to everyone trying to navigate this brave new world without job security.

but even some of my tenured friends are depressed.  so what's my deal? maybe it's just my sunny disposition.* more likely, i think i am particularly good at focusing on the right counterfactuals.

when we learn that the evidence for one of the most well-known effects in our field (e.g., ego depletion, but it will happen to other effects as well, so pick your favorite effect and insert it here) is extremely shaky, it's natural to be sad and to think of the most obvious counterfactual: i wish the effect was true.

it's easy to feel like it was true until those goddamn replicators came along and destroyed it.  but the reality is, if it's not true, it was never true.  so, to me, the more useful counterfactual is: what if we had never discovered the flaws and continued to be pretty certain it was true?

the same goes for our research practices.  it might be tempting, when we're told that social/personality psychology research actually requires not 60 but more like 600 participants per study,** that the world just got a little bit uglier.  it might be easy to think of the counterfactual: i wish i could still do informative research with 60 participants. 

but we never could.  we just thought we could, but we were always wrong. so, the more appropriate counterfactual is: what if we had never discovered that our practices were flawed and we went on believing we could run informative studies with 60 people?  

when i think of these more relevant counterfactuals (what if we hadn't learned any of this 'bad news' and kept doing things as we had been?), i am ecstatic.  i am so glad we are figuring this out.  i am incredibly proud of our field for doing the hard and unpleasant work of questioning and correcting ourselves.  isn't it amazing that we are figuring all this out and we're doing something about it?  i feel lucky to be a part of this process, and excited that, going forward, our research is going to be more informative.  

also, i feel like it is much more exciting to be living through this period in the history of psychology than what i had originally signed up for.  when i became a researcher, i thought that, at best, i'd contribute a few findings to the literature and then i would die and the world would be exactly the same.***  but now i get to be part of something that i think will change the course of psychology.  my role is tiny, but i still think the impact of participating in these changes will be bigger than anything i had imagined when i signed up for this gig.  and that makes me happy.

sometimes i worry that my not being upset about all this comes across as glee in the face of other people's sadness.  it's not.  it's just that i can't for a second wish that we hadn't learned these lessons, that we were still in the dark.  of course some of the changes are going to be painful, and that's a blog post for another day.  but the alternative is not a world where everything was easy and our effects were robust.  that was never the reality. the only relevant counterfactual is the one where we remain ignorant.

so if you miss the old days when we could be blissfully ignorant and just keep doing our research the old way, i can't relate.  i feel so grateful that we now know what we were doing wrong, and have a chance to do better.  i don't wish we could go back to doing cheap and easy studies, because we never should've been doing them in the first place. it turns out we were not being frugal, we were being incredibly wasteful.

do i think we're stupid for having thought we could do research on the cheap?  no.  people had been screaming and shouting about statistical power for a while, but we needed the missing piece of the puzzle, p-hacking, to fully understand why power was such a big problem.  until p-hacking, the main danger of low power was type II error (false negatives), and if we were still producing plenty of significant results, then why should we worry about power?  now we know why, and we are changing our practices.  but until we knew about p-hacking and QRPs, it made sense to trust our significant results.  thank god we know now.

* ok, maybe not sunny, but at least like sitting under a nice shady tree on a warm day. with a light sweater.  or, as my grad students put it, 'very hard to annoy.' (possibly the nicest thing anyone has said about me.)

** relax.  i just picked a number that starts with 6.  it's more like 550.  

*** except the 1999 carleton college badminton barber bonanza.  i was pretty key to that.

it could be worse.

 

Comments

[Julia]

As a grad student, I have to add: I'd rather know about QRP, sample size and statistical limitations and everything else that went wrong in the past now than in 2-3 years, at the end of my thesis. it might not make it easier to design and run studies, but their contribution to science might hold up longer. I prefer contributing a little bit in high quality than something that sounds great but will never hold up. Will I be able to stay in research with fewer studies and potentially less seductive results? I don't know. As before, the only way to find out is to go ahead and try. But I will enjoy the time that I have!

[Question]

"The same goes for our research practices. it might be tempting, when we're told that social/personality psychology research actually requires not 60 but more like 600 participants per study,** that the world just got a little bit uglier. it might be easy to think of the counterfactual: i wish i could still do informative research with 60 participants. "

I am trying to figure out when a study is appropriately/highly powered, but i have a hard time doing so. I keep reading that we need bigger samples, but exact numbers or computations are often missing. Apparently looking at available effect sizes might still lead to low-powered studies because these are often inflated. So what am i to do when designing my study?

Would it be possible for someone smart to write a clear and informative article on optimal sample sizes-benchmarks for certain research designs given "all that we've learned over the past years"?

[Mark Andrews]

"people had been screaming and shouting about statistical power for a while, but we needed the missing piece of the puzzle, p-hacking, to fully understand why power was such a big problem...."

Even in the absence of p-hacking, low power is still a problem.

The probability that the null hypothesis is True, given a statistically significant effect, is dependent on the power of the test (as well as the prior probability of the null):

P(Null|Significant)
=
P(Significant|Null)P(Null)
/ (P(Significant|Null)P(Null) + P(Significant|not Null)P(not Null))

This paper explains it nicely:
http://www.nature.com/nrn/journal/v14/n5/pdf/nrn3475.pdf

[Anne]

I'm happy about living in exciting times for the same reasons as you are, and I am happy about science changing for the better.
But I still feel horrible. Just horrible. I have learned all these bitter truths (which you don't call bitter and you're probably right) and I want to incorporate them into my work to do meaningful research.
But I am a small PhD student in an environment that is convinced about n=20/cell being a really good sample size (actually it's considered borderline greedy) for extremely noisy infant research. I simply can't do it right at the moment. I don't get the time and the funding necessary to do it right. I try to compensate with Bayes factors that only tell me I don't have enough data.
I've never felt so much cognitive dissonance (if that effect still holds) in my life. I feel that what I do is really nothing but throwing taxpayer's money out the window. It's almost physically painful.

I certainly don't want my old blissful ignorance back. But I am depressed, and right now I don't see a way out. It's pretty ugly.

[Kristin Bain]

Several of my colleagues have mentioned being depressed by the replicability crisis and the current state of science but I, like you, have actually been pretty happy with the current trend. Maybe it is my sunny disposition (mostly powered by caffeine) or maybe it's the fact that I'm joining the scientific community at a time when it is fighting to become its "ideal self." I'd much rather join now, when so many are banding together to bring to light the failings of our past and communally find ways of improving, than to join while science was still proceeding as if p-hacking and all the other tricks and hoops scientists have been jumping through for years were still the norm. At least this way I get to keep my ideals and my naive belief that doing science right could still lead to a pay-off. And think of all the new stuff we get to discover because what we thought was discovered turned out to be not discovered! Personally, as a first year graduate student, I think there could be worse times for joining the scientific community that the current era. Thanks for acknowledging the positives!