several people have told me recently that they are incredibly depressed about the news from psychology (and medicine, and political science, and economics, and biology...). which has made me wonder, why am i completely fine?
let's be real. one reason i'm not upset is because i have tenure (and a job that i love). my heart goes out to everyone trying to navigate this brave new world without job security.
but even some of my tenured friends are depressed. so what's my deal? maybe it's just my sunny disposition.* more likely, i think i am particularly good at focusing on the right counterfactuals.
when we learn that the evidence for one of the most well-known effects in our field (e.g., ego depletion, but it will happen to other effects as well, so pick your favorite effect and insert it here) is extremely shaky, it's natural to be sad and to think of the most obvious counterfactual: i wish the effect was true.
it's easy to feel like it was true until those goddamn replicators came along and destroyed it. but the reality is, if it's not true, it was never true. so, to me, the more useful counterfactual is: what if we had never discovered the flaws and continued to be pretty certain it was true?
the same goes for our research practices. it might be tempting, when we're told that social/personality psychology research actually requires not 60 but more like 600 participants per study,** that the world just got a little bit uglier. it might be easy to think of the counterfactual: i wish i could still do informative research with 60 participants.
but we never could. we just thought we could, but we were always wrong. so, the more appropriate counterfactual is: what if we had never discovered that our practices were flawed and we went on believing we could run informative studies with 60 people?
when i think of these more relevant counterfactuals (what if we hadn't learned any of this 'bad news' and kept doing things as we had been?), i am ecstatic. i am so glad we are figuring this out. i am incredibly proud of our field for doing the hard and unpleasant work of questioning and correcting ourselves. isn't it amazing that we are figuring all this out and we're doing something about it? i feel lucky to be a part of this process, and excited that, going forward, our research is going to be more informative.
also, i feel like it is much more exciting to be living through this period in the history of psychology than what i had originally signed up for. when i became a researcher, i thought that, at best, i'd contribute a few findings to the literature and then i would die and the world would be exactly the same.*** but now i get to be part of something that i think will change the course of psychology. my role is tiny, but i still think the impact of participating in these changes will be bigger than anything i had imagined when i signed up for this gig. and that makes me happy.
sometimes i worry that my not being upset about all this comes across as glee in the face of other people's sadness. it's not. it's just that i can't for a second wish that we hadn't learned these lessons, that we were still in the dark. of course some of the changes are going to be painful, and that's a blog post for another day. but the alternative is not a world where everything was easy and our effects were robust. that was never the reality. the only relevant counterfactual is the one where we remain ignorant.
so if you miss the old days when we could be blissfully ignorant and just keep doing our research the old way, i can't relate. i feel so grateful that we now know what we were doing wrong, and have a chance to do better. i don't wish we could go back to doing cheap and easy studies, because we never should've been doing them in the first place. it turns out we were not being frugal, we were being incredibly wasteful.
do i think we're stupid for having thought we could do research on the cheap? no. people had been screaming and shouting about statistical power for a while, but we needed the missing piece of the puzzle, p-hacking, to fully understand why power was such a big problem. until p-hacking, the main danger of low power was type II error (false negatives), and if we were still producing plenty of significant results, then why should we worry about power? now we know why, and we are changing our practices. but until we knew about p-hacking and QRPs, it made sense to trust our significant results. thank god we know now.
* ok, maybe not sunny, but at least like sitting under a nice shady tree on a warm day. with a light sweater. or, as my grad students put it, 'very hard to annoy.' (possibly the nicest thing anyone has said about me.)
** relax. i just picked a number that starts with 6. it's more like 550.
*** except the 1999 carleton college badminton barber bonanza. i was pretty key to that.