unbelievable.

one of the most fascinating dimensions along which psychology researchers differ is in their reaction to counterintuitive findings.  when some people say 'i can't believe it!' they mean it as a compliment.  when others say it, you should take cover.  how should we feel about counterintuitive findings?

i'll come out and say it: i have not drunk the bayesian kool-aid. i do like the idea that the amount of evidence required to support a claim should depend on the plausibility of that claim to begin with, but the reason i'm not a whole-hearted bayesian is that i am skeptical that there will be much consensus in psychology about which claims are more or less probable.  (have you ever asked a group of psychologists what proportion of our a priori hypotheses are likely to be right? you should try it, it's a fun party trick.) but i have seen cases where pretty much everyone agrees that a conclusion is very counterintuitive (in fact, there are quite a few cases where the authors themselves appeal to this as a selling point of their work).  and in those cases we can ask: given that we all agree this is surprising, should we hold the research to a higher standard? do the authors need more evidence if the claim they are making is widely acknowledged to be shocking?

bayesians would say yes (spoiler: i agree with them). but let's explore the counterargument. the counterargument, as far as i can tell, goes something like this: counterintuitive findings do more to advance our knowledge because they force us to rethink our old theories. in kuhnian language,* these are the anomalies that lead to a crisis in the paradigm, and sometimes force scientific revolutions. findings that are intuitive, or fit with the existing paradigm, can add more bricks to the wall but will never lead to huge leaps in knowledge.

sounds good.  i agree that counterintuitive findings can be breakthroughs. but if they cause so much upheaval, isn't that all the more reason to hold them to a higher evidentiary standard? if they have the potential to undo entire paradigms, don't we want to be super sure about them?

as an editor, i am torn about what to do when handling a manuscript with very counterintuitive findings.  i would like to require more evidence than for more intuitive findings.  but several factors make me hesitate. first, i don't want to put much weight on what i personally find counterintuitive. (my intuitions are not exactly normal. for example, i have the strong intuition that an avocado-raspberry smoothie would be delicious.** luckily i don't go near kitchen equipment so there is little danger of me finding out.)  second, even if the authors and reviewers agree that the finding is counterintuitive, it is hard to know how much evidence is enough.  twice as much as for an intuitive finding? three times? 

so, in conclusion, i do think that counterintuitive findings should be held to a higher standard, but it's hard to know for sure which findings are counterintuitive, and how much higher the standard should be. 

i bet you're glad i settled that one.

*i attended a great lecture on kuhn by one of my colleagues a few weeks ago.  i have not actually read kuhn. and while i'm confessing, i don't know much about bayesian statistics, either.

**it's possible i've spent too much time in california.

Comments

[Weskaggs]

I'm as Bayesian as they come, but I would argue that there is a difference between the evidence required to publish and the evidence required to change one's views. It takes more evidence to convince me of a counterintuitive claim than of a plausible claim, but that doesn't mean it should take more evidence to publish a paper. I don't generally put massive weight on a single publication even if the claims that it makes are highly plausible. A single paper with findings at the p<0.001 level might not be enough to convince me, but if I see a steady accumulation of papers at that level, it's a different story.

Regards, Bill Skaggs

[Jennifer T]

I think it's important to differentiate findings that are counterintuitive from findings that are paradigm-shifting. It’s possible to be in favor of the latter, but not the former. There are a lot of potential problems with counterintuitive findings, including the perverse incentive structure that's been built up around sound-bite research in psychology. There's also the problem that they often go against the very principles that make us scientists - you know, being reasonable, logical, and thoughtful. There is a good reason we shouldn't *expect* crazy things to happen. There is also a good reason we should be concerned if crazy things start coming out of our work at greater than chance levels (Keep Calm and Read Meehl). These are all reasons why I think the prioritization and overvaluation of counterintuitive findings in psychology has become a real problem.

That’s not the same thing as devaluing paradigm-shifting research, though. I think it’s possible to see paradigm-shifting research come about via processes we all find logical and reasonable. Let’s think about the derivation of my favorite formula, Euler’s formula (because who can pass up a chance to talk about that!). When Euler derived this formula, he supposedly claimed that it “proved the existence of God”. How’s that for paradigm shifting? It is remarkably parsimonious and elegant yet bridges complex concepts (transcendental numbers and trigonometric functions). But it’s completely reasonable, and logical, and follows clear pathways in its identification. So, why can’t psychology have paradigm shifts like that? There’s no reason to jump ahead, to the crazy punchlines, and then try to piece together a posthoc pathway that appeals to logic. That’s not paradigm-shifting, in my opinion (at least not in a good way!).

I think it’s possible to stay the logical course, to be thoughtful, put bricks in the wall, and still have the possibility of paradigm-shifting work emerging from that. It certainly isn’t easy, and the vast majority of us will not shift paradigms in our lifetime. But it shouldn’t be easy! If we look at the number of labs/researchers producing counterintuitive findings, do we really believe that many people are producing meaningful scientific paradigm shifts in psychology? If we follow Kuhn’s ideas further, then we probably shouldn’t anticipate even knowing who is producing real paradigm shifting work in our lifetime. That knowledge will likely only come later. It’s like the ultimate marshmallow test for psychological scientists. Who is willing to take the boring and tedious path toward greater potential scientific payoff?

[Roger Giner-Sorolla]

For me it's clear: to be useful, a counterintuitive finding must illuminate and support a theory, as Festinger's $1/$20 findings did.

And, counter whose intuitions? If lay beliefs about our own psychology are tested and found wanting, this is surely as interesting as finding they have merit, although laypeople may find it more "mind-blowing" etc.

It is the *non*-intuitive proposition sans theory that should be entered into warily ... ideas like "sweet tastes make people more ambitious," or whatever. If it fails to be supported, you have zero story, and not even a well-known theory to start debunking with a solid null result. Therefore it is more risky to do and report research on such topics. Perhaps this accounts for the skepticism and suspicions of moral hazard surrounding such research, as well.

[Jdottan]

I liked the post (and the comments above), just have a small piece to add:

It is possible (not always) to set a prior empirically, as demonstrated in my pet favorite recent Bayesian psychology paper: http://onlinelibrary.wiley.com/doi/10.1111/cdev.12169/full

[PsychChrisNave]

Interesting post, Simine. Two points: 1) Why not go Milgram and ask a series of experts on a topic (or maybe the general public?) to make predictions on what we would expect from a proposed study- why not move the topic from a fun party trick to collecting survey data on what people expect to find? I suppose you could pre-register the hypothesis or competing hypotheses and add more objectivity to what truly is counterintuitive. The implications of our work may be put in context and adjusted by how “counterintuitive” our results are.

2) One way of looking at “counterintuitive” could be from an empirical standpoint- what flies in the face of past findings of psychological phenomena. Meta-analysis can be useful in creating a Bayesian prior or threshold one might have for evaluating new findings in the context of what we know from past findings. Before meta-analysis, we had to rely on lit reviews and have a rough qualitative understanding of what evidence we have for an effect. I’m cautiously optimistic that with the sophistication of meta-analytic techniques and with increased dissemination of knowledge (e.g. online databases, OSF), we can empower journal editors, reviewers and the general public to make an educated assessment of what evidence is needed to challenge or “undo” past findings (I realize we still have file-drawer issues but this is not new- Rosenthal and others have come up with fail-safe N and other ways to estimate the amount of studies needed to overturn an effect decades ago. I’m sure meta-analysts are coming up with new and improved ways for accounting for file-drawer issues). Malle’s 2006 meta-analysis showing no actor-observer effects can help inform future studies on the phenomenon (interestingly- the actor-observer effect is still largely taught in Social Psychology as a counterintuitive effect—so counterintuitive there is perhaps no evidence of its existence!).

I’m not suggesting we hold counterintuitive findings to a greater threshold at the publication or dissemination level if there is full transparency of the methodology, large N, rigorous analytic strategies and cautious interpretations of the implications of the work. Raising the bar too high for counter-intuitive findings to be published gets us back into making silly, arbitrary decision rules like overly stringent alphas/corrections or over-concerning ourselves with Type I error at the expense of Type II. We need to be more honest in our work being exploratory (Sanjay Srivastava eloquently makes this point about work being ground-breaking OR definitive- see link below), be careful in the extrapolations we make from our work and understand that many of our initial conclusions on human behavior may end up being wrong (While we are at it, let’s continue to de-stigmatize the fact that great, high quality research can and will be “wrong” after replications are performed- that failure to replicate has nothing to do with fraud or poor research design).

Sanjay’s blog/article:
http://spsptalks.wordpress.com/2011/12/31/groundbreaking-or-definitive-journals-need-to-pick-one/

[Daniel Lakens]

If you want a little sip of the Bayesian kool-aid, and know best case scenario's for the p-values you'd need, depending on the prior probability you come up with subjectively, see: http://daniellakens.blogspot.nl/2014/05/prior-probabilities-and-replicating.html