one of the most fascinating dimensions along which psychology researchers differ is in their reaction to counterintuitive findings. when some people say 'i can't believe it!' they mean it as a compliment. when others say it, you should take cover. how should we feel about counterintuitive findings?
i'll come out and say it: i have not drunk the bayesian kool-aid. i do like the idea that the amount of evidence required to support a claim should depend on the plausibility of that claim to begin with, but the reason i'm not a whole-hearted bayesian is that i am skeptical that there will be much consensus in psychology about which claims are more or less probable. (have you ever asked a group of psychologists what proportion of our a priori hypotheses are likely to be right? you should try it, it's a fun party trick.) but i have seen cases where pretty much everyone agrees that a conclusion is very counterintuitive (in fact, there are quite a few cases where the authors themselves appeal to this as a selling point of their work). and in those cases we can ask: given that we all agree this is surprising, should we hold the research to a higher standard? do the authors need more evidence if the claim they are making is widely acknowledged to be shocking?
bayesians would say yes (spoiler: i agree with them). but let's explore the counterargument. the counterargument, as far as i can tell, goes something like this: counterintuitive findings do more to advance our knowledge because they force us to rethink our old theories. in kuhnian language,* these are the anomalies that lead to a crisis in the paradigm, and sometimes force scientific revolutions. findings that are intuitive, or fit with the existing paradigm, can add more bricks to the wall but will never lead to huge leaps in knowledge.
sounds good. i agree that counterintuitive findings can be breakthroughs. but if they cause so much upheaval, isn't that all the more reason to hold them to a higher evidentiary standard? if they have the potential to undo entire paradigms, don't we want to be super sure about them?
as an editor, i am torn about what to do when handling a manuscript with very counterintuitive findings. i would like to require more evidence than for more intuitive findings. but several factors make me hesitate. first, i don't want to put much weight on what i personally find counterintuitive. (my intuitions are not exactly normal. for example, i have the strong intuition that an avocado-raspberry smoothie would be delicious.** luckily i don't go near kitchen equipment so there is little danger of me finding out.) second, even if the authors and reviewers agree that the finding is counterintuitive, it is hard to know how much evidence is enough. twice as much as for an intuitive finding? three times?
so, in conclusion, i do think that counterintuitive findings should be held to a higher standard, but it's hard to know for sure which findings are counterintuitive, and how much higher the standard should be.
i bet you're glad i settled that one.
*i attended a great lecture on kuhn by one of my colleagues a few weeks ago. i have not actually read kuhn. and while i'm confessing, i don't know much about bayesian statistics, either.
**it's possible i've spent too much time in california.