some people worry that having a loud and public debate about the reproducibility of psychology findings may be detrimental to our public image. in this blog post, i make the bold argument that not only is this not what will happen, but if we have a public relations problem it's the opposite: we sometimes come across as too naive, not skeptical enough of our own preliminary results.why do i believe the replicability discussion is not going to cause harm to our reputation?i'm not generally known for my deep respect for the average person, but i do think people understand the basic concept of science - that we are getting closer and closer to the truth, but that all current knowledge is incomplete and subject to revision. in his essay The Relativity of Wrong, Asimov makes the point that science is all about becoming less and less wrong. undergoing the kind of critical self-examination psychology is currently in the midst of is a normal part of science.[for a fascinating example, see kuo's march 2014 discovery, at a 'five sigma' level of confidence (i.e., p < .0000003), that the universe expanded rapidly immediately after the big bang ('chaotic inflation theory'). when the discovery was made, the guy who had come up with the theory thirty years before (andrei linde) said "If this is true, ..." which seemed super modest to me at the time. fast forward to january 2015, and it turns out the evidence they thought they had was wrong. welcome to science. these errors don't mean we aren't hurtling towards the truth, they mean that the path is not a perfect, smooth, straight line.]i think we are successfully communicating to the world that we are having this discussion because we care about truth. the very fact that we are debating these issues, and not sweeping them under the rug, shows that we value getting it right more than portraying a glossy, airbrushed version of our science to the public.it's not news to anyone that we are a young science, and that we tackle extremely noisy phenomena. if we tried to pretend that our theories are complete, that we can predict complex human behavior with high precision, that our methods are perfect, no one would believe us anyway. if we show that we are getting better and better, and willing to revise our theories and methods when we get new, better ones, people will take us more seriously, not less.what do i think is the bigger threat to our public image?in my view, the bigger threats to people taking us seriously are the exaggerated headlines and conclusions often appearing in the media. when a preliminary study suggests an association between two variables, and that gets presented in the media as a definite and deterministic relationship, we look bad. to some extent, this is not our fault. we can be careful in our journal articles, and even in our conversations with the media, and they can still blow our claims out of proportion. but we do have some responsibility to try to correct this.first, we can be more careful about what we do write in our articles. a laundry list of caveats in the limitations section does not give us free reign to make wild speculations everywhere else in the paper. speculations have their place, but they should be clearly marked as such. and it's our responsibility to put the important limitations of our research front and center. (also, if you're tempted to say that there is 'a gene for X' or that 'brain region X lights up', read this.) (thanks sanjay.)second, we can resist the hype that reporters try to put in our mouths. some of my conversations with journalists go something like this:them: "so you're saying that people don't know themselves at all?"me: "no"them: "but people are really really bad at knowing themselves?"
me: "no"them: "but wouldn't you say that, in general, people are astonishingly bad at knowing themselves?"me: "no"**third, we can do a better job of helping science writers and readers evaluate psychological science. to that end, here are some tips for science writers and readers of science writing. when a study claims that doing X will make you smarter/happier/more popular, ask these questions:1. have there been multiple studies, ideally by different researchers, all showing the same thing?2. did the studies have a lot of participants (for social/personality psychology, at least 200***)?3. are the researchers actually studying the phenomenon they are drawing conclusions about? if not, is the phenomenon they are studying at least similar to the one they are attempting to draw conclusions about?4. if the researchers are drawing conclusions about what causes what, did they at least conduct an experiment or look at changes over time?****5. if the study is an experiment, can you rule out all likely confounds? are you pretty sure that nothing besides the experimental manipulation could have caused the differences between the control and experimental conditions?6. did the researchers use appropriate measures to quantify the variables they are studying? are they better than any other (feasible) measures you can think of? (e.g., if the researchers used self-reports, do you trust that people can report on this honestly and accurately?)7. is the statistical evidence strong? if the researchers used p-values, are they well below .05 (say, smaller than .01)?if you answered no to any of these questions, you may be reading about a preliminary study. it's totally fine to get excited about it, and if you're a science journalist, you can even write about it. just think about it (and write about it) as a thought-provoking possibility that needs to be followed-up on, not a fact.indeed, even if the answers to all the above questions are yes, it's a good idea to remind yourself (and your readers) that all psychological research findings are probabilistic - at best they tell us that variable X tends to be associated with variable Y (or, tends to cause an increase/decrease in Y), but there are many exceptions, and many other things that influence variable Y. also, remember chaotic inflation theory.knowledge ain't cheap. it should be hard, and slow, to accumulate enough evidence for an effect that we should be confident we understand what's going on, especially when we're talking about human beings interacting with each other. when the media coverage makes it look like we're discovering incredible, counterintuitive new things that deterministically predict important outcomes every other week, the public starts to think we're either dumb, or we think they're dumb. if we still don't know for sure, after years of nutrition research, whether coffee is good for you or not, how could we know for sure after one study with 45 college students whether reading about X, thinking about Y, or watching Z is going to improve your social relationships, motivation, or happiness?i don't mean to put all the blame on the media. it's partly on us to insist that journalists convey the uncertainty in our results. it's less sexy, but i suspect the public will find the more circumscribed conclusions more palatable, and will take us more seriously if they believe that we are well-calibrated. we sound more scientific, not less, when we say "this is our best guess so far, but new evidence could suggest otherwise".also, there are many good science journalists out there, conveying the messiness***** and noisiness in our research in a way that still captures the public's interest in, and respect for, what we do. let's talk to them, and hang up on the ones that want to make us look like hucksters.** there are any number of****** popular science articles about self-knowledge that were not written because of my stubbornness. i think the world will find a way to live without them.*** more or less arbitrary. but gives researchers 80% power to detect effects of r = .20 or greater .**** i don't really want to get into all the subtleties of causal reasoning (mostly because i don't think i understand them well enough). how about a massive oversimplification instead? yes? ok: in social/personality research, i believe even experiments and longitudinal designs typically don't give us super conclusive evidence about what actually causes what in the world. in the best case, they tell us that X is one of the (probably many) things that can have a causal effect on Y. probably best to keep causal claims tentative.***** ok, this article still glosses over a lot of important issues (especially causality), but it's better than most!****** at LEAST two.