does psychology have a public relations problem?*

 
near holland, mi

 

some people worry that having a loud and public debate about the reproducibility of psychology findings may be detrimental to our public image. in this blog post, i make the bold argument that not only is this not what will happen, but if we have a public relations problem it's the opposite: we sometimes come across as too naive, not skeptical enough of our own preliminary results.

 

why do i believe the replicability discussion is not going to cause harm to our reputation? 

 

i'm not generally known for my deep respect for the average person, but i do think people understand the basic concept of science - that we are getting closer and closer to the truth, but that all current knowledge is incomplete and subject to revision. in his essay The Relativity of Wrong, Asimov makes the point that science is all about becoming less and less wrong. undergoing the kind of critical self-examination psychology is currently in the midst of is a normal part of science. 

 

[for a fascinating example, see kuo's march 2014 discovery, at a 'five sigma' level of confidence (i.e., p < .0000003), that the universe expanded rapidly immediately after the big bang ('chaotic inflation theory').  when the discovery was made, the guy who had come up with the theory thirty years before (andrei linde) said "If this is true, ..." which seemed super modest to me at the time.  fast forward to january 2015, and it turns out the evidence they thought they had was wrong.  welcome to science. these errors don't mean we aren't hurtling towards the truth, they mean that the path is not a perfect, smooth, straight line.]

 

i think we are successfully communicating to the world that we are having this discussion because we care about truth.  the very fact that we are debating these issues, and not sweeping them under the rug, shows that we value getting it right more than portraying a glossy, airbrushed version of our science to the public.  

 

it's not news to anyone that we are a young science, and that we tackle extremely noisy phenomena.  if we tried to pretend that our theories are complete, that we can predict complex human behavior with high precision, that our methods are perfect, no one would believe us anyway.  if we show that we are getting better and better, and willing to revise our theories and methods when we get new, better ones, people will take us more seriously, not less.

 

what do i think is the bigger threat to our public image?

 

in my view, the bigger threats to people taking us seriously are the exaggerated headlines and conclusions often appearing in the media. when a preliminary study suggests an association between two variables, and that gets presented in the media as a definite and deterministic relationship, we look bad.  to some extent, this is not our fault.  we can be careful in our journal articles, and even in our conversations with the media, and they can still blow our claims out of proportion. but we do have some responsibility to try to correct this.

 

first, we can be more careful about what we do write in our articles. a laundry list of caveats in the limitations section does not give us free reign to make wild speculations everywhere else in the paper.  speculations have their place, but they should be clearly marked as such. and it's our responsibility to put the important limitations of our research front and center.  (also, if you're tempted to say that there is 'a gene for X' or that 'brain region X lights up', read this.) (thanks sanjay.)

 

second, we can resist the hype that reporters try to put in our mouths.  some of my conversations with journalists go something like this:

them: "so you're saying that people don't know themselves at all?"

me: "no"

them: "but people are really really bad at knowing themselves?"
me: "no"

them: "but wouldn't you say that, in general, people are astonishingly bad at knowing themselves?"

me: "no"**

 

third, we can do a better job of helping science writers and readers evaluate psychological science. to that end, here are some tips for science writers and readers of science writing.  when a study claims that doing X will make you smarter/happier/more popular, ask these questions:

 

1. have there been multiple studies, ideally by different researchers, all showing the same thing?

 

2. did the studies have a lot of participants (for social/personality psychology, at least 200***)?

 

3. are the researchers actually studying the phenomenon they are drawing conclusions about?  if not, is the phenomenon they are studying at least similar to the one they are attempting to draw conclusions about?

 

4. if the researchers are drawing conclusions about what causes what, did they at least conduct an experiment or look at changes over time?****

 

5. if the study is an experiment, can you rule out all likely confounds? are you pretty sure that nothing besides the experimental manipulation could have caused the differences between the control and experimental conditions?

 

6. did the researchers use appropriate measures to quantify the variables they are studying? are they better than any other (feasible) measures you can think of? (e.g., if the researchers used self-reports, do you trust that people can report on this honestly and accurately?)  

 

7. is the statistical evidence strong?  if the researchers used p-values, are they well below .05 (say, smaller than .01)?  

 

if you answered no to any of these questions, you may be reading about a preliminary study.  it's totally fine to get excited about it, and if you're a science journalist, you can even write about it.  just think about it (and write about it) as a thought-provoking possibility that needs to be followed-up on, not a fact.  

 

indeed, even if the answers to all the above questions are yes, it's a good idea to remind yourself (and your readers) that all psychological research findings are probabilistic - at best they tell us that variable X tends to be associated with variable Y (or, tends to cause an increase/decrease in Y), but there are many exceptions, and many other things that influence variable Y.  also, remember chaotic inflation theory.

 

knowledge ain't cheap.  it should be hard, and slow, to accumulate enough evidence for an effect that we should be confident we understand what's going on, especially when we're talking about human beings interacting with each other. when the media coverage makes it look like we're discovering incredible, counterintuitive new things that deterministically predict important outcomes every other week, the public starts to think we're either dumb, or we think they're dumb.  if we still don't know for sure, after years of nutrition research, whether coffee is good for you or not, how could we know for sure after one study with 45 college students whether reading about X, thinking about Y, or watching Z is going to improve your social relationships, motivation, or happiness?

 

i don't mean to put all the blame on the media.  it's partly on us to insist that journalists convey the uncertainty in our results. it's less sexy, but i suspect the public will find the more circumscribed conclusions more palatable, and will take us more seriously if they believe that we are well-calibrated.  we sound more scientific, not less, when we say "this is our best guess so far, but new evidence could suggest otherwise". 

 

also, there are many good science journalists out there, conveying the messiness***** and noisiness in our research in a way that still captures the public's interest in, and respect for, what we do.  let's talk to them, and hang up on the ones that want to make us look like hucksters.

 

 

* ummmmmm yes.  but that's a blog post for another day.

 

** there are any number of****** popular science articles about self-knowledge that were not written because of my stubbornness.  i think the world will find a way to live without them.

 

*** more or less arbitrary. but gives researchers 80% power to detect effects of r = .20 or greater .

 

**** i don't really want to get into all the subtleties of causal reasoning (mostly because i don't think i understand them well enough).  how about a massive oversimplification instead?  yes?  ok: in social/personality research, i believe even experiments and longitudinal designs typically don't give us super conclusive evidence about what actually causes what in the world.  in the best case, they tell us that X is one of the (probably many) things that can have a causal effect on Y.  probably best to keep causal claims tentative.

 

***** ok, this article still glosses over a lot of important issues (especially causality), but it's better than most!

 

****** at LEAST two.

 

cosmic microwave background radiation (photo credit: nasa)

Comments

[Sam Schwarzkopf]

Great post. Considering that you have mentioned the cosmic background radiation result I don't know why we should single out psychology here. The media are full of doubts about science - just think of climate research or evolution. Another interesting point about the expansion of the universe is that I have a hunch that the theory is probably still true. It's just that this "smoking gun" type evidence for it didn't turn out to be.

Anyway, your conversation with the journalist sounds like interviews normally go with me as well (and then they can still somehow manage to misquote you).

You are probably right that we can trust the average person to get that science is a gradual homing in on the truth. But all to often you hear people about how science has proven this or that. I think it doesn't hurt to remind everyone (including scientists) that this isn't how it works.

[L.J Zigerell]

For what it's worth, here are questions that I ask when evaluating research:

1. Did the researchers preregister their research design choices so that we can be sure that the research design choices were not made based on the data? If not, are the research design choices consistent with the choices that the researcher has previously made in other research?

2. Have the researchers publicly posted documentation and all the data that were collected, so that other researchers can check the analysis for errors and assess the robustness of the reported results?

3. Did the researchers declare that there are no unreported file drawer studies, unreported manipulations, and unreported variables that were measured?

4. Were the data collected by an independent third party?

5. Is the sample representative of the population of interest?

[Chris C.]

Science is not "homing in on the truth," nor is it "getting closer and closer to the truth." If that were so, how would we know we were making progress?

Because we don't already know truth (if we did, why would we need science?), we cannot know if we are getting closer to it. There is no treasure map with waypoints.

Science has to be judged by its efficacy, by solving problems, by DOING things like enhancing public health, getting to the moon, decreasing suicide rates, lowering human costs, by reducing uncertainty and affording explanatory coherence.

This is not trivial. How can we judge scientific practice if we cannot measure scientific progress? Since truth is unknowable, and suggesting that we are moving toward an uncharted and unknowable destination, we must measure ourselves with a yardstick that does not rely on airy-fairy concepts.

In such a world, replication is not evidence of truth (and failure to replicate evidence of untruth), but rather it evidence of the power or effectiveness of the idea.

And so we are, as Simine suggests, working toward greater effectiveness, and the pathway is not straight. But the move toward a better position is almost always met by public setback--there is no Obergefell without DOMA and Prop. 8 first--public humiliation is almost always part of social change. It passes.

[Sam Schwarzkopf]

Chris C: Science is not "homing in on the truth," nor is it "getting closer and closer to the truth." If that were so, how would we know we were making progress?

I think this question is based on a misreading. I tend to liken scientific progress to a model fitting procedure. Essentially this is what science is, trying to come up with models to explain a chaotic world.

We know we are making progress if we increase the explanatory power of our models while minimising their complexity. No one scientist can know if their models are really getting closer to "the Truth".

Of course, sometimes any algorithm can get stuck in a local minimum that it may be hard to get out of. This means the process can go massively astray (like, say, social priming research :P). It takes some additional energy to get out of those holes. Homing in does not mean that the only direction is always downhill.

But in the long-term the process is doing circling the drain of the true explanation. If it didn't we probably shouldn't even be trying it at all.

[Andy DeSoto]

Love the post -- thanks for sharing. Not sure I agree, but I am glad that you feel that the reproducibility discussion is making us look good. "It's not news to anyone that we are a young science" -- that's interesting to me. I wonder if others would agree.

Would love to collect some data on public perceptions of psychology at some point. Thanks again for writing.

[maryjane]

informative.

[Dorcas]

I agree that "science is all about becoming less and less wrong". These errors don't mean we aren't hurtling towards the truth, they mean that the path is not a perfect, smooth, straight line. This makes our researchers to continue because we are moving on to the truth. Keep on Scientists.

[Junaid]

[Disclaimer: my response here is based on anecdotal, personal experience, and not on a validated study on the topic] This is an interesting perspective, but I feel the viewpoints presented here only really apply to scientists working in other fields, and the educated elite. My biggest issue with this post is with the main premise that "people understand the basic concept of science - that we are getting closer and closer to the truth, but that all current knowledge is incomplete and subject to revision." Nearly every conversation about science that I have with an 'average' person indicates to me that people view science as a static state of knowledge, similar to how science is taught in text books. Any aberration from this -- any time a new, splashy study contradicting a previous scientific understanding that makes it to popular media -- causes people to further distrust science.