near holland, mi
some people worry that having a loud and public debate about the reproducibility of psychology findings may be detrimental to our public image. in this blog post, i make the bold argument that not only is this not what will happen, but if we have a public relations problem it's the opposite: we sometimes come across as too naive, not skeptical enough of our own preliminary results.why do i believe the replicability discussion is not going to cause harm to our reputation?i'm not generally known for my deep respect for the average person, but i do think people understand the basic concept of science - that we are getting closer and closer to the truth, but that all current knowledge is incomplete and subject to revision. in his essay The Relativity of Wrong, Asimov makes the point that science is all about becoming less and less wrong. undergoing the kind of critical self-examination psychology is currently in the midst of is a normal part of science.[for a fascinating example, see kuo's march 2014 discovery, at a 'five sigma' level of confidence (i.e., p < .0000003), that the universe expanded rapidly immediately after the big bang ('chaotic inflation theory'). when the discovery was made, the guy who had come up with the theory thirty years before (andrei linde) said "If this is true, ..." which seemed super modest to me at the time. fast forward to january 2015, and it turns out the evidence they thought they had was wrong. welcome to science. these errors don't mean we aren't hurtling towards the truth, they mean that the path is not a perfect, smooth, straight line.]i think we are successfully communicating to the world that we are having this discussion because we care about truth. the very fact that we are debating these issues, and not sweeping them under the rug, shows that we value getting it right more than portraying a glossy, airbrushed version of our science to the public.it's not news to anyone that we are a young science, and that we tackle extremely noisy phenomena. if we tried to pretend that our theories are complete, that we can predict complex human behavior with high precision, that our methods are perfect, no one would believe us anyway. if we show that we are getting better and better, and willing to revise our theories and methods when we get new, better ones, people will take us more seriously, not less.what do i think is the bigger threat to our public image?in my view, the bigger threats to people taking us seriously are the exaggerated headlines and conclusions often appearing in the media. when a preliminary study suggests an association between two variables, and that gets presented in the media as a definite and deterministic relationship, we look bad. to some extent, this is not our fault. we can be careful in our journal articles, and even in our conversations with the media, and they can still blow our claims out of proportion. but we do have some responsibility to try to correct this.first, we can be more careful about what we do write in our articles. a laundry list of caveats in the limitations section does not give us free reign to make wild speculations everywhere else in the paper. speculations have their place, but they should be clearly marked as such. and it's our responsibility to put the important limitations of our research front and center. (also, if you're tempted to say that there is 'a gene for X' or that 'brain region X lights up', read this.) (thanks sanjay.)second, we can resist the hype that reporters try to put in our mouths. some of my conversations with journalists go something like this:them: "so you're saying that people don't know themselves at all?"me: "no"them: "but people are really really bad at knowing themselves?"
me: "no"them: "but wouldn't you say that, in general, people are astonishingly bad at knowing themselves?"me: "no"**third, we can do a better job of helping science writers and readers evaluate psychological science. to that end, here are some tips for science writers and readers of science writing. when a study claims that doing X will make you smarter/happier/more popular, ask these questions:1. have there been multiple studies, ideally by different researchers, all showing the same thing?2. did the studies have a lot of participants (for social/personality psychology, at least 200***)?3. are the researchers actually studying the phenomenon they are drawing conclusions about? if not, is the phenomenon they are studying at least similar to the one they are attempting to draw conclusions about?4. if the researchers are drawing conclusions about what causes what, did they at least conduct an experiment or look at changes over time?****5. if the study is an experiment, can you rule out all likely confounds? are you pretty sure that nothing besides the experimental manipulation could have caused the differences between the control and experimental conditions?6. did the researchers use appropriate measures to quantify the variables they are studying? are they better than any other (feasible) measures you can think of? (e.g., if the researchers used self-reports, do you trust that people can report on this honestly and accurately?)7. is the statistical evidence strong? if the researchers used p-values, are they well below .05 (say, smaller than .01)?if you answered no to any of these questions, you may be reading about a preliminary study. it's totally fine to get excited about it, and if you're a science journalist, you can even write about it. just think about it (and write about it) as a thought-provoking possibility that needs to be followed-up on, not a fact.indeed, even if the answers to all the above questions are yes, it's a good idea to remind yourself (and your readers) that all psychological research findings are probabilistic - at best they tell us that variable X tends to be associated with variable Y (or, tends to cause an increase/decrease in Y), but there are many exceptions, and many other things that influence variable Y. also, remember chaotic inflation theory.knowledge ain't cheap. it should be hard, and slow, to accumulate enough evidence for an effect that we should be confident we understand what's going on, especially when we're talking about human beings interacting with each other. when the media coverage makes it look like we're discovering incredible, counterintuitive new things that deterministically predict important outcomes every other week, the public starts to think we're either dumb, or we think they're dumb. if we still don't know for sure, after years of nutrition research, whether coffee is good for you or not, how could we know for sure after one study with 45 college students whether reading about X, thinking about Y, or watching Z is going to improve your social relationships, motivation, or happiness?i don't mean to put all the blame on the media. it's partly on us to insist that journalists convey the uncertainty in our results. it's less sexy, but i suspect the public will find the more circumscribed conclusions more palatable, and will take us more seriously if they believe that we are well-calibrated. we sound more scientific, not less, when we say "this is our best guess so far, but new evidence could suggest otherwise".also, there are many good science journalists out there, conveying the messiness***** and noisiness in our research in a way that still captures the public's interest in, and respect for, what we do. let's talk to them, and hang up on the ones that want to make us look like hucksters.** there are any number of****** popular science articles about self-knowledge that were not written because of my stubbornness. i think the world will find a way to live without them.*** more or less arbitrary. but gives researchers 80% power to detect effects of r = .20 or greater .**** i don't really want to get into all the subtleties of causal reasoning (mostly because i don't think i understand them well enough). how about a massive oversimplification instead? yes? ok: in social/personality research, i believe even experiments and longitudinal designs typically don't give us super conclusive evidence about what actually causes what in the world. in the best case, they tell us that X is one of the (probably many) things that can have a causal effect on Y. probably best to keep causal claims tentative.***** ok, this article still glosses over a lot of important issues (especially causality), but it's better than most!****** at LEAST two.
cosmic microwave background radiation (photo credit: nasa)
self-knowledge
this is what p-hacking looks like
keri russell plotting her next QRP
i am teaching a seminar called 'oh you like that finding do you? well it's probably FALSE.'
the students are a little bit shell-shocked.
i am having the time of my life.
the hardest part is explaining how perfectly smart, truth-seeking scientists can repeatedly do idiotic things. happily, i have plenty of examples from my own life.
just this week, i almost p-hacked. the details are a little dry, but i think it's worth telling this tale of mundane p-hackery, because this is what p-hacking looks like in the wild. it is not sinister and dark like a good episode of the americans. it is super boring like the stories you read to your children at night.* in fact, if you run out of bedtime material, read them this.
once upon a time, there were some personality researchers who wanted to measure personality.
they collected self-reports and six peer reports from each of their participants.
they wrote a paper examining how the big five personality traits correlate with friendship satisfaction.
because the paper was due right when the data came in, and organizing the peer reports takes time, they only used self-reports of personality (bad personality researchers!).
the reviewers and editor said 'dude, we know you have peer reports. and you shouldn't just correlate self-reports (of personality) with self-reports (of friendship satisfaction).'**
so they compiled the data and ran the analyses with the peer reports of personality. not surprisingly, all the correlations with (self-reported) friendship satisfaction were weaker. still significant (yay large samples), and with a similar pattern as the self-reports, but weaker.
they started off by reporting all of their results, with the self-reports and the peer reports side by side.
then their tables got unwieldy, and they decided that they should just aggregate the self- and peer-reports into a composite measure for each personality trait. because obviously that is the best measure of personality. and it would make their tables easier to read.***
so now, gentle reader, we have a self-report and (up to) six peer reports for each participant. we want to aggregate them. what are we to do?
option a is to first aggregate the six peer reports, and then average that composite with the self-report, in which case the self-report is weighted as much as ALL the peer reports put together.
option b is to just average all seven reports (self-report and six peer reports) all at once, in which case the self-report is weighted only as much as any single peer report.****
pop quiz hotshot: which one will give us "better" results?
option a would give us bigger effects. and many many researchers in this area***** would say option a is the better option because self-reports are special and should be weighted more than each individual peer report. so our protagonists would have cover if they went with option a.
option b, however, is what our protagonists have typically done in the past. partly because they are convention-busting mavericks and partly because they believe that the self is not that special - each of your close friends knows about as much about your personality as you do yourself.
so the noble researchers sat there and, for several minutes, contemplated what to do. 'contemplated' is the wrong word. it was immediately obvious to them what they should do. they even said 'this is p-hacking, what we're contemplating doing right now.' out loud. and still, they sat there for another minute. squirming.
then they did the right thing because that's what protagonists do and would i really be telling you this story if we p-hacked? come on.
the moral of the story is: not p-hacking is HARD. side effects include redness, swelling, and deep, deep frustration. and self-righteousness.
what's that? your kids are still awake? here is a fun little postscript:
another thing the reviewers said was 'look, i know your analyses were exploratory, but anyone with half a brain would have predicted finding ABC. stop dicking around and, in the introduction, tell your readers about how all existing personality theory and research would predict ABC.'
what are our protagonists to do? they did not predict ABC, but mostly because they were too damn lazy to make specific predictions (bad bad researchers!). they did not want to lie in the introduction. but they also agreed that only a dumbass would not predict ABC.
in a brilliant stroke of genius, they decided to write: 'previous theory and research on A and B and C would clearly suggest that ABC. although this would be a reasonable prediction, we did not actually make this prediction a priori' and they lived happily ever after.
the end.******
* i am extrapolating from the stories my dog makes me read at night. i hear parents love it when you compare their kids to dogs.
** this is called getting a taste of your own medicine. your kids will appreciate this lesson someday.
*** of course we are putting the data and disaggregated results on OSF. we're not total idiots.
**** the younger kids might get confused here. just write out the formulas for them.
***** almost all seven of them.
****** everything in this story is true (you know, without the dicking around and the dumbass bits), but in our defense, our paper (and methods) were a little more sophisticated than described here.
sweet dreams.
self-correction hurts
i know a thing or two about pain. just the other day, i was enjoying* a giants game when a foul ball landed on my knee. (i failed to catch the ball. this guy (above) ended up with it. i do, however, still have the bruise. i would post a picture of that but i don't do selfies.) some things in life just hurt. expecting them not to hurt is like expecting kale to taste good.
self-correction is one of those things. it is never going to be pleasant. whether it is in the domain of science or in the personal domain, self-improvement sucks. if we say that we should only self-correct if we can do it in a way that feels good for everyone, then we are giving up on the idea of self-correction. if we are that delicate, we are doomed.
there are more and less painful ways of correcting ourselves and each other. there are more and less strident ways to talk about replication. i am all for less stridence** and more humility. i think this is very important.
however, i don't think we can do self-correction without anyone's feelings getting hurt. i don't think it's going to feel good even if we all act as polite, humble, and professional as we absolutely ought to. as a field, we have to decide whether we can accept some amount of pain, whether we are willing to pay some price for the benefits of self-correction.
in the comments section of a recent chronicle of higher education piece on repligate, someone going by the name of bob sternberg*** wrote "we need replication but we do not need the acrimony that is going with it." i agree that we don't need some of the more personal and emotional discourse. but i think some amount of friction is unavoidable if we are going to put ourselves to the test. ideally, we will get to the point where the disagreement is all in good fun, and the pain is fleeting, and leaves fewer bruises. like a wiffle ball to the knee, perhaps. but in the meantime, i think we need to walk that very delicate line between being jerks and avoiding challenging each other altogether. and i think most of the time we are doing that quite well.
* it is possible that i was wrong and baseball is actually a real sport. just not as depicted on television.
** i'm not sure this is actually a word. it is, however, the name of my new fragrance.
*** i have no reason to think it was not actually bob sternberg but you never know. someone could be impersonating bob sternberg to make completely innocuous comments on the chronicle of higher ed website. crazier things have happened.****
**** actually, that might be crazier than anything that has happened ever.
why i study self-knowledge
someone recently asked my why i do what i do. it's easy to come up with just-so stories. for all i know, i could just as easily have ended up inventing new ice cream flavors for tara's ice cream (i still haven't ruled it out). and if i had, i could probably make up a story about why it was always meant to be. narratives are deceptively easy to construct, and amazingly convincing after the fact. it would be especially ironic for a self-knowledge researcher to deceive herself about why she studies self-knowledge. so i mostly try to resist giving an explanation. but then there's this:
this is a note from my best friend, written when we were 15. (i have honored her request to remain anonymous -- she undoubtedly has much more embarrasing material on me.)
a few excerpts:
'at miriam's birthday party, i got the idea that you wanted an honest evaluation from me about you.'
i now study this and this. coincidence? probably, yes.
let's continue:
'i think you are very good at understanding people, but sometimes a little too harsh in your judgment.'
this has little to do with my career, but i think my friends, colleagues, family members, students, neighbors, acquaintances, and penpals will be somewhat relieved (horrified?) to know that i was always this way.
'if i had to rate your personality on a 1-10 scale, 10 being the best, you would be a 9.'
also, my mom thinks i'm beautiful.* [note: i feel i should point out that there is no evidence for a general factor of personality. and the scale really should have a midpoint. why aren't i a 10 again?]
'you're always learning about yourself and growing. [...] you're very perceptive and it helps you to understand yourself well.'
this is a polite way to say i was already irritating people with my obsession with self-knowledge at age 15. seems like a trend. then again, i was also obsessed with reggie miller (i was young and naive. also, rondo** was barely alive at the time) and koko the gorilla (i'm still trying to outgrow that one).
of course, it's easy to see the signs in hindsight. if i had become an nba referee,*** i would be telling you about all of the kindergarten basketball games i refereed in high school (if you think that's an easy job, you have not met palo alto parents).
so, do i believe that i was a budding self-knowledge researcher at age 15? no. but it seems i was a pretty annoying friend.
*actually, she's not the type. the other day, she said of my banjo playing: 'tu fais des progres quand meme'. i'm not sure there is an adequate translation, but it is something along the lines of 'you don't suck quite as much as you used to.' i was pretty excited.
**i strongly recommend against watching the first 2:15 of this clip. the rest is magical.
***wouldn't it be awesome if an nba referee had a blog called 'sometimes i'm wrong'?
the self-deception alarm system
'those who venture to criticize us perform a remarkable act of friendship,
for to undertake to wound and offend a man for his own good
is to have a healthy love for him.'-michel de montaigne*
(all this time, reviewers were just expressing their healthy love for me!)
today i am going to talk about self-deception. don't worry, i will connect it back to scientific integrity.
being a researcher who studies self-knowledge and self-deception is a little nerve-wracking. watching other people delude themselves and be entirely convinced by their self-deception, you start to wonder whether all of your own self-beliefs might not also be delusional. it can make a person paranoid. so i started wondering, could there be an internal marker of self-deception? a red flag that, if trained, one could detect and catch oneself in the act of self-deception?
if there is such a clue, that would be extremely useful not just in everyday life, to know when we are deluding ourselves about personal matters, but also as scientists who recognize that we are susceptible to motivated cognition in our research. sometimes we might exploit researcher degrees of freedom without fully realizing that we are doing it. if there was a sign that we are doing this, and we could learn to detect that sign, that would be fantastic.
i think there might be such a sign. my own and other people's research suggests that people who know us well (our friends, family, etc.) sometimes can see things about us that we can't see about ourselves. in some ways, they know us better than we know ourselves. obviously this is not true for all close others, and there are also things we know that others don't. but let's assume that close others do sometimes have insight into what we're like, why we do what we do, etc.**
if that's the case, then an important clue that you might be deceiving yourself is when there is something you are avoiding telling the people you normally tell everything to. if you find yourself leaving out a crucial detail when retelling a story, or hoping your best friend won't ask you about something, maybe there is a truth buried under there that you are avoiding. of course there are other reasons we might avoid a topic, but with very close friends, one possible reason is that we're trying to maintain a fragile self-deception. we fudge the facts in order to shelter our self-deception from the sharp eye of those who are most likely to detect it.
with p-hacking, the same idea applies. any reluctance to disclose the steps you took to get the result you are presenting is probably a sign that you gave yourself too much leeway. sure, that study in the file drawer wouldn't change your overall conclusion anyway, but then why are you hiding it away in the file drawer? if you don't trust others to understand the rationale for your decisions and come to the same conclusion as you did, that is a sign that you might be deluding yourself. your colleagues are smart, if you need to hide the messy details from them, it's probably not because they won't 'get it.' it's because they will.
next time you find yourself tempted to hide some ugly details - about your life or about your data - find a trusted (and critical) confidante and force yourself to tell that person all the gory details. then, listen to what they say. and remember: to love is to criticize.
*i came across this quote in this essay by the fantastic bill deresiewicz. and if you like that, read this.
**i like to assume things. it's one of my top three hobbies.
i always live without knowing
magical things can still happen in used bookstores.
i was killing time in one and came across a book by richard feynman called 'the meaning of it all'. i noticed that the first essay was called 'the uncertainty of science'. i bought it. it was the best $4 i've spent in a while.
it is tempting to just re-type the entire essay here, but in the interest of trying not to violate copyright, and trying to contribute something of my own, i will share some excerpts and reflections.
first, feynman on p-hacking:
'it is necessary to look at the results of observation objectively, because you, the experimenter, might like one result better than another. you perform the experiment several times, and because of irregularities, like pieces of dirt falling in, the result varies from time to time. you do not have everything under control. you like the result to be a certain way, so the times it comes out that way, you say, "see, it comes out this particular way." the next time you do the experiment it comes out different. maybe there was a piece of dirt in it the first time, but you ignore it.'
in psychology, we don't have literal pieces of dirt messing up our experiments, but we have the other kind. we sometimes call them pieces of &*$%. and then we call them outliers and drop them from our studies. it is kind of amazing that feynman was talking about how motivated cognition can corrupt the scientific process. if a physicist in 1963 can recognize this, then social/personality psychologists in 2014 surely must.
next, feynman makes an interesting observation about disagreement among scientists:
'the relations among scientists were at first very argumentative, as they are among most people. this was true in the early days of physics, for example. but in physics today the relations are extremely good. a scientific argument is likely to involve a great deal of laughter and uncertainty on both sides, with both sides thinking up experiments and offering to bet on the outcome. [...] many sciences have not developed this far, and the situation is the way it was in the early days of physics, when there was a lot of arguing because there were not so many observations.'
psychology is much younger than physics. it is forgivable that we are not yet at their stage. adverserial collaborations are a good start, but eventually we need to get to the point where adversaries regularly settle things the adult way: bets and laughter.
next, feynman on guessing vs. knowing:
'i come now to an important point. the old laws may be wrong. how can an observation be incorrect? if it has been carefully checked, how can it be wrong? why are physicists always having to change the laws? the answer is, first, that the laws are not the observations and, second, that experiments are always inaccurate. the laws are guessed laws, extrapolations, not something that the observations insist upon. they are just good guesses that have gone through the sieve so far. and it turns out later that the sieve now has smaller holes than the sieves that were used before, and this time the law is caught. so the laws are guessed; they are extrapolations into the unknown.[...]
it would have been unscientific not to guess. it has to be done because extrapolations are the only things that have any real value. [...] knowledge is of no real value if all you can tell me is what happened yesterday. it is necessary to tell what will happen tomorrow if you do something--not necessary, but fun. only you must be willing to stick your neck out. [...] it is better to say something and not be sure than not to say anything at all.'
there are many things i like about this passage. one is that laws (or even conclusions) are not something that the observations insist upon. too often in discussion sections of papers i see people claim conclusions and implications from their results, as if their data insist upon those conclusions. observations insist upon very little, except themselves, and sometimes not even that. that doesn't mean we shouldn't stick our necks out. but we should be aware and honest that this is what we are doing when we draw inferences from our observations. say something, but don't pretend that your 74 participants' worth of data mean that it has to be true.
finally, on living without knowing:
'so what we call scientific knowledge today is a body of statements of varying degrees of certainty. some of them are most unsure; some of them are nearly sure; but none is absolutely certain. scientists are used to this. we know that it is consistent to be able to live and not know. some people say, 'how can you live without knowing?' i do not know what they mean. i always live without knowing. that is easy. how you get to know is what i want to know.'
if you cannot be happy without knowing, then i don't think you're cut out for science. some people need to know. that is fine. but those people should not become scientists, and especially not social scientists. predicting human behavior is not for the faint of heart. you must be prepared to be wrong, and wrong again, and again, and again. richard feynman seems to have been an extremely happy man. knowledge is not necessary for happiness.
'knowledge may have its purposes
but guessing is always
more fun than knowing'
-w. h. auden