sometimes i'm wrong

when are two things really one thing?

the question came up a few weeks ago: if two measures are correlated .60, should they be aggregated into a single measure or kept separate?  this might seem like a narrow question, but it raises some deep and complicated issues.  at the heart of the matter is: when are two measures measuring the same construct?

 

thinking about this reminded me of the jingle and jangle fallacies.  i have not read the original papers (the jingle fallacy apparently dates back to edward thorndike, 1904, and the jangle fallacy to truman kelley, 1927.  i have not tracked down either book/paper.  sorry.) the jingle fallacy is calling two things by the same name that are actually different constructs.  the jangle fallacy is using two different names for things that are actually the same construct.

 

let's get specific.  let's take the constructs 'neuroticism' and 'negative affect'.  are these two the same thing? (maybe, maybe not).

one obvious difference is that neuroticism is usually thought of as a stable personality trait, whereas negative affect is typically thought of as a momentary state. but either can be conceptualized either way.  in our own studies, we* measure trait neuroticism and trait negative affect, and we also get people to report their state neuroticism and state negative affect several times a day for several weeks (using experience sampling methods).  how do we decide whether neuroticism and negative affect are really a single construct (either at the trait or the state level)? 

 

i won't get into the specific results (because then we would not be able to publish them in a peer reviewed journal and get lots of fame and glory), but instead i will reflect on the factors that go into making this decision. there are two broad considerations relevant to deciding whether two measures are actually measuring the same construct: the empirical and the theoretical. 

1. the empirical part: how strongly are they correlated with each other?

 

the strength of the correlation between the two measures is an important clue to whether they are measuring the same construct.  but it is tough to know how strong of a correlation counts as strong enough to decide they are not distinct. 

 

the question i opened with - if two measures correlate .60 with each other, should they be aggregated? - is not a straightforward one.  in fact, .60 is just about at the threshold of my own intuition about where the line is drawn between convergent and discriminant validity (or maybe it's more like .55, or .568). but how to interpret the .60 correlation depends on a few other things. perhaps most importantly, what is the reliability of each measure? if each measure has a reliability of .60, then the fact that they correlate .60 with each other suggests that they are highly similar, because a measure's reliability is, roughly speaking, the theoretical upper limit to how strongly it can correlate with other measures. this is because a measure's reliability is, in essence, its correlation with itself, so it should not correlate with another measure more strongly than it correlates with itself. (but this is in theory only, in reality measures regularly exhibit correlations with other measures that are stronger than their own reliability.  partly because reliability estimates can be wrong, and partly because, as charcot said, 'theory is nice, but it doesn't prevent things from existing.')  so if the correlation between two measures is high, and close to the reliabilities of the two measures, then that is pretty strong evidence that they are measuring the same construct. 

 

there is more to the empirical part. for example, do the two measures show similar patterns of association with other measures? do they covary not just across people, but within-person over time? when combined, do they load onto a single factor? but this is a blog so i am going to ignore those issues.

 

2. the theoretical part: are there important conceptual distinctions between the two constructs?

 

it is possible that, even if all empirical results point to the conclusion that your two measures are indeed getting at the same construct, you are persuaded by strong theoretical reasons to keep them separate.  you are probably wrong, but you can try to make the case.

 

i am a dust bowl empiricist at heart.  i am inclined to believe the data over anyone's theory.  so it will be difficult to convince me that two measures that don't pass the empirical test of distinctiveness are actually measuring distinct constructs.

however, if you have a strong theoretical argument why the two things you are attempting to measure are distinct constructs, i will entertain the possibility that your theory is right and your measures are crap.  this is perhaps the best use i can think of for theory - sometimes it helps us figure out that we are using really bad measures.

 

what are some examples of good theoretical reasons to maintain that two things are distinct constructs, even in the face of empirical evidence to the contrary? one is that you believe they arise from different processes.  these could be developmental processes, cognitive processes, biological processes, etc.  but you still need empirical evidence to back up the process difference, so this is not really a theoretical argument.

 

another is that you can describe scenarios where the two constructs come apart (this is related to the 'different processes' reason - if they can come apart, there must be some difference in the processes that produce them).  this is one purpose of thought experiments. for example, if you can paint a compelling picture of a time when a person would experience neuroticism but not negative affect, then you might be able to convince me that they are different constructs, even if, practically speaking, they almost never come apart.  in my view, the scenario you depict has to be relatively plausible (philosophers and others may disagree).  i don't care if neuroticism and negative affect could come apart on twin earth, i want to know if they could come apart in a situation human beings are liable to encounter here on this actual planet.

 

but in the end, i think you have to back up your theoretical argument with empirical evidence.  if you really believe your two things are different constructs, you need to be able to observe them coming apart in actual data.  that might mean developing new, better measures, or tracking down rare cases or rare situations, or doing other difficult things, but until you do those things, you are not going to convince me based on theory alone.

those are my half baked thoughts.  many people have said much smarter things about construct validation.  paul meehl, jane loevinger, campbell and fiske, cronbach... and also some people who are currently alive.  i'm not sure why you are reading my blog instead of reading them. come to think of it, i'm not sure why i'm writing my blog instead of reading them.

*by 'we' i mean my awesome graduate students and our fantastic research assistants.

why i study self-knowledge

someone recently asked my why i do what i do.  it's easy to come up with just-so stories. for all i know, i could just as easily have ended up inventing new ice cream flavors for tara's ice cream (i still haven't ruled it out).  and if i had, i could probably make up a story about why it was always meant to be.  narratives are deceptively easy to construct, and amazingly convincing after the fact. it would be especially ironic for a self-knowledge researcher to deceive herself about why she studies self-knowledge.  so i mostly try to resist giving an explanation.  but then there's this:

this is a note from my best friend, written when we were 15. (i have honored her request to remain anonymous -- she undoubtedly has much more embarrasing material on me.)

a few excerpts:

'at miriam's birthday party, i got the idea that you wanted an honest evaluation from me about you.'

i now study this and this.  coincidence? probably, yes.

let's continue:

'i think you are very good at understanding people, but sometimes a little too harsh in your judgment.'

this has little to do with my career, but i think my friends, colleagues, family members, students, neighbors, acquaintances, and penpals will be somewhat relieved (horrified?) to know that i was always this way. 

'if i had to rate your personality on a 1-10 scale, 10 being the best, you would be a 9.'

also, my mom thinks i'm beautiful.* [note: i feel i should point out that there is no evidence for a general factor of personality.  and the scale really should have a midpoint.  why aren't i a 10 again?]

'you're always learning about yourself and growing. [...] you're very perceptive and it helps you to understand yourself well.'

this is a polite way to say i was already irritating people with my obsession with self-knowledge at age 15.  seems like a trend.  then again, i was also obsessed with reggie miller (i was young and naive. also, rondo** was barely alive at the time) and koko the gorilla (i'm still trying to outgrow that one).

of course, it's easy to see the signs in hindsight. if i had become an nba referee,*** i would be telling you about all of the kindergarten basketball games i refereed in high school (if you think that's an easy job, you have not met palo alto parents).

so, do i believe that i was a budding self-knowledge researcher at age 15?  no. but it seems i was a pretty annoying friend.

*actually, she's not the type.  the other day, she said of my banjo playing: 'tu fais des progres quand meme'.  i'm not sure there is an adequate translation, but it is something along the lines of 'you don't suck quite as much as you used to.'  i was pretty excited.

**i strongly recommend against watching the first 2:15 of this clip.  the rest is magical.

***wouldn't it be awesome if an nba referee had a blog called 'sometimes i'm wrong'? 

 

the self-deception alarm system

 

'those who venture to criticize us perform a remarkable act of friendship,
for to undertake to wound and offend a man for his own good
is to have a healthy love for him.'

-michel de montaigne*

(all this time, reviewers were just expressing their healthy love for me!)

today i am going to talk about self-deception. don't worry, i will connect it back to scientific integrity.

being a researcher who studies self-knowledge and self-deception is a little nerve-wracking.  watching other people delude themselves and be entirely convinced by their self-deception, you start to wonder whether all of your own self-beliefs might not also be delusional. it can make a person paranoid. so i started wondering, could there be an internal marker of self-deception? a red flag that, if trained, one could detect and catch oneself in the act of self-deception?

if there is such a clue, that would be extremely useful not just in everyday life, to know when we are deluding ourselves about personal matters, but also as scientists who recognize that we are susceptible to motivated cognition in our research.  sometimes we might exploit researcher degrees of freedom without fully realizing that we are doing it.  if there was a sign that we are doing this, and we could learn to detect that sign, that would be fantastic.

i think there might be such a sign.  my own and other people's research suggests that people who know us well (our friends, family, etc.) sometimes can see things about us that we can't see about ourselves. in some ways, they know us better than we know ourselves.  obviously this is not true for all close others, and there are also things we know that others don't. but let's assume that close others do sometimes have insight into what we're like, why we do what we do, etc.**

if that's the case, then an important clue that you might be deceiving yourself is when there is something you are avoiding telling the people you normally tell everything to. if you find yourself leaving out a crucial detail when retelling a story, or hoping your best friend won't ask you about something, maybe there is a truth buried under there that you are avoiding. of course there are other reasons we might avoid a topic, but with very close friends, one possible reason is that we're trying to maintain a fragile self-deception. we fudge the facts in order to shelter our self-deception from the sharp eye of those who are most likely to detect it.

with p-hacking, the same idea applies. any reluctance to disclose the steps you took to get the result you are presenting is probably a sign that you gave yourself too much leeway.  sure, that study in the file drawer wouldn't change your overall conclusion anyway, but then why are you hiding it away in the file drawer?  if you don't trust others to understand the rationale for your decisions and come to the same conclusion as you did, that is a sign that you might be deluding yourself.  your colleagues are smart, if you need to hide the messy details from them, it's probably not because they won't 'get it.' it's because they will.

next time you find yourself tempted to hide some ugly details - about your life or about your data - find a trusted (and critical) confidante and force yourself to tell that person all the gory details.  then, listen to what they say.  and remember: to love is to criticize.

*i came across this quote in this essay by the fantastic bill deresiewicz.  and if you like that, read this.

**i like to assume things.  it's one of my top three hobbies.

buckets of tears

i learned a new word the other day. bucketing. it almost made me cry.

one of the most common mistakes i see when reviewing papers is authors who take a continuous variable and, for no good reason, mutilate it by turning it into a categorical variable. our old friend the median split is one example.  (whose idea was it to befriend the median split? and why won’t he stop harassing us?)

bucketing, from what i can tell, is another such technique.  i had a hard time finding a definition, but i think it’s basically creating categories out of multiple response options and grouping the data that way. for example, you can turn the continuous variable ‘age’ into a categorical variable by categorizing people into age ‘buckets’ (e.g, 20-29, 30-39, etc.).

why is bucketing so sad? first, turning continuous variables into categorical variables is almost always a bad idea because you lose information. unless you have good reason to believe that all variance within each bucket is error (e.g., you know that the difference between someone at the bottom end vs. the top end of the response options in a given bucket is not at all meaningful), you are losing good variance (i.e., valuable information) by lumping them all together.  age is a particularly curious variable to ‘bucket’ because there is usually very little error in people’s self-reports of their age,* so there is no reason not to take their reported age at face value.  relatedly, turning continuous variables into categorical variables leads to treating two data points that are actually very close together (e.g., a 29 year old and a 30 year old) as if they were quite different (essentially treating the 29 year old as a 25 year old, and the 30 year old as a 35 year old).

another distressing thing about bucketing is that it provides an almost irrestistible opportunity to p-hack. because there are many ways to break up a continuous variable into multiple categories, a researcher can try any number of combinations to see which one ‘works best’. before you know it you will have buckets coming out of your ears. and most of the time the reader cannot know how many ways of bucketing were tried, and what the other results looked like. (though if the researcher makes her data publicly available, the reader can find out). other methods for converting continuous variables into categorical ones have the same problem (e.g., you can try splitting your sample into two groups, three groups, four groups, etc.).

i can see why bucketing is appealing.  it's a way to reduce complex data into neater bundles, and may be more intuitive (e.g., you can talk about the category of ‘extreme responders’ instead of talking about the degree of extremity of responses). this is similar to the appeal of personality types, another concept that will not die.** sharp cutoffs are exciting.  but often they are an illusion.

there are cases where turning a continuous variable into a categorical variable makes sense. for example, it is sometimes much easier to visually display results in a categorical fashion (e.g., when plotting an interaction effect in regression). however, it is almost never justifiable not to also report the results of the analyses with the original, continuous variable.  if you don’t know how to do that analysis, bribe a stats friend (mine like espresso and cocktails. and very small dogs). and if you are going to chop up your continuous variable and place it into buckets, remember that the more different ways of bucketing you try, the more likely you are to capitalize on chance.  and if you run a replication, you should commit to using the same buckets when analyzing the second sample. but most of the time, there is just no good reason to bucket.

i am aware that none of this is original, we all learned this in intro stats. but i still see it so often (most recently in a PNAS article), that i think it bears repeating. and what are blogs for if not repeating things we already know?

bears repeating.

* i am making this bold claim with no empirical evidence whatsoever.  luckily my bold claims have very little error.

** have you noticed that over the course of reading this blog, you have learned little tidbits about personality? you're welcome.

life after bem

many people have written about how bem's esp paper was one of the major factors that triggered the latest scientific integrity movement in social/personality psychology.  that is an interesting story.  but that is not the bem paper i want to talk about today.

i have come here today to discuss bem's chapter, 'writing the empirical journal article' (2003) that i - and i suspect many others - used to assign to every undergrad and graduate student taking our psychology research methods classes.  there are many extremely wise points in that chapter.  but there are also many pieces of advice that seem entirely antiquated today.  if the bem chapter is no longer the gold standard for how to write an empirical article, what is? (see also: laura king's article for the spsp dialogue (pdf, p. 6)).

i was reminded of the complexity of this question when a historian friend of mine suggested i read 'the question of narrative in contemporary historical theory' by hayden white (1984). i will share a few quotes with you:

'but it is precisely because the narrative mode of representation is so natural to human consciousness, so much an aspect of everyday speech and ordinary discourse, that its use in any field of study aspiring to the status of a science must be suspect. for whatever else a science may be, it is also a practice which must be as critical about the way it describes its objects of study as it is about the way it explains their structures and processes.'

'a discipline that produces narrative accounts of its subject matter as an end in itself seems methodologically unsound; one that investigates its data in the interest of telling a story about them appears theoretically deficient.' *

 

what struck me about these passages is that white is struggling with the tension between science and storytelling.  this is precisely the tension that has led me to stop assigning bem's article to my research methods classes.  bem is a pretty strong advocate of the narrative/storytelling approach to writing empirical papers.  i don't completely disagree, but he is not conflicted enough about this approach for my tastes.

first, i want to say that there are many things i love about bem's chapter.  (his explanation of 'that' vs. 'which' changed my life.)  he is an amazing writer and there is a massive amount of great advice in the chapter.  but there are problematic parts. 

 

let's start with this: bem recommends writing 'the article that makes the most sense now that you have seen the results' (rather than 'the article you planned to write when you designed your study').  it is pretty clear from the rest of this section that bem is basically telling us to invent whatever a priori hypotheses turn out to be supported by our results.  indeed, he says this pretty explicitly on the next page:

'contrary to the conventional wisdom, science does not care how clever or clairvoyant you were at guessing your results ahead of time. scientific integrity does not require you to lead your readers through all your wrong-headed hunches only to show - voila! - they were wrongheaded.'

actually, science does care.  and scientific integrity does require something along those lines. not that you tell us about all your wrong ideas, but that you not claim that you had the right idea all along if you didn't.  if science didn't care, pre-registration and bayesian statistics would not be enjoying the popularity they are today. **

this alone is enough to make me stop assigning this chapter, but there are other worrisome bits. specifically, bem advocates exploring the hell out of your data ('analyze the sexes separately. make up new composite indexes [...] reorganize the data to bring them into bolder relief.'), which i have no problem with at all, if people are transparent.  but then he seems to be telling us to present these exploratory results as confirmatory.  he talks about the distinction between the context of discovery and the context of justification, but then completely blurs the line a few paragraphs later.  he says:

'when you are through exploring, you may conclude that the data are not strong enough to justify your new insights formally, but at least you are now ready to design the "right" study. if you still plan to report the current data, you may wish to mention the new insights tentatively, stating honestly that they remain to be tested adequately. alternatively, the data may be strong enough to justify recentering your article around the new findings and subordinating or even ignoring your original hypotheses.'

if it weren't for that last sentence, that would be great advice.  either run a new study (and pre-register your hypotheses!), or publish the current study but explain that the analyses were exploratory and should be taken with a grain of salt.  but why recommend ever 'recentering' your article around the p-hacked, exploratory results as if they were predicted a priori?  maybe i am reading too much into what he wrote, but in combination with the other bolded quote (above), i think my concerns are justified. certainly i would not blame an undergrad or graduate student for interpreting him to be saying this.  in fact, at the end of this section he writes:

'think of your dataset as a jewel. your task is to cut and polish it, to select the facets to highlight, and to craft the best setting for it.'

this can no longer be considered an appropriate metaphor.  it is just too blatantly unscientific.  the question is not what is the best setting (or best story you can tell), but what is the presentation that most advances scientific knowledge, is most true to the evidence?

that is a tough question to answer.  a beautifully carved story that omits a bunch of important details is not the answer, but a dry list of brute facts does not necessarily do the most to advance scientific knowledge, either. 

 

it is the duty of scientists to overcome our natural tendency to fit everything to a narrative, and to try to describe things as they actually are.  yet, as a reader of psychology articles i do not want just the brute facts.  it's just not realistic to expect other scientists, much less anyone else, to pay attention to our research results if we do not put any effort into creating a coherent story around them.  we can't leave all of the interpretation and meaning-making up to the reader. 

 

the tension here is between sticking to the (messy) facts, and doing some polishing and interpreting for the reader so she doesn't have to wade around a giant pile of unstructured findings.  i will give bem credit, he seems aware of this tension.  later in the chapter, he writes:

 

'[...] be willing to accept negative or unexpected results without a tortured attempt to explain them away. do not make up long, involved, pretzel-shaped theories to account for every hiccup in the data.' 

so he acknowledged that the results don't need to be perfect (though he seems more worried about the fact that the story you would need to tell to explain the imperfections would itself be ugly and tortured, rather than about the fact that it's important to disclose the imperfections for the sake of scientific rigor).

 

back to the question: how do we balance the mandate to present the facts in all their messy glory and the need to give some structure and coherence to our observations?

one easy step is to clearly and honestly label exploratory analyses as such.  ideally these would be followed up with confirmatory studies, but sometimes exploratory results are exciting enough (and/or hard enough to collect new data about) to publish on their own. 

another (related) step is to stop claiming to have had a priori hypotheses that we didn't actually have.  however, this step is trickier because editors and reviewers still want to see the 'correct' hypothesis up front. there are some good reasons for this - it is kind of a waste of time to read an entire intro building up to a hypothesis that turns out to be false.  one handy solution for this is to select research questions that are interesting whichever way the results come out - then the intro can present the two competing hypotheses and why each result would be interesting/informative.***  i am told that in some areas of research this is just not possible.  all the more reason for everyone to become a personality psychologist.  come join us, we never know which way our results will turn out!

so what is a better metaphor than the jewel cutting metaphor?  more broadly speaking, what is the right way to teach our students to write empirical journal articles?

i don't know.  i'm asking you.

 

* these quotes are both from page 1.  i stopped understanding the essay after about page 4.  in my defense, the footnotes are in french.

 

** in the academic world, 'popularity' just means it gets nerds excited. dan ariely's ama was more 'popular' on my facebook feed than any of the completely outrageous basketball upsets this weekend.  seriously.  sorry spartans.

 

*** do not attempt this in a grant proposal.  i did once and was told to come back with a directional hypothesis.  i flipped a coin and resubmitted the grant.  it was funded. (eventually.  i think it took four tries.)

 

(photo: my dog, longing for answers.)

(actually she is longing for the attention of a newfoundland off to the left.) 

unbelievable.

one of the most fascinating dimensions along which psychology researchers differ is in their reaction to counterintuitive findings.  when some people say 'i can't believe it!' they mean it as a compliment.  when others say it, you should take cover.  how should we feel about counterintuitive findings?

i'll come out and say it: i have not drunk the bayesian kool-aid. i do like the idea that the amount of evidence required to support a claim should depend on the plausibility of that claim to begin with, but the reason i'm not a whole-hearted bayesian is that i am skeptical that there will be much consensus in psychology about which claims are more or less probable.  (have you ever asked a group of psychologists what proportion of our a priori hypotheses are likely to be right? you should try it, it's a fun party trick.) but i have seen cases where pretty much everyone agrees that a conclusion is very counterintuitive (in fact, there are quite a few cases where the authors themselves appeal to this as a selling point of their work).  and in those cases we can ask: given that we all agree this is surprising, should we hold the research to a higher standard? do the authors need more evidence if the claim they are making is widely acknowledged to be shocking?

bayesians would say yes (spoiler: i agree with them). but let's explore the counterargument. the counterargument, as far as i can tell, goes something like this: counterintuitive findings do more to advance our knowledge because they force us to rethink our old theories. in kuhnian language,* these are the anomalies that lead to a crisis in the paradigm, and sometimes force scientific revolutions. findings that are intuitive, or fit with the existing paradigm, can add more bricks to the wall but will never lead to huge leaps in knowledge.

sounds good.  i agree that counterintuitive findings can be breakthroughs. but if they cause so much upheaval, isn't that all the more reason to hold them to a higher evidentiary standard? if they have the potential to undo entire paradigms, don't we want to be super sure about them?

as an editor, i am torn about what to do when handling a manuscript with very counterintuitive findings.  i would like to require more evidence than for more intuitive findings.  but several factors make me hesitate. first, i don't want to put much weight on what i personally find counterintuitive. (my intuitions are not exactly normal. for example, i have the strong intuition that an avocado-raspberry smoothie would be delicious.** luckily i don't go near kitchen equipment so there is little danger of me finding out.)  second, even if the authors and reviewers agree that the finding is counterintuitive, it is hard to know how much evidence is enough.  twice as much as for an intuitive finding? three times? 

so, in conclusion, i do think that counterintuitive findings should be held to a higher standard, but it's hard to know for sure which findings are counterintuitive, and how much higher the standard should be. 

i bet you're glad i settled that one.

*i attended a great lecture on kuhn by one of my colleagues a few weeks ago.  i have not actually read kuhn. and while i'm confessing, i don't know much about bayesian statistics, either.

**it's possible i've spent too much time in california.

is bad replication a sin?

 

should replications be held to a higher standard than original research?

i have seen some very bright and influential people argue that they should, mainly because of the potential damage that a failed replication could do to the original author's reputation.  according to this argument, shoddy original research may be 'irresponsible' but shoddy replication is a 'sin'.* 

i have several objections to this.

1. original research is often treated as precedent.  my impression is that people see the original finding as very likely to be true, and require a lot of new evidence to be convinced otherwise.  this is problematic for many reasons, but if it's true, it's all the more reason to hold original research to a very high standard.  giving original research a pass is dangerous given how hard it is to overturn a finding once it is in the literature.

2. i admit that there are probably exceptions to point #1.   i certainly have seen some shoddy replications get more attention than they deserve.  however, i don't think they do much reputational damage, because the scientific community is rather quick to point out the flaws.  there are many people who have a stake in not letting shoddy replications have the last word - not just those who are inclined to believe the original finding, but also those who would like to promote replications.  the worst publicity for reform is shoddy research in the name of reform. (also: people who are mean in the name of reform.  being nice to people is a really good idea.  especially if you are trying to convince them to make difficult changes.)

3. if we adopt meta-analytic thinking, all evidence is grist for the mill. there is no scientific basis for treating evidence differently based on chronology.  quality, yes.  chronology, no.  [i know very little about meta-analysis, but luckily that does not stop me from using phrases like 'meta-analytic thinking.'  it is one of my favorite phrases. (i also like 'standard normal deviant.'  as in, some of my best friends are standard normal deviants.)]

4.  there is also the very real danger of scaring the daylights out of would-be replicators.  calling shoddy replications a sin might be enough to deter the more timid among us.  maybe that is the point - encourage people not to take replication lightly - but there is a fine line between encouraging good behavior and threatening well-meaning people.  funder has written and spoken about the plight of the replicator.  it is already pretty damn scary for a less established researcher to attempt to replicate a well-established finding.  i'm not sure we should be trying to make it more scary. 

5.  we also should not give in to the narrative that a failed replication, even when it is so rigorous that it is basically definitive, should harm the original author's reputation.  there are cases where it clearly should, but it is good to remember that people sometimes get unlucky (or too lucky, in the p < .05 sense), and a false positive does not make someone a bad person, or even a bad researcher.  it all depends on how rigorous the original study (and analyses) were.  and it also depends on how the original researcher reacts to the strong evidence that their finding was a false positive.  there are several prominent examples of people whose work failed to replicate in a convincing way, but whose reputations were not harmed, and maybe were even improved, because they reacted in an extremely scientifically-minded, non-defensive way.  i hope to dedicate some blog posts to those outstanding examples in the future. (please let me know of any cases you know of.)

*someone you've heard of used those words.

 

happy persian new year!

i always live without knowing

 

magical things can still happen in used bookstores.

i was killing time in one and came across a book by richard feynman called 'the meaning of it all'.  i noticed that the first essay was called 'the uncertainty of science'. i bought it.  it was the best $4 i've spent in a while.

it is tempting to just re-type the entire essay here, but in the interest of trying not to violate copyright, and trying to contribute something of my own, i will share some excerpts and reflections.

first, feynman on p-hacking:

'it is necessary to look at the results of observation objectively, because you, the experimenter, might like one result better than another. you perform the experiment several times, and because of irregularities, like pieces of dirt falling in, the result varies from time to time. you do not have everything under control. you like the result to be a certain way, so the times it comes out that way, you say, "see, it comes out this particular way." the next time you do the experiment it comes out different. maybe there was a piece of dirt in it the first time, but you ignore it.'

in psychology, we don't have literal pieces of dirt messing up our experiments, but we have the other kind. we sometimes call them pieces of &*$%.  and then we call them outliers and drop them from our studies.  it is kind of amazing that feynman was talking about how motivated cognition can corrupt the scientific process.  if a physicist in 1963 can recognize this, then social/personality psychologists in 2014 surely must.

next, feynman makes an interesting observation about disagreement among scientists:

'the relations among scientists were at first very argumentative, as they are among most people. this was true in the early days of physics, for example. but in physics today the relations are extremely good. a scientific argument is likely to involve a great deal of laughter and uncertainty on both sides, with both sides thinking up experiments and offering to bet on the outcome. [...] many sciences have not developed this far, and the situation is the way it was in the early days of physics, when there was a lot of arguing because there were not so many observations.'

psychology is much younger than physics.  it is forgivable that we are not yet at their stage.  adverserial collaborations are a good start, but eventually we need to get to the point where adversaries regularly settle things the adult way: bets and laughter.

next, feynman on guessing vs. knowing:

'i come now to an important point. the old laws may be wrong. how can an observation be incorrect? if it has been carefully checked, how can it be wrong? why are physicists always having to change the laws? the answer is, first, that the laws are not the observations and, second, that experiments are always inaccurate. the laws are guessed laws, extrapolations, not something that the observations insist upon. they are just good guesses that have gone through the sieve so far. and it turns out later that the sieve now has smaller holes than the sieves that were used before, and this time the law is caught. so the laws are guessed; they are extrapolations into the unknown.[...]

it would have been unscientific not to guess. it has to be done because extrapolations are the only things that have any real value. [...] knowledge is of no real value if all you can tell me is what happened yesterday. it is necessary to tell what will happen tomorrow if you do something--not necessary, but fun. only you must be willing to stick your neck out. [...] it is better to say something and not be sure than not to say anything at all.'

there are many things i like about this passage.  one is that laws (or even conclusions) are not something that the observations insist upon.  too often in discussion sections of papers i see people claim conclusions and implications from their results, as if their data insist upon those conclusions.  observations insist upon very little, except themselves, and sometimes not even that.  that doesn't mean we shouldn't stick our necks out.  but we should be aware and honest that this is what we are doing when we draw inferences from our observations.  say something, but don't pretend that your 74 participants' worth of data mean that it has to be true.

finally, on living without knowing:

'so what we call scientific knowledge today is a body of statements of varying degrees of certainty. some of them are most unsure; some of them are nearly sure; but none is absolutely certain. scientists are used to this. we know that it is consistent to be able to live and not know. some people say, 'how can you live without knowing?' i do not know what they mean. i always live without knowing. that is easy. how you get to know is what i want to know.'

if you cannot be happy without knowing, then i don't think you're cut out for science.  some people need to know.  that is fine.  but those people should not become scientists, and especially not social scientists.  predicting human behavior is not for the faint of heart.  you must be prepared to be wrong, and wrong again, and again, and again. richard feynman seems to have been an extremely happy man. knowledge is not necessary for happiness.

'knowledge may have its purposes
but guessing is always
more fun than knowing'
-w. h. auden 

having it all

 

 this blogging thing is pretty rad.  (also: twitter.  wow.)

today's topic: having it all.  i'm not talking about the work/life problem.  i'm talking about the sample size/methodological rigor problem. 

let's start with sample size.  by now, everyone knows that bigger is better.  you can't have too large a sample.  there is no 'double-edged sword'.  there are no downsides to a large sample.  more evidence is always better, and larger samples = more evidence.

this seems very obvious but i've seen at least three different editors criticize manuscripts for having samples that are too big.  so i want to be very clear: there is no such thing as a sample that is too big.  calling a sample too big is like calling a a grizzly bear too cute.  it makes no sense.

many people have written very compelling explanations about why we should want larger samples (more power).  i will trust that you have read those. 

what i want to talk about is the downside of large samples.

ok, i know i just said there are no downsides, but that's once the data have already been collected.  if you are an editor or a reviewer, and you are reading a manuscript with a sample size of, say, 884,328, you should consider this a major asset.  it's not complicated.

but let's say you're a researcher, designing a study.  you've heard large sample sizes are awesome.  you have $1,000 to spend.  what should you do?

one of my biggest fears in life is that people will accept the large-samples mantra unthinkingly, and decide to run all their studies on mturk, where they can get 10,000 participants for their $1,000. my other biggest fear is spiders.

over the last thirty years, my field (personality psychology), has finally grown out of its over-reliance on self-reports.  it is now practically impossible to publish a cross-sectional study of college students that relies exclusively on self-reports.  and that's probably as it should be.  we cannot build our knowledge of human behavior on self-reports, reaction times, and vignette studies alone (experimental philosophers, take note).

however, now that people are catching on to the fact that most of our studies have samples that are too small, people are turning to mturk to increase their sample sizes.  this is a nice sentiment, and there are some great things about mturk (and some not so great...).  but there is a problem: it is exceedingly difficult to collect actual behavioral measures, informant reports, or physiological measures from mturk samples.  and we need those measures. how can we study morality, love, stress, leadership, or any other interpersonal phenomenon if we only have the self's perspective? 

we need good methods.  that means we need multiple methods, and each one needs to be implemented rigorously.  this is very expensive and time consuming.  multiply that by many many participants, and it feels like we have to make a sacrifice.  we can't have it all.

but we can.  there is one ridiculously simple solution: slow down.

it's true that we cannot have both large samples and diverse, intensive methods if we want to continue running billions of studies per year.  but there is another model.  run fewer, better studies. 

despite my skeptical exterior, i have a wildly idealistic streak. i don't like to compromise.  so i believe that we can have large, beautiful studies with multiple, reliable, valid methods.  we just have to value them.  journals need to bend over backwards to publish those studies, and all of us need to treat those studies as much more definitive than small, mono-method studies.  they should be the holy grail of psychological research.

i understand the pressure to do things quickly.  i know that calling on people to collect larger samples necessarily puts pressure on them to use cheaper, quicker methods.  but we should not easily give in to the notion that we must choose one or the other.  we can have it all.  we just have to be patient. 

 

the end.

 

**photo credit: erik pettersson.

sometimes i'm wrong

 

i was having breakfast with friends at beauty's bagel when i saw this picture.  i could not stop staring.  the jesus-like erotic figure is nice of course, but what mesmerized me was the sentiment.  this is the phrase i've been searching for.  i can finally start my blog.

this blog is about being wrong.  specifically, i plan to write about two things:
1. the search for truth in psychological research.
2. the search for self-knowledge in everyday life.

my thesis: both of these pursuits are riddled with obstacles, and should be approached with a great deal of skepticism and humility.

why blog?
when i was in graduate school, a professor warned me never to contradict myself in writing.  i was shocked. my data did not always tell me the same story. my own views fluctuate all the time. why did i need to take a single position and not budge?  because that is how you build a reputation in science, i was told. you associate your name with a position, and spend the rest of your career standing your ground (while brandishing your humungous cv).

of course, this is crazy. we are all wrong.  all the time. some of us are just less wrong than others.  the best we can hope for is that our ideas are interesting enough that other people will bother to refute (and thus refine) them.

sometimes i lay awake at night worried that self-skepticism has been abandoned in psychological science. we are all trying to sell our results, to journal editors and reviewers, to each other, to the media.  we feel threatened when someone suggests we might have been wrong.  we treat people who challenge others' findings as bad sports.  we revere the people who defend their positions most fervently, rather than the ones who invite criticism and are willing to change their minds. the world is upside down.

but there is hope. skepticism is making a comeback -- blogs are going to save the world. this is my attempt to add my voice to the chorus. 

here are some things i think i might bring to the conversation (but i'm probably wrong):
1. i like research methods, and i have a mild crush on statistics.  i can't seem to commit, but i like to dabble.
2. i spend a lot of time with manuscripts during the review process.  i see how the sausages are made.  i am partly responsible for some of the foul sausages.  (and, i'd like to think, for some of the delicious ones, too.)
3.  you might have noticed that i am not male, whereas a mindblowing proportion of voices in this area are.  ask yourself: would a man have posted the picture above?  i am already broadening the scope of the conversation.

*artist credit: matt leibowitz