nothing beats something

[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

hello

i'm reading a book called Nothing is True and Everything is Possible, about Russia, and the name keeps haunting me (the content of the book is good, too).  sometimes i worry that this goes for science, too.  it's not just that when nothing is true, everything is possible.  but when everything is possible, nothing is true.  

sometimes studying human behavior is so wild and messy that it feels like anything goes.  we create ad hoc scales more often than we probably should, and we invent manipulations out of thin air, rarely pausing to validate them.  if the best we can do is a college student sample, that sin will be forgiven.  if we can't get a behavioral measure of a behavior, a self-report will often do.  we do what's possible because, well, what's the alternative?

i'm here to consider the exceedingly unpopular view that the alternative - to do nothing - is sometimes preferable to doing what's possible.

science is hard.  social science can be especially hard, because we usually need to study humans, and humans are a bitch to study.*  there's this idea out there that it's better to collect noisy, messy data than no data at all.  i've often repeated this claim myself.  indeed, i taught it when i used David Funder's excellent personality textbook.  Funder's third law** is that "something beats nothing, two times out of three."  Funder was smart to add the caveat, because i think sometimes nothing beats something.

slowly over the last few years, i've come to the conclusion that sometimes crappy data is actually worse than no data.  i think that we sometimes fall into the trap of thinking that, because a phenomenon is really, really, really hard to study, very flawed studies are the best we can hope for and so are ok.  it's almost like we are committed to the belief that everything must be possible for a reasonably industrious researcher to study, so if the only option is a bad study, then a bad study must be good enough.

let me be clear: i have definitely made this argument myself, possibly as recently as last month.  i'm no stranger to the feeling that just getting adequate data feels almost impossible, and so inadequate data must do.  for example, i was pretty proud of myself when my lab decided to not just code actual behavior from EAR recordings of >300 participants, and not just double-code the recordings, but TRIPLE code the recordings.  when, after two years of coding by over 50 coders (which came on the heels of several years of data collection), we checked the reliabilities on those triple codings and found them wanting, it was very, very tempting to say "well, it's the best we can do."  luckily i had learned about the spearman-brown prophecy formula, and so had no excuse - adding more raters was definitely going to help, it's a mathematical fact.  so we got three more coders per recording (which took about two more years and 50 more coders).  and let me tell you, it sucked.  we are sick of coding.***  we wish we could have spent those two years doing new projects.  but we got the reliabilities up, and that will let us get much better estimates.  science is not all creativity and innovation.  it is a lot of boring, hard work.  if you don't sometimes hate science, you might be doing it wrong.

even better examples of people who kept going when most of us would have said "good enough:" the two researchers i highlighted in my blog post, "super power."  their efforts really make me rethink my definition of impossible.  

but sometimes getting good data really is impossible, at least for any single lab.  and i would like to float the idea that when this is the case, there may be something noble in walking away, and choosing to do nothing rather than something.  sometimes, the nicest thing you can do for your research question is to leave it unanswered.  (my colleague wiebke bleidorn pointed out that this is called "addition by subtraction.")  if you insist on extracting information from really crappy data, you put yourself at really high risk**** of  reading patterns into noise.  just because you need your data to tell you something, to move the needle, doesn't mean it can.  if our field allowed people to get credit for publishing little bits of (really hard-to-collect) data without making any claims whatsoever, this might be a viable approach, but i don't think that currently exists (though there are low-recognition ways to do this, of course).

the wishful thinking that we can always extract some knowledge from even small bits of messy data can lead to serious, widespread problems.  it's easy to point fingers at other fields, but this problem is alive and well in psychology.  i'm very familiar with the argument that this sample size or that method must be good enough because it's all we can realistically do, given limited resources, hiring or promotion expectations, etc..  it used to be one of the most common responses i'd hear to calls for larger samples or better (i.e., harder) methods.  people seem to have backed off of making the argument out loud, but i think it's still very common in people's minds - this attitude that it's inappropriate to say a method or design isn't good enough because it was hard/expensive/time-consuming to do.  (indeed, i think this is the most common internal response most of us have when we get criticized for studying only WEIRD samples.)

here's what i wish we all did (including me):  1) ask yourself what the ideal study would be to test your research question.  2)  ask yourself if you're willing and able to do something pretty close to it.  3)  if not, ask yourself why not.  really push yourself.  don't let yourself off the hook just because it would be a lot harder than what you're used to doing.  if the research question is important, why isn't it worth the really hard work?  4)  if you're still not willing or able to do something close to the ideal study on your own, either a) move on to a different (or narrower) research question, or b) join forces with other labs.  

i know this is idealistic.  i know we have to keep publishing otherwise our food pellets stop coming.  so let's pick questions we can actually hope to answer with the resources we have.  (this post is really one big, public NOTE TO SELF.  i have been extremely guilty of biting off way more than i can chew,***** and want to learn to scale back my expectations of what kinds of research questions i can rigorously test on my own.)  or join collaborative projects that pool resources to tackle the really hard, important questions, and find a way to deal with the issue of spreading credit around.  if we stop trying to do the impossible, i think we'll find that more things are true.

* also, sometimes to interact with.  or be near.  

** Funder calls his lessons "laws" somewhat tongue-in-cheek.  the whole book is a treat, but if you're new to personality psych, i especially recommend chapters 1 through 7.  felicity's professor at the university of new york used it, almost certainly because of it's engaging yet accurate prose.  this is a very serious footnote, i am not kidding even a little bit.  read it.

*** it is kind of like if your lab had to eat asparagus for six meals a week for several years, and then you realized you had several more years of compulsory asparagus-eating.  (except our pee don't stink.)

**** 100% risk

***** true story: my undergrad thesis examined how sex-role identity interacts with gender to predict attitudes towards women... among high school students... in two different countries.... one of which is Samoa, a remote, non-English-speaking island in the middle of the South Pacific.  

did someone say asparagus?