nothing beats something

[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

hello

i'm reading a book called Nothing is True and Everything is Possible, about Russia, and the name keeps haunting me (the content of the book is good, too).  sometimes i worry that this goes for science, too.  it's not just that when nothing is true, everything is possible.  but when everything is possible, nothing is true.  

sometimes studying human behavior is so wild and messy that it feels like anything goes.  we create ad hoc scales more often than we probably should, and we invent manipulations out of thin air, rarely pausing to validate them.  if the best we can do is a college student sample, that sin will be forgiven.  if we can't get a behavioral measure of a behavior, a self-report will often do.  we do what's possible because, well, what's the alternative?

i'm here to consider the exceedingly unpopular view that the alternative - to do nothing - is sometimes preferable to doing what's possible.

science is hard.  social science can be especially hard, because we usually need to study humans, and humans are a bitch to study.*  there's this idea out there that it's better to collect noisy, messy data than no data at all.  i've often repeated this claim myself.  indeed, i taught it when i used David Funder's excellent personality textbook.  Funder's third law** is that "something beats nothing, two times out of three."  Funder was smart to add the caveat, because i think sometimes nothing beats something.

slowly over the last few years, i've come to the conclusion that sometimes crappy data is actually worse than no data.  i think that we sometimes fall into the trap of thinking that, because a phenomenon is really, really, really hard to study, very flawed studies are the best we can hope for and so are ok.  it's almost like we are committed to the belief that everything must be possible for a reasonably industrious researcher to study, so if the only option is a bad study, then a bad study must be good enough.

let me be clear: i have definitely made this argument myself, possibly as recently as last month.  i'm no stranger to the feeling that just getting adequate data feels almost impossible, and so inadequate data must do.  for example, i was pretty proud of myself when my lab decided to not just code actual behavior from EAR recordings of >300 participants, and not just double-code the recordings, but TRIPLE code the recordings.  when, after two years of coding by over 50 coders (which came on the heels of several years of data collection), we checked the reliabilities on those triple codings and found them wanting, it was very, very tempting to say "well, it's the best we can do."  luckily i had learned about the spearman-brown prophecy formula, and so had no excuse - adding more raters was definitely going to help, it's a mathematical fact.  so we got three more coders per recording (which took about two more years and 50 more coders).  and let me tell you, it sucked.  we are sick of coding.***  we wish we could have spent those two years doing new projects.  but we got the reliabilities up, and that will let us get much better estimates.  science is not all creativity and innovation.  it is a lot of boring, hard work.  if you don't sometimes hate science, you might be doing it wrong.

even better examples of people who kept going when most of us would have said "good enough:" the two researchers i highlighted in my blog post, "super power."  their efforts really make me rethink my definition of impossible.  

but sometimes getting good data really is impossible, at least for any single lab.  and i would like to float the idea that when this is the case, there may be something noble in walking away, and choosing to do nothing rather than something.  sometimes, the nicest thing you can do for your research question is to leave it unanswered.  (my colleague wiebke bleidorn pointed out that this is called "addition by subtraction.")  if you insist on extracting information from really crappy data, you put yourself at really high risk**** of  reading patterns into noise.  just because you need your data to tell you something, to move the needle, doesn't mean it can.  if our field allowed people to get credit for publishing little bits of (really hard-to-collect) data without making any claims whatsoever, this might be a viable approach, but i don't think that currently exists (though there are low-recognition ways to do this, of course).

the wishful thinking that we can always extract some knowledge from even small bits of messy data can lead to serious, widespread problems.  it's easy to point fingers at other fields, but this problem is alive and well in psychology.  i'm very familiar with the argument that this sample size or that method must be good enough because it's all we can realistically do, given limited resources, hiring or promotion expectations, etc..  it used to be one of the most common responses i'd hear to calls for larger samples or better (i.e., harder) methods.  people seem to have backed off of making the argument out loud, but i think it's still very common in people's minds - this attitude that it's inappropriate to say a method or design isn't good enough because it was hard/expensive/time-consuming to do.  (indeed, i think this is the most common internal response most of us have when we get criticized for studying only WEIRD samples.)

here's what i wish we all did (including me):  1) ask yourself what the ideal study would be to test your research question.  2)  ask yourself if you're willing and able to do something pretty close to it.  3)  if not, ask yourself why not.  really push yourself.  don't let yourself off the hook just because it would be a lot harder than what you're used to doing.  if the research question is important, why isn't it worth the really hard work?  4)  if you're still not willing or able to do something close to the ideal study on your own, either a) move on to a different (or narrower) research question, or b) join forces with other labs.  

i know this is idealistic.  i know we have to keep publishing otherwise our food pellets stop coming.  so let's pick questions we can actually hope to answer with the resources we have.  (this post is really one big, public NOTE TO SELF.  i have been extremely guilty of biting off way more than i can chew,***** and want to learn to scale back my expectations of what kinds of research questions i can rigorously test on my own.)  or join collaborative projects that pool resources to tackle the really hard, important questions, and find a way to deal with the issue of spreading credit around.  if we stop trying to do the impossible, i think we'll find that more things are true.

* also, sometimes to interact with.  or be near.  

** Funder calls his lessons "laws" somewhat tongue-in-cheek.  the whole book is a treat, but if you're new to personality psych, i especially recommend chapters 1 through 7.  felicity's professor at the university of new york used it, almost certainly because of it's engaging yet accurate prose.  this is a very serious footnote, i am not kidding even a little bit.  read it.

*** it is kind of like if your lab had to eat asparagus for six meals a week for several years, and then you realized you had several more years of compulsory asparagus-eating.  (except our pee don't stink.)

**** 100% risk

***** true story: my undergrad thesis examined how sex-role identity interacts with gender to predict attitudes towards women... among high school students... in two different countries.... one of which is Samoa, a remote, non-English-speaking island in the middle of the South Pacific.  

did someone say asparagus?
 

Comments

[Marcus Crede]

Thanks for writing this! This - like many of your other posts - will be required reading for the grad students in the methods course I teach this fall.

[Rob]

One of the numerous examples when bad methodology kills a whole paper (in this case it at least has received a critical analysis):
https://pubpeer.com/publications/B052DE668A4DA0B8D5644343BEE1D4

[Anonymous]

"or join collaborative projects that pool resources to tackle the really hard, important questions, and find a way to deal with the issue of spreading credit around. "

I am a bit worried that large-scale "collaborative" projects like the Psychological Science Accelerator might not be a very good idea due to several reasons. One of which might be that way too many resources are possibly unnecessarily spent.

Also see: "More data is always better but enough is enough"
http://daniellakens.blogspot.com/2014/12/more-data-is-always-better-but-enough.html

(side note I: perhaps the 1st comment on Lakens' blogpost there could point to an interesting way to possibly investigate when "enough is enough". I am not skilled to do that myself, but i still think it could be easily done by someone interested in these issues like yourself or readers of this relevant blogpost, and could perhaps even be considered to be intersting/useful enough to warrent a paper or blogpost)

(side note II: thank you for pointing to Studyswap as i reason that could be used in a way that will be a) truly collaborative, b) not entail many (possible) problematic issues like the Psychological Science Accelerator potentially has, and c) a much better way to accelerate psychological science)

[Anonymous]

"I am a bit worried that large-scale "collaborative" projects like the Psychological Science Accelerator might not be a very good idea due to several reasons."

For some of these (possibly) other problematic issues with the "Psychological Science Accelerator" see here:

http://andrewgelman.com/2018/05/08/what-killed-alchemy/#comment-728154

Side note: as far as i can reason, StudySwap doesn't have any of these (possibly) problematic issues.

[Anonymous]

For some of these (possibly) other problematic issues with the "Psychological Science Accelerator", and a place to discuss them, possibly also see here:

http://andrewgelman.com/2018/06/26/psychological-science-accelerator-probably-good-idea-im-still-skeptical/

[No One]

But the incentives are to publish your creative ideas and designs first! Let someone else come along later with the resources to actually conduct the rigorous, ideal experiment, and you'll get the citations and career boost

[Anonymous]

"But the incentives are to publish your creative ideas and designs first! Let someone else come along later with the resources to actually conduct the rigorous, ideal experiment, and you'll get the citations and career boost"

The "incentives", lol. Can you point me to actual research about if, why, and how these "incentives" actually influence(d) things? I mean there must have been done lots, and lots, of research into "the incentives" because it seems to be all i hear about. Wouldn't that be useful to investigate before coming up with all kinds of "solutions" that are supposed to influence all these "incentives".

The problem i have with talking about "incentives" is that it is an indirect term for other things that in turn could stop you from 1) really thinking about things, and 2) could therefore result in coming up with an entire new set of "incentives" and/or "solutions" which may not even be good for science.

Just take the Psychological Science Accelerator, and compare it to the alternative of small-group collaborations, in the following discussion and try and spot where they might differ, and how, and why, that could be useful if you want to improve Psychological Science:

http://andrewgelman.com/2018/06/26/psychological-science-accelerator-probably-good-idea-im-still-skeptical/