[DISCLAIMER: The opinions expressed in my posts, and guest posts, are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
Guest post by Shira Gabriel
Don’t go chasing waterfalls, please stick to the rivers and the lakes that you're used to.
I haven’t always been the most enthusiastic respondent to all the changes in the field around scientific methods. Even changes that I know are for the better, like attending to power and thus running fewer studies with more participants, I have gone along with grudgingly. It is the same attitude I have towards eating more vegetables and less sugar. I miss cake and I am tired of carrots, but I know that it is best in the long run. I miss running dozens of studies in a semester, but I know that it is best in the long run.
It is not like I never knew about power, but I never focused on it, like many other people in the field. I had vague ideas of how big my cell sizes should be (ideas that were totally wrong, I have learned since) and I would run studies using those vague ideas. If I got support for my hypotheses-- great! But when I didn’t, I would spend time with the data trying to figure out what went "wrong" -- I would look to see if there was something I could learn. I would look to see if there was some other pattern in the data that could tell me why I didn’t find what I predicted and perhaps clue me into some other interesting phenomena.
You know, like Pavlov trying to figure out why his saliva samples were messed up and discovering classical conditioning. That was me, just waiting for the moment when I would discover my own version of classical conditioning1.
I am going to be honest here: I love doing that. I love looking at data like a vast cave filled with goodies where one never knows what can be found. I love looking for patterns in findings and then thinking hard about what those mean and whether, even though I had been WRONG at first, there was something else going on – something new and exciting. I love the hope of discovering something that makes sense in a brand new way. It is like detective work and exploration and mystery, all rolled in one. I’m like Nancy Drew in a lab coat2.
Before anyone calls the data police on me, the next step was never publication. I didn’t commit that particular sin. Instead if I found something I didn’t predict, I would design another study to test whether that new finding was real or not. That was step two.
But this movement made me look back at my lab and the work we have done for the past 17 years3 and realize that although this has worked for me a couple of times I have also chased a lot -- A LOT -- of waterfalls that have turned into nothing. In other words, I have thrown a lot of good data after bad.4
And looking back, the ones that did work -- that turned into productive research areas with publishable findings -- were the ones that had stronger effects that replicated over different DVS right from the start.
When I chased something that wasn't as strong, I wasted huge amounts of my time and resources and, worse yet, the precious time of my grad students. That happened more than is comfortable for me to admit.
So, I think a big benefit for me of our new culture and rules is that I spend less time chasing waterfalls. My lab spends more time on each study (since we need bigger Ns) but we don't follow up unexpected findings unless we are really confident in them. If just one DV or one interaction looks interesting, we let it go for what it likely is -- a statistical fluke.
And we don't just do that because it is what we are now supposed to do, we do it because empirically speaking I SHOULD have been doing that for 17 years. I spent too much time chasing after patterns that turned out to be nothing5.
So I think my lab works better and smarter now because of this change.
As long as I am being so honest, I should admit that I miss chasing waterfalls. Just last week, one of my current PhD students6 and I were looking at a study that is a part of a really solid research program of hers that thoughtfully and carefully increases our science. And in her latest dataset, I felt the mist of a far-off possible waterfall in an unexpected interaction. Could this be the big one? It was tempting, but we aren’t going to chase the waterfall. As much as it seems fun and exciting, our science isn’t built on the drama and danger of waterfalls. To paraphrase the wise and wonderful TLC, I am sticking to the rivers and lakes that I am used to. That is how science advances.
Footnotes
- Still waiting, in case you were wondering.
- I don’t wear a lab coat. More like Nancy Drew in Yoga pant and a stained sweatshirt, but same difference.
- I am really old.
- Or is it bad after good? I can never remember which way it is supposed to go.
- How can you tell if an unexpected finding is a waterfall or classical conditioning? You can’t. But here are the four things I now look for: are the effects consistent across similar DVs; can we look back and find similar things in old datasets; do we have a sound theoretical explanation for the surprising findings; and, finally, can that theoretical explanation lead to other hypotheses that we can look at in the data. Only if a good chunk of that works out will we move on to collect more data. And yah, “good chunk” is not quantifiable. Sometimes Nancy Drew has to follow her instincts.
- Elaine Paravati. She rocks.
« July 2017 | Main | January 2018 »
October 2017
results blind vs. results bling*
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
show-offin many areas of science, our results sections are kind of like instagram posts. beautiful, clear, but not necessarily accurate. researchers can cherry-pick the best angle, filter out the splotches, and make an ordinary hot dog look scrumptious (or make a lemon look like spiffy car).** but what's even more fascinating to me is that our reaction to other people's results are often like our reactions to other people's instagram posts: "wow! that's aMAZing! how did she get that!"
i've fallen prey to this myself. i used to teach bem's chapter on "writing the empirical journal article," that tells researchers to think of their dataset as a jewel, and "to cut and polish it, to select the facets to highlight, and to craft the best setting for it." i taught this to graduate students, and then i would literally turn around, read a published paper, and think "what a beautiful jewel!"***
as with instagram, it's impossible to mentally adjust our reaction for the filtering the result could have gone through. it's hard to imagine what the rough cut might've looked like. it's hard to keep in mind that there could've been other studies, other measures, other conditions, other ways to clean the data or to run the analyses. and we never know - maybe this is one of those #nofilter shots.
in short, we can't avoid being blinded by shiny results.
what can we do?
there are a few stopgaps. for example, as an author, i can disclose as much as possible about the process of data collection and analysis, and the results (e.g., the 21 word solution). as a reader, i'll often pause when i get to the end of the method section and ask myself - is this study well-suited to the researchers' goals? would i think it should be published if i had to evaluate it just based on the method?
another partial solution is pre-registration, the #nofilter of scientific research. by pre-registering, a researcher is committing to showing you the raw results, without any room for touching-up (for the planned analyses - the exploratory analyses can be looked at from any and all angles). with a good pre-registration, readers can be pretty confident they're getting a realistic picture of the results, except for one problem. the editors and reviewers make their evaluations after seeing the results, so they can still consciously or unconsciously filter their evaluation through biases like wanting only counterintuitive, or significant, findings. so pre-registration solves our problem only as along as editors and reviewers see the value of honestly-reported work, even if it's not as eye-catching as the filtered stuff. as long as editors and reviewers are human,**** this will likely be a problem.
the best solution to this problem, however, is to evaluate research before anyone knows the results. this is the idea behind registered reports, now offered by Collabra: Psychology, the official journal of the Society for the Improvement of Psychological Science. an author submits their paper before collecting data, with the introduction, proposed method, and proposed analyses, and makes a case for why this study is worth doing and will produce informative results. the editor and reviewers evaluate the rationale, the design and procedures, the planned analyses and the conclusions the authors propose to draw from the various possible results. the reviewers and editor give feedback that can still be incorporated into the proposed method. then, if and when the editor is satisfied that the study is worth running and the results will be informative or useful regardless of the outcome, the authors get an "in principle acceptance" - a guarantee that their paper will be published so long as they stick to the plan, and the data pass some basic quality checks. the final draft goes through another quick round of review to verify these conditions are met, and the paper is published regardless of the outcome of the study.
registered reports have many appealing characteristics. for the author, they can get feedback before the study is conducted, and they can get a guarantee that their results will get published even if their prediction turns out to be incorrect, freeing them to be genuinely open to disconfirmation of their predictions. it's nice to be able to have less at stake when running a study - it makes for more objectivity, and greater emotional stability.*****
for science, the advantage is that registered reports do not suffer from publication bias - if all results are published, the published literature will present an unbiased set of results, which means science can be cumulative, as it's meant to be. meta-scientists can analyze a set of published results and get an accurate picture of the distribution of effects, test for moderators, etc. the only downside i can think of is that journals will be less able to select studies on the basis of projected citation impact - the 'in principle acceptance' means they have to publish even the findings that may not help their bottom line. call me callous but i'm not going to shed too many tears over that.
not everything can be pre-registered, or done as a registered report. for one thing, there's lots of very valuable existing data out there, and we shouldn't leave it hanging out to dry. for another, we should often explore our data beyond testing the hypothesis that the study was originally designed to test. many fascinating hypotheses could be generated from such fishing expeditions, and so long as we don't fool ourselves into thinking that we're testing those hypotheses when we're in fact just generating them, this is an important part of the scientific process.
the fact that we keep falling for results bling, instead of evaluating research results-blind, just means we're human. when you know the results are impressive, you're biased to think the method was rigorous. the problem is that we too easily forget that there are many ways to come by impressive-looking results. and even if we remember that filtering was possible, it's not like we can just magically know what the results look like unfiltered. it's like trying to imagine what your friends' unfiltered instagram pictures look like.
are we ready to see what science looks like without the filters? will we still get excited when we see it au naturel? let's hope so - to love science is to accept it for what it really looks like, warts and all.
* this title was inspired by a typo.
** although i'm using agentic verbs like "filtering" or "prettying up," i don't believe most of these distortions happen intentionally. much of the fun in exploring a dataset is trying to find the most beautiful result we can. it's hard to remember everything we tried, or what we might have thought was the best analysis before we knew how the results looked. most of the touching up i refer to in this post comes from researchers engaging in flexible data analysis without even realizing they're doing so. of course this is an assumption on my part, but the pretty large discrepancies between the results of pre-registered studies and similar but not-pre-registered studies suggests that flexibility in data analysis leads to exaggerated results.
*** words you'll never actually hear me say.
**** canine editing: not as effective as it sounds.
***** personality change intervention prediction: if registered reports become the norm, scientists' neuroticism will drop by half a standard deviation. (side effect: positive affect may also take a hit)
octopus vulgaris