[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
show-offin many areas of science, our results sections are kind of like instagram posts. beautiful, clear, but not necessarily accurate. researchers can cherry-pick the best angle, filter out the splotches, and make an ordinary hot dog look scrumptious (or make a lemon look like spiffy car).** but what's even more fascinating to me is that our reaction to other people's results are often like our reactions to other people's instagram posts: "wow! that's aMAZing! how did she get that!"
i've fallen prey to this myself. i used to teach bem's chapter on "writing the empirical journal article," that tells researchers to think of their dataset as a jewel, and "to cut and polish it, to select the facets to highlight, and to craft the best setting for it." i taught this to graduate students, and then i would literally turn around, read a published paper, and think "what a beautiful jewel!"***
as with instagram, it's impossible to mentally adjust our reaction for the filtering the result could have gone through. it's hard to imagine what the rough cut might've looked like. it's hard to keep in mind that there could've been other studies, other measures, other conditions, other ways to clean the data or to run the analyses. and we never know - maybe this is one of those #nofilter shots.
in short, we can't avoid being blinded by shiny results.
what can we do?
there are a few stopgaps. for example, as an author, i can disclose as much as possible about the process of data collection and analysis, and the results (e.g., the 21 word solution). as a reader, i'll often pause when i get to the end of the method section and ask myself - is this study well-suited to the researchers' goals? would i think it should be published if i had to evaluate it just based on the method?
another partial solution is pre-registration, the #nofilter of scientific research. by pre-registering, a researcher is committing to showing you the raw results, without any room for touching-up (for the planned analyses - the exploratory analyses can be looked at from any and all angles). with a good pre-registration, readers can be pretty confident they're getting a realistic picture of the results, except for one problem. the editors and reviewers make their evaluations after seeing the results, so they can still consciously or unconsciously filter their evaluation through biases like wanting only counterintuitive, or significant, findings. so pre-registration solves our problem only as along as editors and reviewers see the value of honestly-reported work, even if it's not as eye-catching as the filtered stuff. as long as editors and reviewers are human,**** this will likely be a problem.
the best solution to this problem, however, is to evaluate research before anyone knows the results. this is the idea behind registered reports, now offered by Collabra: Psychology, the official journal of the Society for the Improvement of Psychological Science. an author submits their paper before collecting data, with the introduction, proposed method, and proposed analyses, and makes a case for why this study is worth doing and will produce informative results. the editor and reviewers evaluate the rationale, the design and procedures, the planned analyses and the conclusions the authors propose to draw from the various possible results. the reviewers and editor give feedback that can still be incorporated into the proposed method. then, if and when the editor is satisfied that the study is worth running and the results will be informative or useful regardless of the outcome, the authors get an "in principle acceptance" - a guarantee that their paper will be published so long as they stick to the plan, and the data pass some basic quality checks. the final draft goes through another quick round of review to verify these conditions are met, and the paper is published regardless of the outcome of the study.
registered reports have many appealing characteristics. for the author, they can get feedback before the study is conducted, and they can get a guarantee that their results will get published even if their prediction turns out to be incorrect, freeing them to be genuinely open to disconfirmation of their predictions. it's nice to be able to have less at stake when running a study - it makes for more objectivity, and greater emotional stability.*****
for science, the advantage is that registered reports do not suffer from publication bias - if all results are published, the published literature will present an unbiased set of results, which means science can be cumulative, as it's meant to be. meta-scientists can analyze a set of published results and get an accurate picture of the distribution of effects, test for moderators, etc. the only downside i can think of is that journals will be less able to select studies on the basis of projected citation impact - the 'in principle acceptance' means they have to publish even the findings that may not help their bottom line. call me callous but i'm not going to shed too many tears over that.
not everything can be pre-registered, or done as a registered report. for one thing, there's lots of very valuable existing data out there, and we shouldn't leave it hanging out to dry. for another, we should often explore our data beyond testing the hypothesis that the study was originally designed to test. many fascinating hypotheses could be generated from such fishing expeditions, and so long as we don't fool ourselves into thinking that we're testing those hypotheses when we're in fact just generating them, this is an important part of the scientific process.
the fact that we keep falling for results bling, instead of evaluating research results-blind, just means we're human. when you know the results are impressive, you're biased to think the method was rigorous. the problem is that we too easily forget that there are many ways to come by impressive-looking results. and even if we remember that filtering was possible, it's not like we can just magically know what the results look like unfiltered. it's like trying to imagine what your friends' unfiltered instagram pictures look like.
are we ready to see what science looks like without the filters? will we still get excited when we see it au naturel? let's hope so - to love science is to accept it for what it really looks like, warts and all.
* this title was inspired by a typo.
** although i'm using agentic verbs like "filtering" or "prettying up," i don't believe most of these distortions happen intentionally. much of the fun in exploring a dataset is trying to find the most beautiful result we can. it's hard to remember everything we tried, or what we might have thought was the best analysis before we knew how the results looked. most of the touching up i refer to in this post comes from researchers engaging in flexible data analysis without even realizing they're doing so. of course this is an assumption on my part, but the pretty large discrepancies between the results of pre-registered studies and similar but not-pre-registered studies suggests that flexibility in data analysis leads to exaggerated results.
*** words you'll never actually hear me say.
**** canine editing: not as effective as it sounds.
***** personality change intervention prediction: if registered reports become the norm, scientists' neuroticism will drop by half a standard deviation. (side effect: positive affect may also take a hit)
octopus vulgaris
Comments