reviewer 2 is not buying it.
i've had a blog post stuck in my head* for a few months now, and the new post on datacolada.org is finally spurring me to write it.
i'll admit it. i'm reviewer 2. i'm the one who doesn't quite believe that you're telling the whole story, or that things are as simple and neat as you make them out to be. that's because we've all been taught to pretty things up (pp. 8-10). it sucks for authors that reviewers are super skeptical, but it's the price we are now paying for all those years of following bem's advice. there are going to be more and more reviewer 2s out there. i'm pretty sure most people have already run into the skeptical reviewer who suspects you had other measures, other data, or that you didn't plan those covariates a priori. if you haven't yet, it's just a matter of time.
there's nothing personal about it. the reason reviewer 2 thinks that you're telling the best possible story you could extract from your data is because why wouldn't you? that's the game we were taught to play, those were the rules until now. so when reviewer 2 asks you to disclose all flexibility in your data analyses, to share your data, to describe all your measures, etc., she isn't accusing you of anything other than having been trained in a pre-revolution era. it's not rude, it's just healthy skepticism.
when i read a paper, i assume the authors are showing the most beautiful result they could get, unless they give me a reason to think otherwise. the fantastic thing about pre-registration is that it's an almost foolproof way of convincing readers and reviewers that you're being honest about what was planned and what wasn't. if you can pre-register, that is definitely what you should do, for all the reasons described in the datacolada post and more.**
what if you can't pre-register? what if the data are already collected? or what if you pre-register one thing, but then you get a new idea about what you can do with the data after you collect them? in other words, when you want to publish something that was in fact exploratory, what's the best you can do to assuage reviewer 2 (and all the skeptical readers)?
first, acknowledge that any solution is second best to pre-registration. as someone who has never pre-registered anything and possibly never will,*** i think it's important to admit this. the fact that some research is really hard to pre-register doesn't mean that it wouldn't be better if it was pre-registered. just like even though i will probably continue to use college student participants, i don't pretend that my research wouldn't be better if i used a more diverse sample. i am perfectly comfortable acknowledging that a certain practice is ideal, and that i lack the resolve to do it. it's called hypocrisy and i'm sticking with it.
second, be transparent. if you want to convince readers that you're not selling them a touched-up version of your results, show them the flaws.**** tell the reader what other analyses you tried, what other studies you ran, how the results look with different decision rules about outliers and covariates, etc. in other words, write paper two in mickey's tale of two papers.
what if being totally transparent takes up too much space? one solution is to put some stuff in an appendix or supplementary materials. i'm not against that, but i don't think it's a good enough solution because most readers won't notice it, so it still ends up letting authors get away with pretending the touched-up version is the whole truth. so, i came up with an additional solution, a new subsection for every results section: your most damning result.
i am pretty enamored of this 'most damning result' idea. i would love to make it a standard section of every paper. i think it would help readers evaluate the strength of your evidence - if you can still convince readers that you have strong evidence for your conclusion, even after showing them the most damning result, that's pretty impressive. like getting a date after posting your best and worst selfie.
i think the "most damning result" section would also encourage us to be more skeptical of our own results. probably most of us have no problem writing off our most damning result as not informative because, in retrospect, of course we shouldn't have expected that manipulation to work/that item to make sense/that cover story to be convincing. but having to at least write up that result like any other might give us some pause. if we look at touched-up data too long, we start to forget what real data look like, and being forced to stare at the blemishes could do us some good.
if you can pre-register, pre-register.
if you can't, be transparent, and show us that you're telling the whole story. like by highlighting your ugliest, most damning result.
* i think it's responsible for my recurring ear infection.
** here's another: if you can get your pre-registered study to work with 25 people per condition,***** i won't harass you about your sample size.
*** i know, i'm a bad, bad person.
**** of course, if showing all the flaws makes your results inconclusive, it's a sign that you should probably go do some more research before writing anything up. transparency is not a silver bullet - if the data don't give a clear answer, transparency won't save you (and will, in fact, make it harder for you to publish your result. but that's a good thing. for science. not for you.)
***** good luck with that.
the nice reviewer. (don't try to pet him.)