open letter to editors

 

 

dear editors,

i love journals.  i love editors.  i love editors of journals.  that's why i want to help.  we need more quality control in our journals, and you are the ones who can do it. 

in july 2012, in a comment to a blog post, chris fraley wrote 'What we might need, in other words, is a formal “consumer reports” for our leading journals.' i was so excited by this idea that i wrote to him and told him it was the best actionable idea that has come out of the replicability discussion.* fast forward 27 months, and our paper, 'N-Pact Factor: Evaluating the quality of empirical journals with respect to sample size and statistical power' is out.

you can read the actual paper here, and you can read chris's blog post about it here.

the N-Pact Factor (NF) has many flaws.  it is not a perfect measure of quality.  but it is better than nothing (or better than the Impact Factor alone).  if you want to improve on the NF, please do.  we hope you will.  i would love to see a world in which there are many indices of journal quality, all tapping into various facets of quality.  in my ideal world, i could look up not only the average sample size of studies published in a journal, but also how often they publish studies with multiple methods, field studies, non-student samples, longitudinal designs, actual behavior, replications, open data, preregistration, author disclosures, etc.  but if i can know only one thing about the studies published in a journal, it would be their sample size. 

so this is a call to editors to please please please pay attention to sample size when evaluating a paper.

why am i so obsessed with sample size?  in short, because it is shorthand for 'amount of information'.  and ultimately, that's what we're in the business of producing: information/knowledge. larger samples lead to more accurate conclusions. they reduce type II error, and indirectly reduce the proportion of published findings that are likely to be type I errors (read the paper for full explanation).  in short, large samples mean fewer mistakes. *** (also, for a great summary of the problem with small sample sizes, see my favorite spsp talk ever.)

below i address some potential objections to this obsession with sample size, and make some controversial statements.  i speak only for myself here.

how much is enough? one problem with saying 'we need bigger samples' is that it's hard to know how much is enough.  i agree this is a problem.  it has an easy solution: 200.

let's start with 200.  it might turn out not to be enough****.  but there are some good reasons to start there.  first of all, a sample size of 200 gives you 80% power to detect an effect size of r = .20 (d = .40), which is about the average published effect size in social and personality psychology*****.  also, this. and sanjay's 2013 arp talk.  (you had to be there******).

what about power analyses?  you don't need power analyses.  if there is an effect, it will probably be somewhere in the ballpark of all the other effects in social and personality psych.  so just go with 200 (or 250.  or 300.).  my view on power analyses is we can do one for the entire field, and i just did it, and it told me we need 200 people (or 100 per condition).  there, we're done with power analyses. (yes, there are exceptions, blah blah blah.)

but i can't afford to run 100 participants per condition. then don't do the study.

but seriously, if you are doing a typical social/personality study, that is, you are running college student participants in a lab and they are doing some stuff on computers and maybe talking a little bit and eating some chocolate, you can run 100 people per condition.  if you can't, i'm sorry.  i feel bad that you can't.  but that doesn't mean we should publish your paper.

if you are not doing a typical social/personality study, that is, if you have non-college-student participants*******, or you are using intensive or expensive methods, or studying something rare, etc., then we should absolutely take that into consideration and be flexible about sample size.  each editor needs to weigh the value of the evidence, which includes how important it is and how hard it would be to collect more data. 

my recommendation for editors: if a study has fewer than 200 people, the authors should give a convincing reason why it should be published anyway.  there are many potentially good reasons, and editors need to use their judgment all the time anyway.  make this part of your evaluation.

but this will take forever! yup. there will be fewer papers to read. and review.  and for the media to sensationalize.  what a terrible, terrible loss. 

in my view, increasing sample sizes kills two birds with one stone.  (actually i think it kills about eighteen birds.  in the good sense.) it improves the quality of our published findings, and it address the looming crisis of The Dearth of Reviewers********.

in conclusion. every time i see a paper with a simple design and 36 participants, i die a little inside.  the point of the NF is to encourage editors (and reviewers, and ultimately authors) to place more value on sample size.  don't tolerate small samples, unless there is a good reason to.  let's push ourselves and each other to do the hard work of doubling or tripling our samples.  it is painful.  it sucks.  it's not fun.  but we can't go on like this.  and i think in the end we may actually come to enjoy living in a world where there are only 228 new papers to read each month.  all of us can help make this happen, but you, editors, can do the most.

xoxo,

simine

* i also wrote 'it wouldn't be hard to do, right?' i crack myself up.

** not literally 'we'.  thank you yuchen and ashley.

*** sample sizes do not decrease all kinds of mistakes (i.e., systematic error is not reduced), but they reduce random error and they don't increase other kinds of error.

**** i hope to one day look as foolish as simmons, nelson, and simonsohn now look for their n = 20 recommendation in 2011.

***** note that this means that half of all published effects are smaller than this.  which means you should probably assume the effect you are studying is smaller than this.  which means 200 people is probably not enough.  i am starting to feel foolish already.

****** happily, you can attend the 2015 arp conference.  i am almost positive sanjay will be there.

******* i don't mean mturk.  if you are using plain old mturkers, you better not have fewer than 200 people or i'll get really mad.

******* soon to be a major motion picture in which editors go insane and start stalking people who submit 42 manuscripts a year and turn down all review requests.  starring mindy kaling.

 

mindy kaling is not impressed.