this blogging thing is pretty rad. (also: twitter. wow.)
today's topic: having it all. i'm not talking about the work/life problem. i'm talking about the sample size/methodological rigor problem.
let's start with sample size. by now, everyone knows that bigger is better. you can't have too large a sample. there is no 'double-edged sword'. there are no downsides to a large sample. more evidence is always better, and larger samples = more evidence.
this seems very obvious but i've seen at least three different editors criticize manuscripts for having samples that are too big. so i want to be very clear: there is no such thing as a sample that is too big. calling a sample too big is like calling a a grizzly bear too cute. it makes no sense.
many people have written very compelling explanations about why we should want larger samples (more power). i will trust that you have read those.
what i want to talk about is the downside of large samples.
ok, i know i just said there are no downsides, but that's once the data have already been collected. if you are an editor or a reviewer, and you are reading a manuscript with a sample size of, say, 884,328, you should consider this a major asset. it's not complicated.
but let's say you're a researcher, designing a study. you've heard large sample sizes are awesome. you have $1,000 to spend. what should you do?
one of my biggest fears in life is that people will accept the large-samples mantra unthinkingly, and decide to run all their studies on mturk, where they can get 10,000 participants for their $1,000. my other biggest fear is spiders.
over the last thirty years, my field (personality psychology), has finally grown out of its over-reliance on self-reports. it is now practically impossible to publish a cross-sectional study of college students that relies exclusively on self-reports. and that's probably as it should be. we cannot build our knowledge of human behavior on self-reports, reaction times, and vignette studies alone (experimental philosophers, take note).
however, now that people are catching on to the fact that most of our studies have samples that are too small, people are turning to mturk to increase their sample sizes. this is a nice sentiment, and there are some great things about mturk (and some not so great...). but there is a problem: it is exceedingly difficult to collect actual behavioral measures, informant reports, or physiological measures from mturk samples. and we need those measures. how can we study morality, love, stress, leadership, or any other interpersonal phenomenon if we only have the self's perspective?
we need good methods. that means we need multiple methods, and each one needs to be implemented rigorously. this is very expensive and time consuming. multiply that by many many participants, and it feels like we have to make a sacrifice. we can't have it all.
but we can. there is one ridiculously simple solution: slow down.
it's true that we cannot have both large samples and diverse, intensive methods if we want to continue running billions of studies per year. but there is another model. run fewer, better studies.
despite my skeptical exterior, i have a wildly idealistic streak. i don't like to compromise. so i believe that we can have large, beautiful studies with multiple, reliable, valid methods. we just have to value them. journals need to bend over backwards to publish those studies, and all of us need to treat those studies as much more definitive than small, mono-method studies. they should be the holy grail of psychological research.i understand the pressure to do things quickly. i know that calling on people to collect larger samples necessarily puts pressure on them to use cheaper, quicker methods. but we should not easily give in to the notion that we must choose one or the other. we can have it all. we just have to be patient.
**photo credit: erik pettersson.