Enter your Email:
Preview | Powered by FeedBlitz

« and now for something a little more uplifting | Main | is this what it sounds like when the doves cry? »

Comments

Julia

As a grad student, I have to add: I'd rather know about QRP, sample size and statistical limitations and everything else that went wrong in the past now than in 2-3 years, at the end of my thesis. it might not make it easier to design and run studies, but their contribution to science might hold up longer. I prefer contributing a little bit in high quality than something that sounds great but will never hold up. Will I be able to stay in research with fewer studies and potentially less seductive results? I don't know. As before, the only way to find out is to go ahead and try. But I will enjoy the time that I have!

Question

"The same goes for our research practices. it might be tempting, when we're told that social/personality psychology research actually requires not 60 but more like 600 participants per study,** that the world just got a little bit uglier. it might be easy to think of the counterfactual: i wish i could still do informative research with 60 participants. "

I am trying to figure out when a study is appropriately/highly powered, but i have a hard time doing so. I keep reading that we need bigger samples, but exact numbers or computations are often missing. Apparently looking at available effect sizes might still lead to low-powered studies because these are often inflated. So what am i to do when designing my study?

Would it be possible for someone smart to write a clear and informative article on optimal sample sizes-benchmarks for certain research designs given "all that we've learned over the past years"?

Mark Andrews

"people had been screaming and shouting about statistical power for a while, but we needed the missing piece of the puzzle, p-hacking, to fully understand why power was such a big problem...."

Even in the absence of p-hacking, low power is still a problem.

The probability that the null hypothesis is True, given a statistically significant effect, is dependent on the power of the test (as well as the prior probability of the null):

P(Null|Significant)
=
P(Significant|Null)P(Null)
/ (P(Significant|Null)P(Null) + P(Significant|not Null)P(not Null))


This paper explains it nicely:
http://www.nature.com/nrn/journal/v14/n5/pdf/nrn3475.pdf

Anne

I'm happy about living in exciting times for the same reasons as you are, and I am happy about science changing for the better.
But I still feel horrible. Just horrible. I have learned all these bitter truths (which you don't call bitter and you're probably right) and I want to incorporate them into my work to do meaningful research.
But I am a small PhD student in an environment that is convinced about n=20/cell being a really good sample size (actually it's considered borderline greedy) for extremely noisy infant research. I simply can't do it right at the moment. I don't get the time and the funding necessary to do it right. I try to compensate with Bayes factors that only tell me I don't have enough data.
I've never felt so much cognitive dissonance (if that effect still holds) in my life. I feel that what I do is really nothing but throwing taxpayer's money out the window. It's almost physically painful.

I certainly don't want my old blissful ignorance back. But I am depressed, and right now I don't see a way out. It's pretty ugly.

Kristin Bain

Several of my colleagues have mentioned being depressed by the replicability crisis and the current state of science but I, like you, have actually been pretty happy with the current trend. Maybe it is my sunny disposition (mostly powered by caffeine) or maybe it's the fact that I'm joining the scientific community at a time when it is fighting to become its "ideal self." I'd much rather join now, when so many are banding together to bring to light the failings of our past and communally find ways of improving, than to join while science was still proceeding as if p-hacking and all the other tricks and hoops scientists have been jumping through for years were still the norm. At least this way I get to keep my ideals and my naive belief that doing science right could still lead to a pay-off. And think of all the new stuff we get to discover because what we thought was discovered turned out to be not discovered! Personally, as a first year graduate student, I think there could be worse times for joining the scientific community that the current era. Thanks for acknowledging the positives!

The comments to this entry are closed.