Enter your Email:
Preview | Powered by FeedBlitz

« Guest Post: A Tale of Two Papers | Main | give the gift of data »

Comments

Michael Inzlicht

I totally agree with footnote 4 (and disagree with Neuroskeptic here). If your worst results mean your data are inconclusive, I guess that means they might not be published, at least not in a top-tier journal. Yes, I understand that we want a complete record of all the data out there, but I'm not yet convinced that all data need to be published. Don't get me wrong, some inconclusive data NEED to be published, especially if they disconfirm previously held beliefs or if the dependent variable (e.g., mortality) if very important. But I'm still not convinced that all inconclusive results are publishable. Great post!

Chris

I believe that the "most damning result" question was put to the GOP presidential candidates in a debate. It did not result in useful information. Scientists are likely to behave in the same manner; why would they go out of their way to make themselves "look bad?"

Jennifer

You know, it's always so easy for professors like you to say these things from your high horse. Then you get applause from all other distinguished professors who think you are doing a fantastic job and making the world a better place. It's one giant circle-jerk.

In the end, you are all focusing on treating the symptoms and not the disease. The focus now is on decreasing questionable research practices by requiring pre-registration, posting data, etc. Those are the symptoms. None of you, none, ever discuss the reason why people engage in such practices. That is the disease. And that disease is the incredibly crushing pressure there is to publish articles. It is no wonder that people cave and start behaving unethically, when getting a job requires X publications. Of course people will cheat to get there! Until that specific issue is addressed researchers will find ways to work the system. This I absolutely guarantee you. You can require pre-registration and someone will find a way around it. It's just a matter of time. I am certainly not saying that pre-registration is not useful, but it is not the vaccine. It is the Tylenol. How to treat the disease requires a much more provocative discussion.

It is reviewers like you, who assume the worst of people, who write mean and unhelpful comments that propagate this vicious cycle.

I will leave a fake email address because I know if I left my real name, it would affect the way reviewers like you view my submissions. That is a fucked up system.

Simine Vazire

Hi Jennifer,

Thanks for your comment. I completely agree with you that ultimately we need to treat the disease, which means changing the incentive structure and the way we evaluate people for jobs and promotion. I do think I have spoken out about this, on my blog and in talks, but I agree that's not enough. I also try to emphasize the importance of looking beyond quantity when I am on search committees and when I write tenure letters. Finally, I hope that being involved in the peer review process as editor and reviewer can also help treat the disease, by making it easier for people who do the slow, meticulous kind of science we'd all like to see more of publish in good journals. I am sure there is more I and people in positions like mine could do, and I would be very interested in hearing ideas about this.
Many of us do care about the unrealistic expectations that junior scholars face, even if we are not always doing enough about this. Here are some papers and blog posts that have touched on these issues:
https://pps.sagepub.com/content/7/6/615.full
https://opim.wharton.upenn.edu/~uws/papers/fewer.pdf
https://psych-your-mind.blogspot.com/2013/06/quality-v-quantity-in-publication.html#more
https://onlinelibrary.wiley.com/doi/10.1111/spc3.12166/abstract (paywall, sorry)
https://sometimesimwrong.typepad.com/wrong/2014/03/having-it-all.html

Although I agree with much of what you write, I am not sure how reviewers who ask for more transparency are propogating this vicious cycle. Perhaps I am naive, but I think reviewers who do this are doing a service to those researchers who want to be transparent and do research without cutting corners, by rewarding that kind of research and holding everyone to that standard. I agree that mean comments in reviews are never a good idea, but we can't equate skepticism with meanness. In my view, skeptical reviewers who push for transparency are part of the solution.

But the bottom line is, I agree the system is messed up. I want to help fix it. I am eager to hear ideas about how we can do this better.
Best,
Simine

Chris Chambers

Nice post Simine. My only comment is that by presenting one's ugliest and most damning results one also dramatically increases one's chances of the paper being rejected. We've all seen how reviewers gleefully expose "weak" or "unconvincing" findings and that is enough to resoundingly kill one's chances with a prominent journal. As Jennifer says above, this is a cost most heavily felt by junior researchers.

The answer, in my view, is to submit papers to a format that doesn't assess the publishability based on results - and that of course (as you know) is Registered Reports. https://osf.io/8mpji/wiki/home/ If our goal is to change the incentive structure we need to change the power structure, and RRs declaw reviewers from sinking papers because they find the results weak, unconvincing, ugly (insert descriptor) etc.

As an editor for RRs at a number of journals, I've seen just how biased peer review actually is. It is becoming thematic that reviewers decide that the method of a pre-registered study was flawed because the reviewer disliked the results, even when the reviewer explicitly *approved* these exact methods before results existed. With RRs, the author is protected against such biased decision making - manuscripts cannot be rejected on these grounds - but with conventional review authors are fully exposed and are easy prey.

That's why I feel that your suggestion of presenting one's ugliest results, while absolutely correct from a scientific and ethical point of view, will never ever (ever ever ever) happen under traditional articles formats at prominent journals, where more than anything else, the authors need a good story.

And the moral (for me) is -- everyone get pre-registering and supporting Registered Reports and together we can fix this problem.

Dave Geyer

I would agree with the point that Jennifer is making, but might take it one step further. Individuals whom are drawn to academia are generally very achievement oriented, which can and does influence how they interpret and present data. I would therefore argue that there is as much pressure coming from within the individual as there is from the system.

Also, from a philosophical standpoint, most things in this world are a mix of strengths & weaknesses, and attempts to correct one weakness in a system often led to another. As a result, it will always be an imperfect process, no matter how you structure it. I have also been surprised on occasion by successes that have occurred, in spite of the poorly structured systems that guide them (i.e. a beautiful mess).

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)