Enter your Email:
Preview | Powered by FeedBlitz

« on ferguson | Main | open letter to editors »

Comments

Brett

Really like the use of #RepliGate. Can I ask where it came from? :D

Like the analysis on conceptual and direct as well. Also the parts about needing to test more varied phenomenon and stimuli sets.

Best,
Brett

Richard A. Lawhern, Ph.D.

There is a tired but sometimes true cliche to the effect that "liars figure and figures lie." To that we might add "wishful or self-interested authors lie too but do not allow themselves to know that they do." It seems to me that psychology and psychiatry are both pl;now victims of a few dissemblers, and far more than a few blinded professionals suffering from an over-abundance of the need to be considered wise or unique.

We probably already know much of what's needed to correct known biases in research: protocols published before trials or analysis, required publication of outcome summaries on all investigations, and unrestricted access to raw data by critics. Yes, this is a "hard" standard. But the alternative may be to give up on science and become shamens, each of us with an unconfirmed opinion.

Sincerely,

Highly Adequate

I don’t think this post really comes to terms with the extent of the problem with replication in social psychology.

The final word in the post basically appeals to the value of meta-analyses to demonstrate effects of interest, such as priming.

But the value of meta-analyses hangs entirely on the legitimacy of the component studies over which it ranges. If those component studies are based on bad methodology (p-hacking, poor study design or implementation, etc.), or, worse, fraud of some kind or another (which has been demonstrated in certain cases – e.g. Stapel), or are as a whole subject to publication bias or file drawer effect, then the basic assumptions of the meta-analyses are violated. Under those conditions, the meta-analyses really tell us nothing. This is emphatically so when we have no good idea at all as to the extent of the problems separately or together.

The only sensible approach is to treat all such results as suspect at this time, and to refuse to draw inferences that require them. What we really don’t know, we really don’t know, and there's no use pretending otherwise.

I would include the supposed phenomenon of stereotype threat as likewise doubtful. Stereotype threat is a subset of priming, and has also failed quite often to be replicated. And it is perhaps even more suspect precisely because so many people desperately want to believe it’s true and important. Whatever incentives might exist for bad methodology and worse in the case of priming more generally can be found in spades with regard to stereotype threat.

Shen-yi Liao

I've written a response to an aspect of Doris's post at the Experimental Philosophy blog. I would definitely appreciate non-philosophers' perspective on what I've said there! http://philosophycommons.typepad.com/xphi/2014/09/why-do-experiments.html

@ Highly Adequate
I think that is a good point about the metaanalyses that Doris cites. However, I wonder if Doris can still keep the basic point by appealing to new tools like P-curve which are supposed to be robust against publication bias?

@tomstafford

Another criticism of some of these studies, such as the Infirm Words experiment, is the logic of the methodology. How was the nonconscious nature of the priming established? By asking the participants informally if they thought that the task they had been given had affected their behaviour. That's it! (check for yourself, experiment 2 http://www.yale.edu/acmelab/articles/bargh_chen_burrows_1996.pdf).

There is a basic incoherence used in the definition deployed by Bargh and colleagues (as I argue here) http://journal.frontiersin.org/Journal/10.3389/fpsyg.2014.01067/full

This is a something which does not require you to get to grips with the biases of scholarly publishing in Psychology to take a position on.

Nick Brown

For Brett: I (@sTeamTraen) may be the originator of the #repligate meme. Although I normally hate the way that any public controversy automatically gets turned onto something-gate, the pun with "replicate" was clearly too strong for my unconscious to resist. So I mentioned the word "repligate" in a tweet on June 2, 2014. It was immediately picked up by Brian Nosek, one of the two guest editors of the special issue of Social Psychology dedicated to replications, and it went on from there.

I subsequently discovered, while searching Twitter, that someone else had come up with the same term independently a few days earlier (which is hardly surprising, as we're not talking about the creativity required to write a symphony here), but I hadn't seen that when I thought of it, and as far as I know it was "my" version that took off.

simine

since we are talking about the origin of 'repligate', i feel compelled to point out that john doris was using it in early drafts of his book years ago, and possibly in talks. for whatever that's worth...

The comments to this entry are closed.