i have been sitting on this paul meehl gem for a few months now, ruminating on how it relates to our current situation:
"The two opposite errors to which psychologists, especially clinical psychologists, are tempted are the simpleminded and the muddleheaded (as Whitehead and Russell labeled each other in a famous dinner exchange). The simpleminded, due to their hypercriticality and superscientism and their acceptance of a variant of operationalist philosophy of science (that hardly any historian or logician of science has defended unqualifiedly for at least 30 years), tend to have a difficult time discovering anything interesting or exciting about the mind. The muddleheads, per contra, have a tendency to discover a lot of interesting things that are not so. I have never been able, despite my Minnesota “simpleminded” training, to decide between these two evils. At times it has seemed to me that the best solution is sort of like the political one, namely, we wait for clever muddleheads to cook up interesting possibilities and the task of the simpleminded contingent is then to sift the wheat from the chaff. But I do not really believe this, partly because I have become increasingly convinced that you cannot do the right kind of research on an interesting theoretical position if you are too simpleminded to enter into its frame of reference fully (see, e.g., Meehl, 1970b). One hardly knows how to choose between these two methodological sins." *
here is what i have come up with (i am trying to fit what probably belongs in several separate blog posts into one because i think the points are interconnected. bear with me.)1. another way to describe these groups is that the simpleminded are terrified of type I error while the muddleheaded are terrified of type II error. pick your paranoia.2. this seems like a pretty accurate (if caricatured) way to describe the two extremes of the scientific integrity debate. throughout the rest of the post, i am going to use these labels. i apologize to alexa tullett for speaking in dichotomies, and to all simpleminded and muddleheaded people out there for calling you (us) names.3. one thing that has struck me as i've observed discussions is that people at both extremes have incredibly strong intuitions about which is the bigger problem. the simpleminded are absolutely convinced that the literature is littered with false positives and that is the major threat to our field. the muddleheaded are equally certain that the proposed reforms would stifle scientific discovery and lead us to abandon ideas that are in fact correct and would significantly improve our understanding of human behavior. what is really striking is that not only are both sides full of conviction, they are so sure that they can't even believe someone would sincerely have the opposite intuition. i have seen people on both sides accuse those with opposite intuitions of being disingenuous - their own perception of reality seems so blatantly obvious that they think anyone who denies having that intuition is putting them on.**i'll admit, i am pretty far on the simpleminded side of the continuum, and i have sometimes caught myself completely flabbergasted at the vast distance between my own intuitions and others'.this is a terrible situation to be in. when we don't even believe that our colleagues sincerely hold the intuitions they profess to hold, we are at an impasse. when both sides believe that their (conflicting) assumptions are absurdly self-evident, attempts at reasoning with each other are futile.4. so what can we do?first, i think we need to give each other the benefit of the doubt that, at the very least, we all actually believe what we claim to believe, and that these beliefs do in fact feel intuitive to the belief holders. we should stop questioning each other's sincerity.second, we need to find a way out that does not rely on intuitions. happily, we are in the business of not relying on intuitions. we need empirical evidence. this leads us to the question:5. what empirical evidence would convince the simpleminded? what empirical evidence would convince the muddleheaded?this is, to me, the fundamental question that each side has to answer. we should make our beliefs/intuitions falsifiable, by making concrete empirical predictions about what the world would look like if our intuitions are correct.let me start with the simpleminded, since i feel more comfortable speaking for that side of the continuum.for me, a good test of the 'false-positives-are-everywhere' paranoia that characterizes the simpleminded is brian nosek and the open science framework's reproducibility project. this project aims to conduct close replications of published studies in some of the top journals in psychology. with a large enough sample size, this could give us an estimate of the proportion of published studies that are replicable. assuming this estimate has a decent amount of validity, it can provide a test of the simpleminded world view. what results would be consistent with simpleminded intuitions? it probably depends on which simpleminded person you're talking to. this simpleminded person would probably guess that the replication rate will be below 60%.*** call this my preregistration. if that is wrong, i will admit that the situation is not as dire as i thought, that my intuitions were wrong. of course even a 40% (or, for that matter, 20%) false positive rate should be alarming and would, in my view, justify some reforms. but some of the more drastic reforms proposed by the simpleminded rely on the assumption that the problem is Very Serious. so if the replication rate is much higher than 60%, we have to admit we were wrong, at least in the degree of our panic.so, now i come to a question i can't answer: what would the muddleheaded think is a fair test of their intuitions, and what evidence would cause them to reconsider those assumptions?i thought about writing an entirely separate blog post about this question, because it is one i've been wondering about for a while, but instead i will just add a heading and keep going:bricks in the wall vs. the wall itselfa common view i've heard in response to single instances of rigorous, conclusive failed replications is that the particular study that failed to replicate was just one brick in the wall, and the wall (i.e., the evidence for the broader phenomenon) is made up of hundreds of bricks (i.e., studies that show the predicted effect).that seems fair. pulling out a few bricks here and there does not do much to undermine the integrity of the entire wall, if it is made up of lots of bricks. but what if we sample 5% of all the bricks at random, and we find that most of them are faulty. then wouldn't we worry about the wall? it seems harsh to say that you can't test the wall by testing its bricks - there is no wall other than the bricks that make it up.**** so it seems like replication attempts - especially systematic ones like the reproducibility project - would be a good source of evidence about the soundness of the wall. i would be curious to know what replication rates the muddleheaded would consider consistent with their intuitions, and what results might lead them to rethink those intuitions.another approach is to look at statistical summaries of the literature that give us a clue about the degree of bias (e.g., the p-curve). i wonder whether the muddleheaded would consider these tools appropriate tests of their intuitions about the (non)-prevalence of false positives.
of course, the intuition that false positives are not a huge problem is only one part of the muddleheaded worldview. another important part is the belief that the proposed reforms would stifle discovery (i.e., increase type II error). it would be important to do an empirical test of the severity of this problem under various practices/policies as well. how would we go about doing that? i don't know.conclusion: because people at both ends of the continuum have such strong intuitions, i am not sure that talking at each other is going to get us anywhere (but i am still here typing, so obviously i have not lost all hope). we need empirical tests of: a) the prevalence of false positives in the published literature (which can be achieved by things like large-scale, systematic replication projects and by statistical estimates of bias like the p-curve), and b) the consequences of proposed reforms not only for false positive rates, but also for type II error ('misses').
* i left out the next part of the paragraph, which is this: "One thing I can say in favor of the simpleminded is that I have seen several cases of it get cured, by personal experience of psychoanalysis or by exposure to sufficiently bright, rational, and articulate intellects of opposite persuasion, or by just getting older, securer, and more “relaxed.” Simplemindedness is (not being correlated with stupidity among academicians) a curable condition. But I have, alas, never seen a muddlehead get well. I am inclined to believe that this condition has a hopeless prognosis." i left it out because i worry that it is unnecessarily inflammatory, but it's also an observation from one our field's great methodologists, so i am keeping it in the footnotes, as i hear humanists like to do with things they know they should cut but can't bring themselves to.** you know, like what abe thought god was doing.*** full disclosure: i was recently at a conference where some preliminary results from the reproducibility project were shared. i did not attend the presentation (in my defense, it was early in the morning and the chickens kept me up the night before*****). i did, however, hear about it from others. i can't remember what they told me, but it's entirely possible that i unconsciously absorbed this information and it is influencing my estimate. in fact, that seems pretty likely. especially if i end up being right.**** i am leaving aside the mortar for now.***** in their defense, their eggs were delicious.photo credits: rich lucas, me, & the chickens.