[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
i'm reading a book called Nothing is True and Everything is Possible, about Russia, and the name keeps haunting me (the content of the book is good, too). sometimes i worry that this goes for science, too. it's not just that when nothing is true, everything is possible. but when everything is possible, nothing is true.
sometimes studying human behavior is so wild and messy that it feels like anything goes. we create ad hoc scales more often than we probably should, and we invent manipulations out of thin air, rarely pausing to validate them. if the best we can do is a college student sample, that sin will be forgiven. if we can't get a behavioral measure of a behavior, a self-report will often do. we do what's possible because, well, what's the alternative?
i'm here to consider the exceedingly unpopular view that the alternative - to do nothing - is sometimes preferable to doing what's possible.
science is hard. social science can be especially hard, because we usually need to study humans, and humans are a bitch to study.* there's this idea out there that it's better to collect noisy, messy data than no data at all. i've often repeated this claim myself. indeed, i taught it when i used David Funder's excellent personality textbook. Funder's third law** is that "something beats nothing, two times out of three." Funder was smart to add the caveat, because i think sometimes nothing beats something.
slowly over the last few years, i've come to the conclusion that sometimes crappy data is actually worse than no data. i think that we sometimes fall into the trap of thinking that, because a phenomenon is really, really, really hard to study, very flawed studies are the best we can hope for and so are ok. it's almost like we are committed to the belief that everything must be possible for a reasonably industrious researcher to study, so if the only option is a bad study, then a bad study must be good enough.
let me be clear: i have definitely made this argument myself, possibly as recently as last month. i'm no stranger to the feeling that just getting adequate data feels almost impossible, and so inadequate data must do. for example, i was pretty proud of myself when my lab decided to not just code actual behavior from EAR recordings of >300 participants, and not just double-code the recordings, but TRIPLE code the recordings. when, after two years of coding by over 50 coders (which came on the heels of several years of data collection), we checked the reliabilities on those triple codings and found them wanting, it was very, very tempting to say "well, it's the best we can do." luckily i had learned about the spearman-brown prophecy formula, and so had no excuse - adding more raters was definitely going to help, it's a mathematical fact. so we got three more coders per recording (which took about two more years and 50 more coders). and let me tell you, it sucked. we are sick of coding.*** we wish we could have spent those two years doing new projects. but we got the reliabilities up, and that will let us get much better estimates. science is not all creativity and innovation. it is a lot of boring, hard work. if you don't sometimes hate science, you might be doing it wrong.
even better examples of people who kept going when most of us would have said "good enough:" the two researchers i highlighted in my blog post, "super power." their efforts really make me rethink my definition of impossible.
but sometimes getting good data really is impossible, at least for any single lab. and i would like to float the idea that when this is the case, there may be something noble in walking away, and choosing to do nothing rather than something. sometimes, the nicest thing you can do for your research question is to leave it unanswered. (my colleague wiebke bleidorn pointed out that this is called "addition by subtraction.") if you insist on extracting information from really crappy data, you put yourself at really high risk**** of reading patterns into noise. just because you need your data to tell you something, to move the needle, doesn't mean it can. if our field allowed people to get credit for publishing little bits of (really hard-to-collect) data without making any claims whatsoever, this might be a viable approach, but i don't think that currently exists (though there are low-recognition ways to do this, of course).
the wishful thinking that we can always extract some knowledge from even small bits of messy data can lead to serious, widespread problems. it's easy to point fingers at other fields, but this problem is alive and well in psychology. i'm very familiar with the argument that this sample size or that method must be good enough because it's all we can realistically do, given limited resources, hiring or promotion expectations, etc.. it used to be one of the most common responses i'd hear to calls for larger samples or better (i.e., harder) methods. people seem to have backed off of making the argument out loud, but i think it's still very common in people's minds - this attitude that it's inappropriate to say a method or design isn't good enough because it was hard/expensive/time-consuming to do. (indeed, i think this is the most common internal response most of us have when we get criticized for studying only WEIRD samples.)
here's what i wish we all did (including me): 1) ask yourself what the ideal study would be to test your research question. 2) ask yourself if you're willing and able to do something pretty close to it. 3) if not, ask yourself why not. really push yourself. don't let yourself off the hook just because it would be a lot harder than what you're used to doing. if the research question is important, why isn't it worth the really hard work? 4) if you're still not willing or able to do something close to the ideal study on your own, either a) move on to a different (or narrower) research question, or b) join forces with other labs.
i know this is idealistic. i know we have to keep publishing otherwise our food pellets stop coming. so let's pick questions we can actually hope to answer with the resources we have. (this post is really one big, public NOTE TO SELF. i have been extremely guilty of biting off way more than i can chew,***** and want to learn to scale back my expectations of what kinds of research questions i can rigorously test on my own.) or join collaborative projects that pool resources to tackle the really hard, important questions, and find a way to deal with the issue of spreading credit around. if we stop trying to do the impossible, i think we'll find that more things are true.
* also, sometimes to interact with. or be near.
** Funder calls his lessons "laws" somewhat tongue-in-cheek. the whole book is a treat, but if you're new to personality psych, i especially recommend chapters 1 through 7. felicity's professor at the university of new york used it, almost certainly because of it's engaging yet accurate prose. this is a very serious footnote, i am not kidding even a little bit. read it.
*** it is kind of like if your lab had to eat asparagus for six meals a week for several years, and then you realized you had several more years of compulsory asparagus-eating. (except our pee don't stink.)
**** 100% risk
***** true story: my undergrad thesis examined how sex-role identity interacts with gender to predict attitudes towards women... among high school students... in two different countries.... one of which is Samoa, a remote, non-English-speaking island in the middle of the South Pacific.
did someone say asparagus?
bitter carrots*
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
carrotsif you had told me five years ago that even one in twenty social/personality psych papers would provide links to their data and code, or to a pre-registration, i would have thought that would be huge progress.** i've long been a fan of the nudges that encourage these kinds of practices (e.g., badges), and until recently i thought going as far as to require this kind of transparency (even with room for legitimate exceptions) was probably unrealistic - our field didn't seem ready for that. i was sympathetic to the carrots-not-sticks approach.but there's a problem with carrots-not-sticks. we're asking researchers to eat the carrots, but some of the carrots are pretty bitter. sometimes, when researchers are transparent, that brings information to light that undermines their claims, and readers don't buy the claims. that's a necessary side effect of transparency. and it means we can't in good faith tell researchers that transparency is always in their best interest and will be its own reward. we can't lure people with carrots, and pretend all of the carrots are delicious and fun to eat. sometimes carrots are hard to swallow.i think it's time to admit that the main argument for transparency isn't self-interest - it's that transparency is just better for science.***imagine the following scenarios:scenario 1: you get a paper to review, and the authors have shared their data and code. you look at the data and realize there is a coding error, or something else that makes the results uninterpretable (i.e., suggests the study needs to be re-run to fix the error). you point this out in your review, the editor agrees and rejects the manuscript.
scenario 2: you get a paper to review, and the authors have shared a link to their pre-registration. by comparing the manuscript and the pre-registration you realize that the analysis that the authors present as their planned analysis, and interpret the p-value for, is not actually the one they had specified a priori as their key planned analysis. knowing this, you can tell that the claims in the paper are not supported by the evidence. the editor agrees and rejects the manuscript.scenarios 1 and 2 seem pretty straightforward. but now consider scenarios 3 and 4:scenario 3: you get a paper to review and the authors do not provide their data and code, but there is no sign of anything wrong.scenario 4: you get a paper to review and the authors did not preregister their study, but they claim that the key result they present was their planned analysis, and interpret the p-value as if it was the only test they ran.what should you do in scenarios 3 and 4?one option, and i think the way most of us have been operating, is to assume that the data have no anomalies, and the key analysis was indeed the one planned test that was run. but is this fair to the authors in scenarios 1 and 2? in scenarios 3 and 4, we're giving the authors the benefit of the doubt because they didn't let us verify their claims. in scenarios 1 and 2 we're punishing them because they did let us verify their claims, and we learned that their claims were not justified.but what else could we do in scenarios 3 and 4? assume that their studies had the same flaws as the studies in scenarios 1 and 2? that doesn't seem fair to the authors in scenarios 3 and 4.when some authors choose to be transparent, we have no choice but to use the extra information they give us to assess the rigor of their study and the credibility of their claims. but that also puts us in an impossible position with respect to the manuscripts in which authors are not transparent. we can't assume these non-transparent studies have flaws, and we can't assume they don't.it seems to me the only fair thing to do is to make transparency the default.**** whenever possible, authors should be expected to share the data and code necessary to reproduce their results unless that's legally or ethically problematic. and if authors claim that their key analysis was planned (i.e., if they're saying they're doing hypothesis testing and/or interpreting a p-value), we should ask that they document this plan (i.e., pre-register), or present their work as exploratory and their conclusions as less sure. it's just not fair to let some authors say "trust me" when other authors are willing to say "check for yourself."i know that's not the world we live in, and as long as transparency is not the default, we have to treat papers like those in scenarios 3 and 4 somewhere in the gray area between flawless and deeply flawed. but my heart really goes out to the authors in scenarios like 1 and 2. it would be completely rational for those authors to feel like they are paying too high a price for transparency. (i've tried to make the case that there is a silver lining - their transparency makes the review process more fruitful for them, because it allows reviewers and editors to pinpoint specific ways they could improve their work which wouldn't have been possible without their openness. but i'm sure that's not much consolation if they see papers like those in scenarios 3 and 4 getting published over theirs.)my sense is that many people are getting on board with the credibility revolution, but only so long as all of the incentives are carrots, and not sticks. as long as we can choose which carrots we want to go for, and not be punished if we don't eat our veggies, everyone is happy. but that won't work in the long run. it was perhaps a necessary step on the way to more widespread changes, but i think we need to start seriously considering making carrot-eating the default (also known as using sticks). i can't think of how to make the current opt-in system we have fair. if you can, i'd love to hear it.* for more on problematic carrot-eating, see this by james heathers.** in honor of the finding that two spaces after periods is the morally superior formatting, i am compensating for years of being bullied into one space by using three paces. (yes, i'm aware of the shakiness of the evidence but i never let that get in the way of a good footnote.) #iwantmyspacesback*** i don't mean that everything should always be transparent, and i don't know anyone who believes that. i mean that things that can legally and ethically be made transparent usually should be.
**** this seems like a good time to remind readers that i do not set policy for any journals, and i am conscious of the difference between my personal fantasies and the realities of editorial responsibility (as tempting as it is to use my vast editorial powers to force everyone to put five spaces between sentences).********** editorial abuses of power is a topic for another blog post. #bringonthelawsuits
mmmm carrots
An Oath for Scientists
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
i've been thinking a lot about what it means to be a scientist. being a scientist comes with certain obligations, and ignoring those obligations can give science a bad name. it seems to me we could do more to make scientists aware of this responsibility when they decide whether or not to join the profession.
our most important obligation as scientists, to my mind, is preserving science's credibility. that doesn't mean we can't make mistakes, but above all else, we should be committed to opening ourselves up to scrutiny and correcting our errors.
to make these values a bit more concrete, i tried to adapt the hippocratic oath to scientists. you can tell how solemn this oath is by the use of capitalization.
the values i tried to capture were inspired by Merton's norms, but in the spirit of Merton's norm of universalism, i refrained from naming the oath after him (or anyone). it is very far from comprehensive, and i know it's cheesy, but i ask you, dear reader: if you can't engage in a little facile sentimentality on new year's day, when can you?
An Oath for Scientists
I swear that I will, according to my ability and judgment, protect the credibility of science by carrying out this oath and this indenture.
To make the grounds for my scientific claims transparent and available for others to scrutinize, to welcome that scrutiny and accept that others will be skeptical of my claims, to help others verify the soundness of my claims; to describe my methods in sufficient detail for others to repeat them, to not obstruct others' attempts to replicate my work; to report all evidence I know of for or against my claim, to not suppress evidence against my conclusions, to correct my past claims if I learn that they were wrong, to support the dissemination of evidence that disconfirms or contradicts my past claims.
I will hold myself and all other scientists to this oath, and I will not exempt any scientist because of her status or reputation. I will judge scientific claims based on the evidence, not the scientist making the claim. Neither will I hold any scientific claim or finding as sacred. Similarly, I will recognize as valuable the work of scientists who aim to correct errors in the scientific record.
In whatever claims I present in my role as scientist, I will not knowingly overstate or exaggerate the evidence, I will not make claims out of interest for advancing my own station, and I will disclose any personal interest that may be perceived as biasing my judgment. I will protect the credibility of my profession by making careful, transparent, calibrated claims.
Now if I carry out this oath, and break it not, may I gain for ever reputation among all scientists for my work; but if I transgress it and forswear myself, may the opposite befall me.
results blind vs. results bling*
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
show-offin many areas of science, our results sections are kind of like instagram posts. beautiful, clear, but not necessarily accurate. researchers can cherry-pick the best angle, filter out the splotches, and make an ordinary hot dog look scrumptious (or make a lemon look like spiffy car).** but what's even more fascinating to me is that our reaction to other people's results are often like our reactions to other people's instagram posts: "wow! that's aMAZing! how did she get that!"
i've fallen prey to this myself. i used to teach bem's chapter on "writing the empirical journal article," that tells researchers to think of their dataset as a jewel, and "to cut and polish it, to select the facets to highlight, and to craft the best setting for it." i taught this to graduate students, and then i would literally turn around, read a published paper, and think "what a beautiful jewel!"***
as with instagram, it's impossible to mentally adjust our reaction for the filtering the result could have gone through. it's hard to imagine what the rough cut might've looked like. it's hard to keep in mind that there could've been other studies, other measures, other conditions, other ways to clean the data or to run the analyses. and we never know - maybe this is one of those #nofilter shots.
in short, we can't avoid being blinded by shiny results.
what can we do?
there are a few stopgaps. for example, as an author, i can disclose as much as possible about the process of data collection and analysis, and the results (e.g., the 21 word solution). as a reader, i'll often pause when i get to the end of the method section and ask myself - is this study well-suited to the researchers' goals? would i think it should be published if i had to evaluate it just based on the method?
another partial solution is pre-registration, the #nofilter of scientific research. by pre-registering, a researcher is committing to showing you the raw results, without any room for touching-up (for the planned analyses - the exploratory analyses can be looked at from any and all angles). with a good pre-registration, readers can be pretty confident they're getting a realistic picture of the results, except for one problem. the editors and reviewers make their evaluations after seeing the results, so they can still consciously or unconsciously filter their evaluation through biases like wanting only counterintuitive, or significant, findings. so pre-registration solves our problem only as along as editors and reviewers see the value of honestly-reported work, even if it's not as eye-catching as the filtered stuff. as long as editors and reviewers are human,**** this will likely be a problem.
the best solution to this problem, however, is to evaluate research before anyone knows the results. this is the idea behind registered reports, now offered by Collabra: Psychology, the official journal of the Society for the Improvement of Psychological Science. an author submits their paper before collecting data, with the introduction, proposed method, and proposed analyses, and makes a case for why this study is worth doing and will produce informative results. the editor and reviewers evaluate the rationale, the design and procedures, the planned analyses and the conclusions the authors propose to draw from the various possible results. the reviewers and editor give feedback that can still be incorporated into the proposed method. then, if and when the editor is satisfied that the study is worth running and the results will be informative or useful regardless of the outcome, the authors get an "in principle acceptance" - a guarantee that their paper will be published so long as they stick to the plan, and the data pass some basic quality checks. the final draft goes through another quick round of review to verify these conditions are met, and the paper is published regardless of the outcome of the study.
registered reports have many appealing characteristics. for the author, they can get feedback before the study is conducted, and they can get a guarantee that their results will get published even if their prediction turns out to be incorrect, freeing them to be genuinely open to disconfirmation of their predictions. it's nice to be able to have less at stake when running a study - it makes for more objectivity, and greater emotional stability.*****
for science, the advantage is that registered reports do not suffer from publication bias - if all results are published, the published literature will present an unbiased set of results, which means science can be cumulative, as it's meant to be. meta-scientists can analyze a set of published results and get an accurate picture of the distribution of effects, test for moderators, etc. the only downside i can think of is that journals will be less able to select studies on the basis of projected citation impact - the 'in principle acceptance' means they have to publish even the findings that may not help their bottom line. call me callous but i'm not going to shed too many tears over that.
not everything can be pre-registered, or done as a registered report. for one thing, there's lots of very valuable existing data out there, and we shouldn't leave it hanging out to dry. for another, we should often explore our data beyond testing the hypothesis that the study was originally designed to test. many fascinating hypotheses could be generated from such fishing expeditions, and so long as we don't fool ourselves into thinking that we're testing those hypotheses when we're in fact just generating them, this is an important part of the scientific process.
the fact that we keep falling for results bling, instead of evaluating research results-blind, just means we're human. when you know the results are impressive, you're biased to think the method was rigorous. the problem is that we too easily forget that there are many ways to come by impressive-looking results. and even if we remember that filtering was possible, it's not like we can just magically know what the results look like unfiltered. it's like trying to imagine what your friends' unfiltered instagram pictures look like.
are we ready to see what science looks like without the filters? will we still get excited when we see it au naturel? let's hope so - to love science is to accept it for what it really looks like, warts and all.
* this title was inspired by a typo.
** although i'm using agentic verbs like "filtering" or "prettying up," i don't believe most of these distortions happen intentionally. much of the fun in exploring a dataset is trying to find the most beautiful result we can. it's hard to remember everything we tried, or what we might have thought was the best analysis before we knew how the results looked. most of the touching up i refer to in this post comes from researchers engaging in flexible data analysis without even realizing they're doing so. of course this is an assumption on my part, but the pretty large discrepancies between the results of pre-registered studies and similar but not-pre-registered studies suggests that flexibility in data analysis leads to exaggerated results.
*** words you'll never actually hear me say.
**** canine editing: not as effective as it sounds.
***** personality change intervention prediction: if registered reports become the norm, scientists' neuroticism will drop by half a standard deviation. (side effect: positive affect may also take a hit)
octopus vulgaris
alpha wars
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
gomii was going to do a blog post on having thick skin but still being open to criticism, and how to balance those two things. then a paper came out, which i’m one of 72 authors on, and which drew a lot of criticism, much of it from people i respect a ton and often agree with (one of them is currently on my facebook profile picture, and one of the smartest people i know). so this is going to be a two-fer blog post. one on my thoughts about arguments against the central claim in the paper, and one on the appropriate thickness for scientists’ skin.
PART I: the substantive argument*
in our paper we argue that we should introduce a new threshold for statistical significance (and for claiming new discoveries), and make it .005.
i want to take a moment to emphasize some other things we said in the paper. we said that .005 should not be a threshold for publication. we also said that we can keep the p < .05 threshold and call results that meet this threshold, but not the .005 threshold, “suggestive” (and not enough to claim a new discovery). so, in some ways, this is not very different from what many people already implicitly (or explicitly?) do when we read papers – treat p-values as a quasi measure of the strength of the evidence, and maintain a healthy amount of skepticism until p-values get pretty low (at least in the absence of pre-registration, more on this below).
here are some arguments against the claim in our paper, and my thoughts on each of these. many of these arguments came out of a long and lively discussion with Ellen Evers and Daniël Lakens (and from Lakens’s blog post). i’m focusing on these arguments because i find them compelling or interesting to think about, so am curious to hear more points in favor of or against any of the points below. this is me playing with the ideas here. i am sure there are flaws in my thinking.
1. if we lower alpha to .005, the ratio of the type I error rate (when the null is true) to the type II error rate (when the null is false) will be way off (1:40).**
thoughts:
that would be true if the type I error rate really were equal to what we say our alpha level is, and the type II error rate really were equal to what our power analyses tell us beta is. i don’t believe either of those is the case. i believe our type I error rate is actually much higher than our field’s nominal alpha level. and the type II error rate is much lower than beta as calculated from power analyses (see my argument for this claim here – basically, p-hacking and QRPs help us make things significant, and that increases type I error, which we know, but also decreases type II error, which we rarely talk about). so, under the current state of affairs in our field, lowering the nominal alpha level to .005 may actually bring the ratio of type I to type II error rates closer to the 1:4 that Cohen and others suggested than it is right now.
of course this would change if practices change a lot. if we consistently pre-register our studies and follow our pre-registration, and only interpret p-values for the planned analyses, then our using a .005 alpha level would indeed lead to a 1:40 ratio of type I error rates when the null is true to type II error rates when the null is false. and i agree that this is too small a ratio in many circumstances. so, i’ll revisit my opinion on what threshold we should use as a default for drawing meaningful conclusions when our practices have changed drastically. (i still use p-values when i’m not doing confirmatory research, so probably the first step is for me to quit doing that.*** one reason i haven’t is that i suspect it would make it very hard for my grad students to publish their work in outlets that would help them get jobs. life is hard. principles ain’t cheap.)
for the same reason, i would be in favor of exempting the results of key, pre-registered analyses from this lower alpha. i don’t know exactly what the right alpha should be, but i think that when the rules of Neyman-Pearson hypothesis testing were followed (pre-registered study with key analysis specified), i’m less worried about an out-of-control type I error rate.
2. we should abandon thresholds, or NHST, altogether.
thoughts:
on thresholds: i have a hard time separating my feelings on the pragmatism of this position (very low) from my answer to “in a perfect world, would this be the right approach?” i don’t know. i definitely think thresholds cause a lot of problems. but it’s hard for me to imagine humans abandoning them. as an editor, i find thresholds useful because otherwise editorial decisions would feel even more arbitrary to authors. i am sympathetic to authors who feel that the criteria on which their submissions will be evaluated are not transparent enough, or are impossibly vague. i know thresholds aren’t the only solution to this problem (and are far, far from sufficient), but i am not imaginative enough to come up with something that would adequately replace them.
also, if we want to encourage less black-and-white thinking, aren’t two thresholds (one for “suggestive” results and one for “statistically significant” results) better than one? isn’t this a (baby) step towards thinking in gray?
but i get it. maybe to accept that thresholds are here to stay is to give away the farm. instead of spending effort tweaking a broken system and trying to make it a little less broken, maybe i should spend more time trying to change the system (see also #4 below).
on abandoning NHST or frequentist approaches: i have to admit, much of this debate is over my head. i have a lot more to learn about the philosophy and math behind Bayesian approaches.
3. requiring results to meet a p < .005 threshold before they’re considered strong evidence is going to make it prohibitively difficult to conduct studies.
thoughts:
i have to admit, i’m a bit surprised by this argument. it seems like one of the few things we all agree on is that of course a single study, especially one with a high p-value, is far from conclusive, and of course we need mounds of evidence before drawing strong conclusions. this kind of view is often expressed, but rarely heeded in discussion sections of papers, or in press releases. in my view, introducing a new threshold is a way to try to enforce this skepticism that we all say we want more of. (particularly because we’re not saying that results shouldn’t be published until they meet this threshold, just that they should be considered nothing more than suggestive). why not make a rule that holds us to this standard we espouse? if we can still consider results below .05 suggestive, we can still claim everything we’ve been saying we should claim, but not more.
even with a much lower threshold, a single study can’t be conclusive. but at least it can have a chance of providing much stronger evidence. and the costs of powering studies to this new threshold are not exorbitant, in my opinion (i know there is strong disagreement on this. it's hard to know what should count as a ridiculous sample size expectation. six years ago many, many people considered n = 20 per cell ridiculously high.). for a correlation of .20, you need 191 people to have 80% power at the .05 threshold, and 324 people to have 80% power at the .005 threshold. I don’t know about you, but that’s a way smaller price than i expected to have to pay for dividing our alpha level by ten. moreover, if you decide to only power your study to the .05 threshold, you should still get a p-value below .005 about 50% of the time. so if you’re running a series of studies and consistently missing the .005 threshold, something’s wrong.****
what does make sense to me about this argument is that lowering the threshold to .005 leads to a more conservative science. yes. absolutely. where i think i disagree with the spirit of the argument is that, implied in that criticism is the view that our past/current standards are reasonable or balanced (i.e., not particularly conservative or liberal), and so a drastic shift towards greater conservatism would be draconian. in my view, our past (i’ll stay agnostic about current) standards were incredibly liberal – i am pretty convinced that it was relatively easy to, without realizing it, get a significant result even when the effect didn’t exist. i know this is controversial, but i think it’s important to be clear about where the disagreement is. i want our standards to become more conservative, but not because i think we should have a very conservative type I error rate or false discovery rate (FDR). i probably want about the same type I error rate or FDR as people who oppose the reforms i’m for. the difference is that i think our current standards put us way, way above that mark, and so we need to shift to a more conservative standard just to hit the target we thought we’d been aiming for all along.
4. the real problem is that p-values, as currently used, are not meaningful, and we should fix that problem rather than hold p-values to a higher standard.
thoughts:
this is the most compelling argument i’ve heard so far. i have tried to push for changes that would bring us closer to being able to interpret p-values in the way that (as i understand it) the Neyman-Pearson approach to NHST describes. that is, more pre-registration, and more transparency (e.g., if you say you predicted something, show us the pre-registration where you wrote down your prediction and how you would test it, and if you didn’t pre-register, refrain from claiming that, or interpreting statistics as if, you did).
if i could choose between introducing a new threshold for significance and getting people (including myself) to follow Neyman-Pearson rules (or not use p-values when we aren’t following those rules – also ok), i would choose the latter. and in a world where we were careful to only interpret p-values when Neyman-Pearson rules were followed, i might advocate lowering alpha, but probably not all the way to .005.
so this is me admitting that by recommending a new threshold, i am caving a bit. my faith in a future in which we achieve transparency and incentivize truly confirmatory research sometimes wavers, and i want a more immediate way to address what i perceive to be a pretty serious problem. this may be a bad compromise to make. i’m still trying to figure that out.
these criticisms, and others, have definitely made me reconsider the recommendations in our paper. i haven’t yet come around to the view that they’re horribly misguided – i still don’t see much harm in them – but i'm considering the possibility that there are better ways to achieve the same goals, or that our efforts are better spent elsewhere. right now, i still feel that introducing a new threshold will actually help the other goals – encourage more transparency and more pre-registration. this is based mostly on my perception that it will be easier to get a result below that threshold by engaging in these practices than by p-hacking (because p-hacking down to .005 is pretty hard, from what i understand). so, if we assume that people respond to incentives, or are drawn towards the least effortful path, adding this hurdle could make suboptimal research practices less attractive, by making them less effective. if p-hacking isn’t going to save you, then the costs of pre-registering are lower (i.e., tying your hands is less costly when the loopholes you’re closing by pre-registering were unlikely to work anyway). but i acknowledge that, for me, the end goal is to improve research design and practices, and the change to statistical interpretation is, in large part, a means to that goal. that, and my playing around with G*power leads me to the conclusion that, when an effect is really there, and you’ve done a halfway decent job of designing your study, you should get a p-value below .005 quite often, so this threshold isn’t as scary as it might seem.
PART II: the process
i talk big about embracing criticism, fostering skepticism, etc. i also agree with many people who’ve expressed the view that you have to have a healthy layer of thick skin to be in science. these two views seem at odds to me, and i’ve been thinking about how to reconcile them.
i now have a bit more direct experience on which to base my reflection. it's still going to sound platitudinous. here’s what i’ve come up with:
-look hard for kernels of logic/reasoning or empirical evidence. sometimes it’s garnished with a joke at your expense, or accompanied by a dose of snark, but often there’s at least a nugget of real argument in there. the criticism may still be wrong, but if it takes the form of a good faith effort at reasoning or empiricism, it’s probably worth entertaining.
-talk it out. find people who you know are willing to tell you things you don’t want to hear. ask them what they think about the criticisms. ask them which ones they think you should take most seriously, and if there are any you can dismiss as not in good faith. if possible, talk directly with the people who are criticizing you. (my thinking benefited tremendously from talking to Lakens and Evers).
-take your time. you’ll notice when your emotional reaction starts to subside and you can laugh a bit at your own initial defensiveness. don’t decide what to do about the criticism until you’ve reached that point. if you don’t reach that point, it’s not because you were never defensive.
-if, after all that, there is some criticism that clearly seems outside the realm of reasoning/empiricism, or is not in good faith, this is where it’s time to suit up and put on the thick skin. let it go. go play with your cat, or watch a funny video, or bake a cake.*****
some people have a harder time with the first three steps (being permeable), some people have a harder time with the last step (being impermeable). it's not easy to know when – and be able – to go back and forth. but developing those skills is pretty important in science (life?). also – i don’t think anyone is very good at doing these things alone. find people who help you be more open to criticism, and people who help you be more resilient. they might be the same people (those are pretty amazing people, keep them), or they might not (those people are ok, too).
* i speak only for myself, and not for my co-authors or anyone else.
** what should the ratio be? i have no idea, and there’s clearly not just one answer, but apparently our field likes 1:4 a lot (alpha = .05, beta = .20).
*** are p-values useful to report even when you’re doing exploratory research? i have some thoughts on this (inspired by Daniël Lakens, again), but that’s for another blog post.
**** this raises the question of whether every study needs to meet the .005 threshold to qualify for the new threshold. here the answer is clearly no – we can identify what the distribution of p-values across multiple studies should look like if your effect is real, and it’s not 100% significant p-values (because of sampling error), regardless of the threshold. but if you have some ugly p-values, you can only confidently chalk up that messiness to sampling error if your readers can rule out other sources of messiness, like flexibility in data analysis or selective reporting – that is, if everything is pre-registered and all studies are reported. also a blog post for another day.
***** emotion suppression gets such a bad rap.
what is rigor?
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
recently i wrote a thing where i was the eight millionth person to say that we should evaluate research based on its scientific qualities (in this case, i was arguing that we should not evaluate research based on the status or prestige of the person or institution behind it). i had to keep the column short, so i ended with the snappy line "Let's focus less on eminence and more on its less glamorous cousin, rigor."
simple, right?
the question of how to evaluate quality came up again on twitter,* and Tal Yarkoni expressed skepticism about whether scientists can agree on what makes for a high quality paper.
there's good reason to be skeptical that scientists - even scientists working on the same topic - would agree on the quality of a specific paper. indeed, the empirical evidence regarding consensus among reviewers during peer review suggests there is ample disagreement (see this systematic review, cited in this editorial that is absolutely worth a read).
so, my goal here is to try to outline what i mean by "rigor" - to operationally define this construct at least in the little corner of the world that is my mind. i have no illusions that others will share my definition, nor do i think that they should - there is some value to different people having different approaches to evaluating work, because then each may catch something the others miss (this is one of the reasons i'm not too concerned about low agreement among peer reviewers - we often select them on purpose to focus on different things or have different biases). nevertheless, i think we should try to build some consensus as a field about what some important criteria are for evaluating the quality of research, and maybe if we start trying to articulate these more explicitly, we can see where people's views overlap and where they don't. i think the exchange between Simmons and Finkel & Eastwick (and Simmons's response) was a nice example of that.
what is rigor? to me, this is what teaching research methods is all about. trying to identify specific features that make for a good scientific study. so it won't surprise you that many of the things on my list of what makes a study rigorous will be super basic things that you learned in your first year of college. but it's amazing to me how often these issues are overlooked or waved away in studies by seasoned researchers. thus, even though many of these things will seem too obvious to mention, i think they need to be mentioned. still, this list isn't even close to exhaustive, and it's very heavily oriented towards social and personality psych studies. i would love to see others' lists, and to work together to come up with a slightly more comprehensive list.**
so, what do i ask myself when i'm reading and evaluating a scientific paper in my field (e.g., when i'm evaluating job candidates, trying to decide whether to build on a finding in my own research, reviewing a manuscript for a journal, etc.)?*** i ask myself two broad questions. there are many more specific questions you could list under each one, but these broad questions are how i orient my thinking.
1. is the narrow claim true: did the authors find the effect they say they found?
questions about the research design:-is the sample size adequate and is it clear how it was determined?-is the population appropriate for the questions the authors are trying to answer?-are the measures and manipulations valid (and validated)?-are there confounds? selection effects? selective dropout? (i.e., were the groups equal to begin with, and treated equally other than the manipulated variable?)-could there be demand characteristics?-did the authors design the study such that null results could be informative? (big enough sample, validity checks, etc.)-do the authors report all conditions and all measures collected?-do the authors provide links to their materials?questions about the analyses:-are the analyses appropriate? do they test the key research question(s)? (this might seem like it's super straightforward, but it's often pretty complicated to identify the key statistical analysis that directly tests your prediction (something you become painfully aware of when you try to do a p-curve).)-are the test statistics correct? (e.g., run statcheck)-do the authors report all equally reasonable analyses they could have done to test their question (e.g., different decisions about data exclusions, transformations, covariates, etc.)?-if there were multiple analyses or multiple ways to conduct each analysis, do the authors address the increased chance of false positives (because of multiple comparisons)?-do the authors provide their data and analysis code? if so, are the results reproducible?2. is the broader claim true: does the finding mean what the authors claim it means?-are there alternate explanations?-if the authors are making causal claims, are those justified?-is the strength of the authors' conclusion calibrated to the strength of their evidence? (there are various ways to think about what "strong evidence" looks like: precise estimate, small p-value, large Bayes Factor, etc.)-if results are not robust/consistent, are the conclusions appropriately calibrated?-do the authors focus on some results and ignore others when interpreting what they found?-do the authors extrapolate too far beyond the evidence they have? are they careful about any generalizations beyond the population/setting/measures they examined? do they include a statement of constraints on generalizability?-if the research is not pre-registered, is it presented as exploratory and is the finding presented as a preliminary one that needs to be followed up on?-do the authors seem open to being wrong? do they take a skeptical approach to their results? do they have any conflict of interest that may limit their openness to disconfirming evidence?then what?if a paper passes both of these hurdles, there are still questions left to ask. what those questions are depends on your goal. you might ask whether the study is important or interesting enough to spend more time on (e.g., build on with your own research). you might ask if the research topic is a good fit with your department. you might ask if the study is novel or groundbreaking or definitive enough for a particular journal. you might ask what implications it has for theory, or how its findings could be applied in the real world.to be honest though, if a paper passes both of these hurdles, i'm pretty impressed.**** maybe my standards will go up as the norms in our field change, but right now, a paper that passes these hurdles stands out to me.if there were three people like me, who defined rigor according to the criteria outlined above, then would we agree about which papers are better than others? which candidate we should hire? i don't know. i'd like to think agreement would be slightly better than without a shared definition. if so, that could be an argument for editorial teams, search committees, grant panels, and other evaluating bodies to try to come up with at least a loose set of criteria by which they define rigor. perhaps that is a step towards this land of unicorns and roses where we don't just rely on metrics, where scientific work is evaluated on its own merit, and not based on status, fads, or the evaluator's hunger level.****** you can literally debate anything on twitter. even powdered cheese.** if you like this idea, you might like the SIPS workshop on "how to promote transparency and replicability as a reviewer" (osf page here. SIPS 2017 program here.)*** i don't actually ask myself each of these sub-questions every time. as with many operationalizations, this is a flawed attempt at articulating some of the things i think i do implicitly, some of the time. mmmmm operationalization. funtimes.**** i don't mean passes every single question - i'm not crazy like reviewer 2. i just mean if i can answer yes to "is the narrow claim true?" and to "is the broader claim true?"***** no, that's probably not a real thing.
be your own a**hole
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
do you feel frustrated by all the different opinions about what good science looks like? do you wish there were some concrete guidelines to help you know when to trust your results? well don't despair!
it's true that many of the most hotly debated topics in replicability don't have neat answers. we could go around and around forever. so in these tumultuous times, i like to look for things i can hold on to - things that have mathematical answers.
here's one: what should we expect p-values for real effects to look like? everyone's heard a lot about this,* thanks to p-curve, but every time i run the numbers i find that my intuitions are off. way off. so i decided to write a blog post to try to make these probabilities really sink in.
do these p-values make me look fat?let's assume you did two studies and got p-values between .02 and .05. should you be skeptical? should other people be skeptical?many of us have gotten used to thinking of p = .049 as the world's thinnest evidence, or maybe even p = .04. but what i'll argue here is that those intuitions should stretch out way further. at least to p = .02, and maybe even to p = .01 or lower.let's get one thing out of the way: if you pre-registered your design, predictions, and key analyses, and you are interpreting the p-values from those key analyses (i.e., the confirmatory results) then you're fine. you're doing real Neyman-Pearson NHST (unlike the rest of us), so you can use the p < .05 cutoff and not ask any further questions about that. go eat your cookie and come back when the rest of us are done dealing with our messy results.now for the rest of us, who don't pre-register our studies or who like to explore beyond the pre-registered key analysis (i.e., who aren't robots**), what should we make of two results with p-values between .02 and .05?the math is simple. using an online app (thanks Kristoffer Magnusson, Daniel Lakens, & JP de Ruiter!) i calculated the probability of one study producing p-values between .02 and .05 when there is actually a true effect. it's somewhere between 11% and 15% (i played around with the sample size to simulate studies with power ranging from 50% to 80%). so what's the probability of getting two out of two p-values between .02 and .05? at "best" (i.e., if you're doing underpowered studies), it's 14% x 14% = 2%. with decent power, it's around 1.4% (i.e., 1 in 70).CORRECTION (11/24/2017): i was wrong! the table that was here before (and the numbers in the paragraph above) were wrong - i'm not sure what happened. the paragraph above has been updated, and here is the corrected table. the biggest difference is that the probability of getting high p-values at 80% power is higher than i had said. the points made in the rest of the post still hold, as far as i can tell!if you agree with this math (and there's really not much room for disagreement, is there?), this means that, if you get two out of two p-values between .02 and .05, you should be skeptical of your own results. if you're not skeptical of your own results, you make everyone else look like an asshole when they are skeptical of your results. don't make them look like assholes. be your own asshole.yes, it's possible that you're in that 2%, but it would be unwise not to entertain the far more likely possibility that you somehow, unknowingly, capitalized on chance.*** and it's even more unreasonable to ask someone else not to think that's a likely explanation.and that's just two p-values between .02 and .05. if you have even more than two studies with sketchy p-values (or if your p-values are even closer to .05), the odds you're asking us to believe are even smaller. you're basically asking anyone who reads your paper to believe that you won the lottery - you managed to get the thinnest evidence possible for your effect that reaches our field's threshold for significance.of course none of this means that your effect isn't real if your p-values are sketchy. i'm not saying you should abandon the idea. just don't stop there and be satisfied with this evidence. the evidence is telling you that you really don't know if there's an effect - whether you remember it or not, you likely did a little p-hacking along the way. that's ok, we all p-hack. don't beat yourself up, just design a new study, and another, and don't stop until the evidence is strong in one direction or the other.****and i don't just mean the cumulative evidence. yes, you can combine the p = .021 study, the p= .025 study, and p = .038 study to get a much smaller p-value, but that still doesn't explain how you got three high p-values out of three studies (extremely unlikely). even with the meta-analytic p-value at p < .005, a reader (including you) should still conclude that you probably exploited flexibility in data analysis and that those three results are biased upwards, making the cumulative (meta-analytic) evidence very hard to interpret. so keep collecting data until you get a set of results that is either very likely if there's a true effect (i.e., mostly small p-values) or very likely under the null (i.e., a flat distribution of p-values). or, if you're brave and believe you can design a good, diagnostic study, pre-register and commit to believing the results of the confirmatory test.if that's too expensive/time-consuming/impossible, then do stop there, write up the results as inconclusive and be honest that there were researcher degrees of freedom, whether you can identify them or not. maybe even consider not using NHST, since you didn't stick to a pre-registered plan. make the argument that these results (and the underlying data) are important to publish because this small amount of inconclusive evidence is super valuable given how hard the data are to collect. some journals will be sympathetic, and appreciate your honesty.*****we talk big about how we want to preserve a role for creativity - we don't want to restrict researchers to pre-registered, confirmatory tests. we need space for exploration and hypothesis generation. i wholeheartedly agree; everything i do is exploratory. but the price we have to pay for that freedom and creativity is skepticism. we can't have it both ways. we can't ask for the freedom to explore, and then ask that our p-values be interpreted as if we didn't explore, as if our p-values are pure and innocent.* brent roberts says it's ok to keep repeating all of the things.** this is not a dig at robots or pre-registerers. some of my favorite people walk like robots.*** yes, that's a euphemism for p-hacking.**** my transformation into a bayesian is going pretty well, thanks for asking. if you drink enough tequila you don't even feel any pain.***** probably not the ones you were hoping to publish in, tbh.
Perspectives You Won't Read in Perspectives: Thoughts on Gender, Power, & Eminence
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
This is a guest post by Katie Corker on behalf of a group of us
Rejection hurts. No amount of Netflix binge watching, nor ice cream eating, nor crying to one's dog* really takes the sting out of feeling rejected. Yet, as scientific researchers, we have to deal with an almost constant stream of rejection - there's never enough grant money or journal space to go around.Which brings us to today's topic. All six of us were recently rejected** from the Perspectives on Psychological Science special issue featuring commentaries on scientific eminence. The new call for submissions was a follow-up to an earlier symposium entitled "Am I Famous Yet?", which featured commentaries on fame and merit in psychological research from seven eminent white men and Alice Eagly.*** The new call was issued in response to a chorus of nasty women and other dissidents who insisted that their viewpoints hadn't been represented by the scholars in the original special issue. The new call explicitly invited these "diverse perspectives" to speak up (in 1,500 words or less****).Each of the six of us independently rose to the challenge and submitted comments. None of us were particularly surprised to receive rejections - after all, getting rejected is just about the most ordinary thing that can happen to a practicing researcher. Word started to spread among the rejected, however, and we quickly discovered that many of the themes we had written about were shared across our pieces. That judgments of eminence were biased along predictable socio-demographic lines. That overemphasis on eminence creates perverse incentives. That a focus on communal goals and working in teams was woefully absent from judgments of eminence.*****Hm. It appeared to us that some perspectives were potentially being systemically excluded from Perspectives!****** Wouldn't it be a shame if the new call for submissions yielded yet more published pieces that simply reinforced the original papers? What would be the point of asking for more viewpoints at all?Luckily it's 2017. We don't have to publish in Perspectives for you to hear our voices. We hope you enjoy our preprints, and we look forward to discussing and improving******* this work.The manuscripts:-Katie Corkeron behalf of the authors (Katie Corker, Fernanda Ferreira, Åse Innes-Ker, Cindy Pickett, Lani Shiota, & Simine Vazire)* Kirby has had enough of your shit:
** Technically, some of the six of us got R&Rs, but the revisions requested were so dramatic that we have a hard time imagining being able to make them without compromising the main themes of our original pieces.*** It shouldn't come as a surprise that Eagly's commentary concerned issues relating to gender and power. It was also the only co-authored piece (David Miller was also an author) in the bunch. Eagly and Miller's title was "Scientific eminence: Where are all the women?" Hi Alice and David - we're over here!**** Doesn't appear that the original special issue had such a tight word limit, but who are we to judge?***** We were also disheartened to notice that, ironically, many of themes we raised in our pieces surfaced in the treatment we received from the editor and reviewers. For instance, we were told to provide evidence for claims like "women in psychological science may face discrimination." One reviewer even claimed that white men are actually at a disadvantage when it comes to receiving awards in our field. We collectively wondered why we, the "diverse voices," were seemingly being held to higher standard of evidence than the pieces in the original symposium. Color us surprised that as members of a group stereotyped as less competent, and as outsiders to the eminence club, we had to work impossibly hard to be seen as competent (see Biernat & Kobrynowicz, 1997, among many others).****** On the other hand, it's entirely possible that there were 20 such submissions, and the six of us represent the weakest of the bunch. Hard to get over that imposter syndrome...******* We've chosen to post the unedited submissions so that you can see them in their original form. Of course, the reviewers did raise some good points, and we anticipate revising these papers in the future to address some of the issues they raised.
looking under the hood
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
lipstick on a hippo
before modern regulations, used car dealers didn't have to be transparent. they could take any lemon, pretend it was a solid car, and fleece their customers. this is how used car dealers became the butt of many jokes.scientists are in danger of meeting the same fate.* the scientific market is unregulated, which means that scientists can wrap their shaky findings in nice packaging and fool many, including themselves. in a paper that just came out in Collabra: Psychology,** i describe how lessons from the used car market can save us. this blog post is the story of how i came up with this idea.last summer, i read Smaldino and McElreath's great paper on "The natural selection of bad science." i agreed with almost everything in there, but there was one thing about it that rattled me. their argument rests on the assumption that journals do a bad job of selecting for rigorous science. they write "An incentive structure that rewards publication quantity will, in the absence of countervailing forces, select for methods that produce the greatest number of publishable results." (p. 13). that's obviously true, but it's not necessarily bad. what makes it bad is that "publishable result" is not the same thing as "solid study" - if only high quality studies were publishable, then this wouldn't be a problem.so this selection pressure that Smaldino and McElreath describe is only problematic to the extent that "publishable result" fails to track "good science." i agree with them that, currently, journals*** don't do a great job of selecting for good science, and so we're stuck in a terrible incentive structure. but it makes me sad that they seem to have given up on journals actually doing their job. i'm more optimistic that this can be fixed, and i spend a lot of time thinking about how we can fix it.****a few weeks later, i read a well-known economics article by Akerlof, "The market for "lemons": Quality uncertainty and the market mechanism" (he later won a nobel prize for this work). in this article, Akerlof employs the used car market to illustrate how a lack of transparency (which he calls "information asymmetry") destroys markets. when a seller knows a lot more about a product than buyers do, there is little incentive for the seller to sell good products, because she can pass off shoddy products as good ones, and buyers can't tell the difference. the buyer eventually figures out that he can't tell the difference between good and bad products ("quality uncertainty"), but that the average product is shoddy (because the cars fall apart soon after they're sold). therefore, buyers come to lose trust in the entire market, refuse to buy any products, and the market falls apart.it dawned on me that journals are currently in much the same position as buyers of used cars. the sellers are the authors, and the information asymmetry is all the stuff the author knows that the editor and reviewers do not: what the authors predicted ahead of time, how they collected their data, what the raw data look like, what modifications to the data or analyses were made along the way (e.g., data exclusions, transformations) and why, what analyses or studies were conducted but not reported, etc. without all of this information, reviewers can only evaluate the final polished product, which is similar to a car buyer evaluating a used car based only on its outward appearance.because manuscripts are evaluated based on superficial characteristics, we find ourselves in the terrible situation described by Smaldino and McElreath: there is little incentive for authors to do rigorous work when their products are only being evaluated on the shine of their exterior. you can put lipstick on a hippo, and the editor/reviewers won't know the difference. worst of all, you won't necessarily know the difference, because you're so motivated to believe that it's a beautiful hippo (i.e., real effect).that's one difference between researchers and slimy used car dealers***** -- authors of scientific findings probably believe they are producing high-quality products even when they're not. journals keep buying them, and until recent replication efforts, the findings were never really put to the test (at least not publicly).the replicability crisis is the realization that we have been buying into findings that may not be as solid as they looked. it's not that authors were fleecing us, it's that we were all pouring lots of time, money, and effort into products that, unbeknownst to us, were often not as pretty as they seemed.the cycle we've been stuck in, and the one described by Smaldino and McElreath, is the same one Akerlof explained with the used car market. happily, that means Akerlof's paper also points to the solution: transparency. journals have the power to change the incentive structure. all they need to do is reduce the information asymmetry between authors and reviewers by requiring more transparency on the part of authors. give the reviewers and editors (and, ideally, all readers) the information they need to accurately evaluate the quality of the science. if publication decisions are strongly linked to the quality of the science, this will provide incentives for authors to do more rigorous work.we would laugh at a buyer who buys a used car without looking under the hood. yet this is what journals (and readers) are often doing in science. likewise, we would laugh at a car dealer who doesn't want us to look under the hood and instead says "trust me," but we tolerate the same behavior in scientists.but, you might say, scientists are more trustworthy than used car dealers! sure,****** but we are also supposed to be more committed to transparency. indeed, transparency is a hallmark of science - it's basically what makes science different from other ways of knowing (e.g., authority, intuition, etc.). in other words, it's what makes us better than used car dealers.* if you've listened to paula poundstone on NPR, you might think it's already too late** full disclosure: i am a senior editor at Collabra:Psychology. i tried to publish this paper elsewhere but they didn't want it. i do worry about the conflict inherent in publishing in a journal i'm an editor at, and i have been trying avoid it. my rationale for making an exception here is that publishing in Collabra: Psychology is not yet so career-boosting that i feel i am getting a major kickback, but perhaps i am wrong. also, if it helps, you can see the reviews and action letter from a previous rejection, which i submitted to Collabra for streamlined review. [pdf]*** the other journals.**** sometimes i lay awake in bed for hours thinking about this. well, doing that or reading the shitgibbon's tweets and planning my next act of resistance.***** sorry used car dealers. i have little reason to smear you. well, except for the time you didn't want to let me test drive a car then called me a "bold lady" when i insisted that i could be trusted with a manual transmission. (i was 28 years old.) (and i have always driven a stick shift.) (because i am french.)****** not actually sure.underwater hippos
power grab: why i'm not that worried about false negatives
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
i've been bitching and moaning for a long time about the low statistical power of psych studies. i've been wrong.our studies would be underpowered, if we actually followed the rules of Null Hypothesis Significance Testing (but kept our sample sizes as small as they are). but the way we actually do research, our effective statistical power is actually very high, much higher than our small sample sizes should allow.
let's start at the beginning.
background (skip this if you know NHST)
Null Hypothesis Significance Testing (over)simplified
in this table, power is the probability of ending up in the bottom right cell if we are in the right column (i.e., the probability of rejecting the null hypothesis if the null is false). in Null Hypothesis Significance Testing (NHST), we don't know which column we're in, we only know which row we end up in. if we get a result with p < .05, we are in the bottom row (and we can publish!* yay!). if we end up with a result with p > .05, we end up in the top row (null result, hard to publish, boo). within each column, the probability of ending up in each of the two cells (top row, bottom row) adds up to 100%. so, when we are in the left column (i.e., when the null is actually true, unbeknownst to us), the probability of getting a false positive (typically assumed to be 5%, if we use p < .05 as our threshold for statistical significance) plus the probability of a correct rejection (95%) add up to 100%. and, when we are in the right column (i.e., when the null is false, also unbeknownst to - but hoped for by - us), the probability of a false negative (ideally at or below 20%) plus the probability of a hit (i.e., statistical power; 80%) add up to 100%.
side note: even if the false positive rate actually is 5% when the null is true, it does not follow that only 5% of significant findings are false positives. 5% is the proportion of findings in the left column that are in the bottom left cell. what we really want to know is the proportion of results in the bottom row that are in the bottom left cell (i.e., the proportion of false positives among all significant results). this is called the Positive Predictive Value (PPV) and would likely correspond closely to the rate of false positives in the published literature (since the published literature consists almost entirely of significant key findings). but we don't know what it is, and it could be much higher than 5%, even if the false positive rate in the left column really was 5%.**
back to the main point.
we have small sample sizes in social/personality psychology. small sample sizes often lead to low power, at least with the effect sizes (and between-subjects designs) we're typically dealing with in social and personality psychology. therefore, like many others, i have been beating the drum for larger sample sizes.
not background
our samples are too small, but despite our small samples, we have been operating with very high effective power. because we've been taking shortcuts.
the guidelines about power (and about false positives and false negatives) only apply when we follow the rules of NHST. we do not follow the rules of NHST. following the rules of NHST (and thus being able to interpret p-values the way we would like to interpret them, the way we teach undergrads to interpret them) would require making a prediction and pre-registering a key test of that prediction, and only interpreting the p-value associated with that key test (and treating everything else as preliminary, exploratory findings that need to be followed up on).
since we violate the rules of NHST quite often, by HARKing (Hypothesizing After Results are Known), p-hacking, and not pre-registering, we do not actually have a false positive error rate of 5% when the null is true. that's not new - that's the crux of the replicability crisis. but there's another side of that coin.
the point of p-hacking is to get into the bottom row of the NHST table - we cherry-pick analyses so that we end up with significant results (or we interpret all significant results as robust, even when we should not because we didn't predict them). in other words, we maximize our chances of ending up in the bottom row. this means that, when we're in the left column (i.e., when the null is true), we inflate our chances of getting a false positive to something quite a bit higher than 5%.
but it also means that, when we're in the right column (i.e., when the null hypothesis is false), we increase our chances of a hit well beyond what our sample size should buy us. that is, we increase our power. but it's a bald-faced power grab. we didn't earn that power.
that sounds like a good thing, and it has its perks for sure. for one thing, we end up with far fewer false negatives. indeed, it's one of the main reasons i'm not worried about false negatives. even if we start with 50% power (i.e., if we have 50% chance of a hit when the null is false, if we follow the rules of NHST), and then we bend the rules a bit (give ourselves some wiggle room to adjust our analyses based on what we see in the data), we could easily be operating with 80% effective power (i haven't done the simulations but i'm sure one of you will***).
what's the downside? well, all the false positives. p-hacking is safe as long as our predictions are correct (i.e., as long as the null is false, and we're in the right column). then we're just increasing our power. but if we already know that our predictions are correct, we don't need science. if we aren't putting our theories to a strong test - giving ourselves a serious chance of ending up with a true null effect - then why bother collecting data? why not just decide truth based on the strength of our theory and reasoning?
to be a science, we have to take seriously the possibility that the null is true - that we're wrong. and when we do that, pushing things that would otherwise end up in the top row into the bottom row becomes much riskier. if we can make many null effects look like significant results, our PPV (and rate of false positives in the published literature) gets all out of whack. a significant p-value no longer means much.
nevertheless, all of us who have been saying that our studies are underpowered were wrong. or at least we were imprecise. our studies would be underpowered if we were not p-hacking, if we pre-registered,**** and if we only interpreted p-values for planned analyses. but if we're allowed to do what we've always done, our power is actually quite high. and so is our false positive rate.
also
other reasons i'm not that worried about false negatives:
- they typically don't become established fact, as false positives are wont to do, because null results are hard to publish as key findings. if they aren't published, they are unlikely to deter others from pursuing the same question.
- when they are published as side results, they are less likely to become established fact because, well, they're not the key results.
- if they do make it into the literature as established fact, a contradictory (i.e., significant) result would probably be relatively easy to publish because it would be a) counter-intuitive, and b) significant (unlike results contradicting false positives, which may be seen as counter-intuitive but would still be subject to the bias against null results).
in short, while i agree with Fiedler, Kutzner, & Krueger (2012)***** that "The truncation of research on a valid hypothesis is more damaging [...] than the replication of research on a wrong hypothesis", i don't think many lines of research get irreversibly truncated by false negatives. first, because the lab that was testing the valid hypothesis is likely motivated to find a significant result, and has many tools at its disposal to get there (e.g., p-hacking), even if the original p-value is not significant. second, because even if that lab concludes there is no effect, that conclusion is unlikely to spread widely.
so, next time someone tells you your study is underpowered, be flattered. they're assuming you don't want to p-hack, or take shortcuts, that you want to earn your power the hard way. no help from the russians.*******
* good luck with that.
** it's not.
*** or, you know, use your github to write up a paper on it with Rmarkdown which you'll put in your jupyter notebook before you make a shinyapp with the figshare connected to the databrary.
**** another reason to love pre-registration: if we all engaged in thorough pre-registration, we could stop all the yapping and get to the bottom of this replicability thing. rigorous pre-registration will force us to face the truth about the existence and magnitude of our effects, whatever it may be. can we reliably get our effects with our typical sample sizes if we remove the p-hacking shortcut? let's stop arguing****** and find out!
***** Fiedler et al. also discuss "theoretical false negatives", which i won't get into here. this post is concerned only with statistical false negatives. in my view, what Fiedler et al. call "theoretical false negatives" are so different from statistical false negatives that they deserve an entirely different label.
****** ok, let's not completely stop arguing - what will the psychMAP moderators do, take up knitting?
******* too soon?
psychMAP moderators, when they're not moderating
now is the time to double down on self-examination
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
it can be tempting, when contemplating the onslaught that science is likely to face from the next administration and congress, to scrub away any sign of self-criticism or weakness that could be used against us. as a "softer" science, psychology has reason to be especially nervous.*but hiding our flaws is exactly the wrong response. if we do that, we will be contributing to our own demise. the best weapon anti-science people can use against us is to point to evidence that we are no different from other ways of knowing. that we have no authority when it comes to empirical/scientific questions. our authority comes from the fact that we are open to scrutiny, to criticism, to being wrong. the failed replications, and the fact that we are publishing and discussing them openly, is the best evidence we have that we are a real science. that we are different from propaganda, appeals to authority, or intuition. we are falsifiable. the proof is that we have, on occasion, falsified ourselves.we should wear our battle with replicability as a badge of honor. it's why the public should trust us to get things right, in the long run. it's why it means something when we have confidence in a scientific discovery. we should be proud of the fact that we don't take that word - discovery - lightly. we scrutinize, criticize, attempt to falsify. that's why we will survive attacks on science.yes, our failures will be used against us. we will lose some battles. but if we let those attacks scare us away from self-criticism and self-correction, we will have lost the war.when we find a flaw in our process or in our literature, we need to responsibly communicate what it means and what it doesn't mean. the failed replications i've seen in the last few years have been careful to do that. those producing these critiques and "failures" are not trying to tear us - or anyone - down. they are demonstrating just how tough we are.the next few years are going to get even more challenging, but we need to resist the temptation to give in to a bunker mentality, to shield ourselves from criticism. we need to be even more transparent, even more honest, than we have been until now. we cannot fight ignorance with secrecy, we must face it head on with openness and faith in the scientific enterprise.
* if you came here for the jokes, you're out of luck. this is a Very Serious post. mostly because my hotel room does not have a mini bar.**
** canada is, apparently, not perfect.
a little bit louder now
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the positions or policies of any institution with which I am affiliated.]
i can't even begin to imagine how many times in her life hillary clinton has had to bite her tongue. through everything, through trump, i never saw her even begin to lose her cool.on nov. 8th, i realized that i had assumed that all of that self-control, all of that turning the other cheek, meant that she had earned a win. i made the same gamble on hillary's behalf that women and minorities make every day: that if we take the blows without flinching, if we don't play the woman/minority card, if we keep demonstrating our competence, people will have no choice but to recognize it.one lesson from this election is that this strategy doesn't always work. it assumes a reality that does not yet exist. we will not get extra credit for our patience and forbearance in the face of sexism, racism, homophobia, or xenophobia. no one will pat us on the back for biting our tongues. it is time to start speaking up.i hope that when people watched the debates, and admired clinton's strength, they made the parallel to the less extreme situations that women and minorities face every day. we don't often face someone as vile as trump, but if we want to be successful we have to be prepared to absorb smaller slights on a regular basis, to keep our cool when faced with ignorant, unfair, or offensive comments.why don't we call them out? because it's extremely difficult. because there are too many things to call out. because there is backlash. because we often don't know for sure how much sexism is to blame for any particular event. because often the person being sexist is someone we like and respect, and we don't want to make them uncomfortable. because we often don't want to derail the conversation, or take away from the larger goal of the group. because there is gaslighting. because it's fucking exhausting to speak up.it is very hard to decide whether and when to speak up.* i don't know the right answer, but hillary's loss has convinced me that the answer is: more. so when it feels like the right thing to do, i will try very hard to say something. saying words out loud is not my strong suit, so we'll see how it goes. i'll probably fail, a lot.**i'll also try to speak up more when people are good. when people stick their necks out for women, for minorities, for people who have less voice, less visibility, more to lose. i think one reason i underestimated how much of an uphill battle hillary was facing is because i know a lot of good people. people who believed in me stubbornly, persistently, fiercely. people who treated me as an authority on my own experience, and on other topics, well before i thought i had anything worth saying, or trusted my own voice. i cannot thank those people enough, but i can try to pay it forward.speaking up scares the shit out of me. but if you do it with me, it'll seem a little less scary.*** i'm not talking about being a jerk, i'm talking about being more honest. flinching, if you feel like flinching. giving other people the chance to hear your experience. telling people how things look and sound to you does not make you a nasty woman. and staying silent will not guarantee any reward.* for example, i deliberated for a long time about whether or not to publish this blog post. i'm still not sure about it. presumably if you are reading this i decided to post it. yikes.** i have already failed, twice, since writing this sentence.
*** reading Lindy West's book, Shrill, also makes it seem a little less scary. go read it.
who will watch the watchers?
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
he makes it look easy.i wasn't going to write a blog post about susan fiske's column. many others have already raised excellent points about the earlier draft of her column, and about the tone discussion more generally. but i have two points to make, and poor self-control.point #1.i have a complicated relationship with the tone issue. on one hand, i hate the "if you can't stand the heat, get out of the kitchen" attitude. being able to stand the heat is not a personal accomplishment. it's often a consequence of privilege. those who have been burnt many times before (by disadvantage, silencing, etc.) are less likely to want to hang out in a place with a ton of heat. and we need to make our field welcoming to those people who refuse to tolerate bullshit. we need more people like that, not fewer.on the other hand, i don't think the self-appointed data police are the most egregious source of bullshit. the vast majority of methodological freedom fighters* are doing something good for the field. they are literally doing the things i teach my undergrad research methods students to do. they are reading papers, critiquing the methods, double-checking the stats, evaluating the conclusions. if more people want to volunteer their time to check our work, we should welcome them. even if they are robots.** i'd rather spend my energy trying to fix the other sources of bullshit that i think are much more pernicious - mainly the fact that many early career researchers receive inadequate support and are often taken advantage of in various ways. we don't treat our young well, and we need to do better.*** this is not new to this era, and it doesn't get enough attention.so, point #1: let's turn down the heat in the kitchen so that everyone feels welcome, but the source of the most heat is not the data police. it's the same source that has been creating heat since the beginning of mankind: status/power/privilege/entitlement.point #2.fiske's column places a great deal of emphasis on the importance of gatekeepers. she criticizes some forms of communication for being "uncurated" and "unmoderated." instead, she advocates for critics to make "their rebuttals and letters-to-the-editor subject to editorial oversight and peer review." she praises changes made through APS because "APS innovates via expert consensus" and this way research is "judged through monitored channels." she prefers these channels because they "offer continuing education, open discussion, and quality control."full disclosure: i am one of those gatekeepers.**** i am also a board member of APS, and proud of some of the things the organization has done. but i find this line of argument extremely problematic.*****the "expert consensus" and "quality control" that fiske trusts is a big part of what got us into this mess. i think we can forgive our colleagues if they no longer feel that the gatekeepers are always wise arbiters of quality.moreover, if a reader wants to critique a published paper, especially if she believes she has found a major flaw in the paper, she is implicitly critiquing the journal it was published in and it's expert judgment. it seems unreasonable to me to require that she only be allowed to express that criticism if it meets with the approval of the journal's editor (and reviewers, who are likely to include the author whose work is being critiqued).more generally, gatekeepers need to be held accountable, and open to criticism. fiske seems to want to grant them more power and control over what views are allowed to be expressed. in contrast, i would like there to be more avenues for people to express criticisms of the system. what fiske calls "quality control" i would call a system that is responsive to some perverse incentives and that is designed to protect itself and its reputation.the healthiest thing we can do for the peer review system is to give its critics a voice. we should expose the process and its flaws to the sunlight, including through uncensored post-publication peer review. journals, editors, and societies should welcome open discussion of the current system, and post-publication critiques of their output. we bemoan how hard it is to find reviewers before publication, and then smear the people who donate their time to post-publication review. if we care about the quality of our published record, we should appreciate both kinds of feedback.a while back on facebook, someone jokingly suggested a ratemyeditor.com website. it wasn't a serious proposal, but it reflects a real concern: what are researchers supposed to do when they want to critique the system? who will watch the watchers? rather than concentrate even more power in the hands of the gatekeepers, i say let's open the doors to more voices.* credit for this phrase goes to sanjay srivastava.** i'm not gonna name names, but a certain someone does walk a lot like a robot. not saying who, just saying, would a non-robot come up with the phrase "methodological freedom fighter"?*** i was lucky to have excellent support from my advisor and many others as an early career researcher, so i am mostly not speaking from personal experience. except for that time a famous social psychologist walked up to me and grabbed my ass at my first SPSP. true story.**** in case it's not already super obvious, the views i express here do not necessarily reflect the views of any journal or society i am affiliated with.*****
i have found the solution and it is us
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
bear, having recently joined SIPS
i have found scientific utopia.*
sometimes, when i lay awake at night, it's hard for me to believe that science will ever look the way i want it to look,** with everyone being skeptical of preliminary evidence, conclusions being circumscribed, studies being pre-registered, data and materials being open, and civil post-publication criticism being a normal part of life.then i realized that utopia already exists. it's how we treat replication studies.i've never tried to do a replication study,*** but some of my best friends (and two of my grad students) are replicators. so i know a little bit about the process of trying to get a replication study published. short version: it's super hard.we (almost always) hold replication studies to an extremely high standard. that's why i'm surprised whenever i hear people say that researchers do replications in order to get an 'easy' publication. replications are not for the faint of heart. if you want to have a chance of getting a failed replication**** published in a good journal, here's what you often have to do:
- carefully match the design and procedure to the original study. except when you shouldn't. adapt the materials to your local time and culture, but don't change too much. pre-test your materials if possible. get the design approved by people who probably wish you weren't doing this study.
- pre-register everything. be thorough - expect people to assume you were biased (more biased than the original researchers) and that these biases might have led you to inadvertently cherry-pick.
- run a shitload of participants.
- repeat.
- one more time, you forgot a potential moderator that the reviewers just thought of.
- make sure that nowhere in your manuscript do you state with confidence that the evidence shows there is no effect. admit that you might have made a type II error. circumscribe all your conclusions and give serious consideration to alternative explanations, even if you think they're very unlikely.
- make all of your materials publicly available. expect people to question your expertise, so provide as much information as possible about all details of your method.
- make all of your data publicly available. this is not optional. you don't get the same liberties as original researchers - open data is an imperative.
- get through peer review, with a decent chance that one of your reviewers will perceive your manuscript as a personal attack on them.
- once your paper is published be prepared for it to be picked apart and reanalyzed. your conclusions will not be allowed to stand on their own, they will be doubted, and your results will be discounted for a variety of reasons.
- you will told by well-respected senior scientists that you are harming your field, or wasting your time, or betraying your lack of creativity, or that you must be trying to take people down.
from the tone of my writing, i wouldn't blame you for assuming that i think we are unfair to replication studies. actually, i think we are unfair to original studies. i would like us to treat original studies more like we treat replication studies (except maybe that last point).imagine, if you can, a world where we hope for the following in original research:
- pour tons of thought and effort into the study design before you think about collecting data. show the design and procedure to skeptics who don't believe your theory. let them pick it apart and make changes. build a study that even they would agree is a strong test.
- think about whether your materials would work in other times and places, so that researchers who want to replicate your study in other contexts know what factors to consider.
- pre-register your study. document everything assuming people will want to rule out the possibility that you got your result by inadvertently exploiting flexibility in the research process.
- run a shitload of participants.
- assume your readers will not believe your first study results, and will propose hidden moderators. repeat it a couple times just to be sure, testing some potential moderators along the way (pre-register these, too).
- draw very circumscribed conclusions. seriously consider the possibility that your result is a type I error. entertain alternative explanations that are not sympathetic to your theory, and do not wave them away. sit with them. cuddle them. let them into your heart.
- make all of your materials and data publicly available.
- tell us about anything in your file drawer, let readers decide if it really belongs in the file drawer.
- expect others to reanalyze your data, criticize your design, analyses, and conclusions, and propose post-hoc alternative explanations. admit that they might be right, we can't know for sure until future research directly tests these explanations.
- be accused of letting your bias creep in, in ineffable ways that cannot be cured by pre-registration and open data, and accept that this is a possibility.
- accept that it will take a lot more to determine if the effect is real. consider an adversarial collaboration.
of course not every study (original or replication) can have all of these features, and we should be flexible about tolerating studies with various limitations. there are important side effects of raising standards in absolute, black and white ways. we need room for studies that fall short on some criteria but are important for other reasons. i know, because my own work falls far short of many of these ideals. but these are nevertheless ideals we should strive for and value.one of the beautiful things about replication studies is that they make everyone see the value of these practices. people who didn't care much for open data, pre-registration, etc., are often in favor of those practices when it comes to replication. people who are resistant to the idea that original researchers' motives and biases could lead to systematic distortions in the published literature are more sympathetic to this concern when it comes to replication researchers' motives and biases.can you imagine what we'd say if a replication author refused to share their data or materials for no good reason? can you imagine the reaction if we found out they hid some studies in the file drawer?***** can you imagine how we'd react if their conclusions went way beyond the evidence?there seems to be a consensus that replication studies, including materials, data, and results, belong to all of us. we feel we are owed complete transparency, and we expect to all have a say in how the studies should be evaluated and interpreted.those are the correct reactions, but not just to replications. these reactions provide an opportunity to build on this new common ground. let's apply those values to all research, not just replications.* no, i don't mean SIPS. i'm barely even going to mention SIPS in this blog post. or our new website. with its new domain name: http://improvingpsych.org/***** hypocrisy is one of my favorite pastimes. for example just this morning i ate a corndog for breakfast while simultaneously believing people should refrain from eating meat AND should eat a healthy balanced breakfast.
**** not that we know whether a replication will succeed of fail ahead of time (in all the cases i know of, the replicators were genuinely curious about the effect and open to any outcome). but i'm talking about failed replications here because i think they're subjected to more scrutiny. if the replication results come out positive, you face different challenges in getting those results published (some people think it's harder to publish successful replications than failed ones, but the evidence is not clear).*****
cj and jöreskog, having recently discovered GIFs
don't you know who i am?
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
elephant seal, throwing his weight around
when i started my first job as associate editor, i was worried that i would get a lot of complaints from disgruntled authors. i wasn't afraid of the polite appeals based on substantive issues, i was worried about the complaints that appeal to the authors' status, the "don't you know who i am?" appeal.i never did get that kind of response, at least not from authors. but i saw something worse - a pretty common attitude that we should be judging papers based, in part, on who wrote them. socially sanctioned status bias. not so much at the journals i worked with, but in the world of journals more broadly. like the Nature editorial, on whether there should be author anonymity in peer review, that argued that "identifying authors stimulates referees to ask appropriate questions (for example, differentiating between a muddy technical explanation and poor experimental technique)." the argument seems to be that some people should be given a chance to clear up their muddy explanations and others should not. or the editor who wrote in The Chronicle of Higher Education just a few days ago that "Editors rarely send work out to trusted reviewers if it comes from unproven authors using jazz-hands titles." leaving aside the contentious issue of jazz-hand titles, when did we accept that it was ok to treat papers from 'unproven authors' differently?once or twice, i've even been explicitly pressured to treat a manuscript differently because of the prestige of the authors or institutions they were from. that pressure didn't come from the authors, it came from someone at the journal.* since then i've learned that this is pretty common - some journals and editors want to publish papers by famous people, because they think this will improve their impact factor (they're probably right). in fact, some of the very top journals actively solicit manuscripts from the elite, and those are often given less scrutiny than unsolicited submissions.you probably already knew that, but i was naive and finding this out kind of blew my mind.** i thought maybe some arrogant people would think they deserve special treatment, but i didn't think the system would give it to them, and defend that practice with a straight face. the Great Man (or Great Woman) culture is stronger than i thought. there will probably always be this status bias for decisions like who gets to give keynotes, who gets media attention, and who is able to get money from private foundations. but - and i know i'm too old to still be this naive - we should eradicate this attitude from the peer review process, especially for publications (grants are more complicated because you're evaluating the person and the project, but i still think status bias is a huge problem there).the more i edit, the more i think we need to resist this Great Man culture. who you are and what institution you're from should have no bearing on how your manuscript is handled during the review process. there should certainly not be any explicit favoritism towards high status people, and we need to be vigilant against implicit status bias, too.for this reason (and others), i have become a big fan of blinding authors' identities during the review process. i know it doesn't always work,*** but it's the right idea, and i can't think of any way in which it causes harm. (the empirical evidence on blinding is completely inconclusive, so that suggests that, for now, we should decide on ethical grounds. and the ethical considerations seem, to me, to squarely favor blinding.)in fact, i'm such a fan of not knowing who the authors are when i review a paper that i have decided not to look at who the authors are when i first receive a manuscript as editor. it's hard - it's tempting to look, and also i literally have to hold my hand up to my computer screen to hide the information from myself - but after doing this for several months, i'm convinced it has improved my editing.****of course, if i decide to send a manuscript out to reviewers, at that point i need to look at who the authors are so i can make sure to avoid conflicts of interest. however, when i am first reading a new submission and deciding whether to desk reject it or not, and when i am writing the desk rejection letter, i don't need to know who the authors are. after writing the desk rejection letter but before sending it, i look at who the authors are, and very rarely, i find out i have a conflict of interest. in that case, i assign it to an associate editor who does not have a conflict of interest, and let them make the decision. otherwise i send the letter i wrote before i knew who the authors were. (i've occasionally added a sentence or two to the letter, based on my knowledge of the authors' past work, but have not changed my decision.)this is not a policy at any journal i work with - no journal that i know of has a policy of blinding the editors to the authors' identities. it's a choice i make - everyone has their own style of editing, what information they choose to look at in what order. i choose not to know who the authors are for as long as possible.this forces me to evaluate the work on its own merits. i can't give someone the benefit of the doubt because i know they've done good work in the past, and i can't unconsciously penalize someone because i've never heard of them or their institution. you would think that it would often be obvious who the authors are, or at least what institution they're from, but so far i've only been able to guess correctly a small fraction of the time (less than 5%). maybe i suck at guessing, but even if someone was better at it than me, guessing is very different than knowing. thinking that a manuscript was submitted by someone i think highly of, but not knowing for sure, makes me less likely to be influenced by that belief.overall, it's kind of annoying and a lot less fun to read manuscripts without knowing who the authors are. but there are several big advantages. first, and most importantly, i think it's more fair to the authors. second, it allows me to know, for sure, how i would have evaluated the manuscript if i didn't who the authors were. it's happened to me a few times already that, upon revealing the authors' identities to myself, i was quite surprised, and had a very distinct feeling that i might have let my biases creep in if i had known their identities when reading the manuscript. (despite the fact that, of course, i try very hard not to let that influence me. i know some social psychology.) third, this practice makes me feel better about the necessary interconnections that naturally exist in any small field. for example, as editors and reviewers, we often find ourselves at the same bar or dinner table as another researcher, and then the next day we receive their manuscript to handle or review. by blinding myself to the authors' identities when i make the initial desk reject decision, i can rest easy when i socialize with researchers knowing that even if i do have to make a decision on their work the next day, it won't be personal.a small tangent on conflicts of interestanother common argument against blinding authors' identities from reviewers is that this makes it harder for the reviewers to know if they have conflicts of interest. i would be more sympathetic to this concern if i saw evidence that reviewers (and editors) take these conflicts of interest seriously when they do know the authors' identities. in my experience, people don't recuse themselves nearly as much as they should. i know because i used to do it, too. i believed i could be impartial (and my pattern of rejecting my friends' papers over and over again gives some plausibility to this claim*****). but i now realize that this is not the question i should be asking myself. the question i should be asking myself is, if the world knew that i was a reviewer or the editor for the manuscript, would they think that was unfair? if the answer is yes, we should recuse ourselves.******back to the main pointi've only rarely heard someone use the "don't you know who i am?" appeal - much more often, it's used in the third person. as in, "we can't say no to [famous person]." i guess i still haven't shaken the young idealist in me that fell in love with academia in part because it was supposed to reject such appeals to authority or fame. i am not disillusioned - i think we often do a good job of evaluating research on its merits. i also had the great privilege of having role models like Dave Kenny, Laura King, and Rich Lucas - people who taught me how to be a reviewer and editor, and who also exemplify how not to let success go to your head, who refuse to play the Great Man/Woman role. we can all contribute to that culture by trying to combat - and protect ourselves from - our own biases. let's not confuse status with quality - the two may be correlated, but relying on that heuristic does a disservice to science.* not that journal, another journal.** frequent readers of this blog will notice that my mind gets blown pretty often.*** you know what works even less? telling reviewers who the authors are. if the argument against blinding authors' identities is that reviewers might guess who the authors are, how is that an argument for telling the reviewers who the authors are?for other (in my view, pretty weak) arguments against blind review, see this: http://pps.sagepub.com/content/4/1/62.full.pdf+html. **** i also learned that if you work at cafes with your hand covering part of your computer screen, people will sit as far away from you as possible.***** sorry alexa.****** at SPPS the handling editor's name is now published with each manuscript, so this hypothetical is now a reality for the editors. in fact, you'll notice that, in the july issue of SPPS, i was the handling editor on a paper with brian nosek as an author. i assigned the manuscript to myself before brian and i really started collaborating on SIPS (and at the time i was not friends with him, except in the facebook sense*******), but even then i should've known better. i'll try to do better in the future, and being identified on articles as handling editor will help keep me honest.******* he finally added me to his professional network on linkedin.
unproven authors.
your inner third grader
[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.]
it felt like a confessional.
'sometimes, we say we predicted things that we didn't actually predict.' i paused, embarrassed. 'i know.'
'i'm sorry,' she said, 'but that sounds like something even a third grader would know is wrong.'
'i know.'
i tried not to make excuses, but to explain how this happened. how an entire field convinced itself that HARKing (Hypothesizing After the Results are Known, Kerr, 1998) is ok.
we all do it, and we all know we do it, i explained, so we basically know to take 'predictions' with a grain of salt. what we didn't know is how much this corrupts the process of science, inflating the rate of false positives. contrary to what Bem wrote in his classic paper on how to write an empirical article, science does care whether you predicted your results. certainly, p-values care.
but perhaps more importantly, it's just wrong to say you predicted something when you didn't. any third grader could tell you that. it's true outside of science, and it's especially true in science - we have a special duty to be honest about our process.
the fact that we could forget this (and i include myself, 100%) blows my mind. it serves as a signal to me about how much we need to re-examine our practices. just because everyone* thinks a research practice is perfectly fine doesn't mean it is.
Kerr's 1998 article has an astounding number of good ideas about how we got into this mess and how to get out of it. we need to destigmatize the presentation of exploratory findings as exploratory. we need to encourage replications. we need to clearly distinguish between confirmatory and exploratory research.** we need to remember our ethical commitments as scientists. i know that last one sounds super idealistic, but i am ashamed that i convinced myself that 'telling a good story' was more important than being honest. i know, the situation was powerful, but still.***
after my confession, this person asked me: 'why is psychology so messed up?'
that's when i knew i was going to ruin her day. it isn't just psychology, i told her. outcome switching (the medical equivalent of HARKing) is rampant in clinical trials,**** and probably in many sciences. psychology is probably not exceptional, except maybe in our efforts to face our problems, and do better.
i'm sure there are other ways in which we've abandoned the ideals we all held about scientists when we were third graders - the ideals we teach our undergrads. let's listen to our inner third graders. some things we do are hard to defend, and it's tempting to say 'don't worry, that's just how science is done.' but that should be a red flag. science means never having to say 'trust me.'
* everyone except norb kerr.
** i know it's a continuum, but because of things like HARKing, we can't have nice things for a while. i think that for now, until we've gotten over our habit of playing fast and loose with the word 'prediction', confirmatory = pre-registered. or, as i overheard at SIPS: let's put the 'pre' back in prediction.
*** i hope the situationists will let me into their club, cause i'm definitely with them on this one. the problem - and the solution - is not individual researchers but the system in which they operate. change the system (journals, hiring committees, award committees, etc.) and individual researchers will change. then again, norb kerr was subject to the same situational pressures, so there's that. damnit, i liked that club.
**** ok, so pre-registration isn't going to help much if people are just going to lie about that, too. but at least with pre-registration, everyone can see that you HARKed. and since pre-registration is relatively new to psychology, we can start with a clean slate and firmly establish the norm that lying about one's pre-registration should be considered fraud.*****
***** this whole post is starting to sound pretty preachy, despite the fact that i drank a generous shot of whiskey before writing it. i predict that whiskey is not causally linked to decreased self-righteousness.
SPPS Special Issue on Research Methods
Social Psychological and Personality Science is now accepting submissions for a forthcoming special issue on “New developments in research methods for social/personality psychology.”
Recent advances in research design (e.g., crossed designs; Westfall, Kenny, & Judd, 2014), analysis (e.g., Bayesian approaches; Wagenmakers et al., under review), and meta-science (e.g., p-curve; Simonsohn, Simmons, & Nelson, in press) have opened up new possibilities for improving research methods in social and personality psychology.
We are looking to collect a number of high quality articles that address one or more of the following goals:
• Present new methods for the design, collection, or analysis of social/personality studies
• Present existing, underused methods in a new, more accessible way that will encourage their adoption, when appropriate, by social/personality researchers
• Present empirical evidence concerning the utility, strengths, and limitations of social/personality research methods (including meta-science)As usual, articles cannot exceed 5,000 words (excluding tables, figures, and references) and preference will be given to articles that are written to be accessible to a wide range of audiences.
Deadline for submissions: August 31, 2016
Submit your manuscript online at mc.manuscriptcentral.com/spps
the good, the bad, and the ugly
one of the themes of the replicability movement has been the Campaign for Real Data (Kaiser, 2012). the idea is that real data, data that haven't been touched up by QRPs, are going to be imperfect, sometimes inconsistent. part of what got us into this mess is the expectation that each paper needs to tell a perfect story, and any inconsistent results need to be swept under the rug.whenever this comes up, i worry that we are sending researchers a mixed message. on one hand we're saying that we should expect results to be messy. on the other hand we're saying that we're going to expect even more perfection than before. p = .04 used to be just fine, now it makes editors and reviewers raise an eyebrow and consider whether there are other signs that the result may not be reliable. so which is it, are we going to tolerate more messiness or are we going to expect stronger results?yes.on the face of it, these two values (more tolerance for messiness vs. more precise/significant estimates) seem contradictory. but when we dig a little deeper, i don't think they are. and i think it's important for people to be clear about what kind of messy is good-messy and what kind of messy is bad-messy.recently, a colleague told me about a result he got that looked pretty robust - it was strong and significant when he analyzed it six out the seven ways possible. but the seventh analysis was not significant. this is probably good-messy. this is the kind of messy that you should absolutely report,* and nobody should really care very much about. assuming the seventh analysis is equally diagnostic as the other six (i.e., it's not a clearly better way to test the question), and the first six results give very compelling evidence (e.g., the confidence intervals are pretty tight, the p-values are mostly close to zero), we should probably chalk up that seventh result to random noise and move on.good messy is when you have mostly very strong evidence (precise estimates, p-values close to zero, etc.), but every now and again there's a misbehaving result. if the misbehaving results are about as frequent as you'd expect by chance when there is a true effect (i.e., if the proportion of null results is consistent with what you think your type II error rate should be), then it shouldn't be a cause for concern. if a reviewer gives you a hard time for it, or asks you to over-interpret it, send them over here.**bad messy looks pretty different. bad messy is where most of the individual results are unconvincing. the prototypical case is one where there are a string of underpowered studies with results just below p = .05. by now most of us know that this is a sign that the result is unreliable, probably due to perfectly common things like HARKing, flexibility in data analysis, or a file drawer. what we've learned is that, when your studies are underpowered, it's super unlikely that you'll get only significant results. what that DOESN'T mean is that throwing in one not-quite-significant result makes things ok. sorry. i know that seems like the obvious conclusion. if too-many-underpowered-but-significant-studies = bad, then it seems perfectly reasonable that many-underpowered-but- significant-studies-plus-one- or-two-not-quite-significant- studies should be good. and, sure, it's better. but it's still not great. if none of the individual studies are high-powered and produce compelling evidence (a highly precise/significant result), a bunch of studies that kinda sorta suggest something is still not going to be convincing. why not? why can't you just do a meta-analysis and show that, overall, you have clear evidence for a result? the blunt answer is because of p-hacking and QRPs. if your studies are pre-registered and you did not deviate from the pre-registered analysis plan,*** and you include all studies, then yes - meta-analyze away and the result should carry a lot of weight (though see this paper about how optional stoping at the study level could threaten the results of your internal meta-analysis). but in the absence of pre-registration, it's entirely possible that each study in the set is a false positive. i know that makes me sound cynical. that's because i am. i'm not cynical about researchers' intentions - i truly believe that almost nobody thinks they're p-hacking and almost everybody believes their effects are real. i'm cynical about people's abilities to recognize when they're capitalizing on chance.how do you know when your results are good messy or bad messy? i think there's one easy question you can ask yourself that provides a pretty good guide: could i clear up the messiness with a new high-powered, pre-registered study? if the answer is yes, then your data are probably bad-messy. that is, if you haven't already done a high-powered study or a pre-registered study, or ideally a high-powered pre-registered study, and your results are messy, there's a chance it's because none of your studies were adequately powered or free of p-hacking. doing a high-powered pre-registered study would likely push the effect either into the 'pretty sure it's real' or the 'pretty sure it's not real' column, and there's not much reason to publish the messy result now when you could do one more study and make it a much-less-messy result.if, on the other hand, the evidence you have is based on one or more high-powered studies, and especially if at least one of those studies was pre-registered, then another high-powered and/or pre-registered study is not likely to provide much more clarity (though of course it might). even with high-powered studies, and even without p-hacking, we're going to see fluctuations in results. those kinds of fluctuations are often just what you'd expect because of noise, type II error, etc. we'll never be able to make them go away completely, and those are the kinds of fluctuations we should tolerate.it sucks when editors and reviewers ask you to do impossible, contradictory things. more messiness! more perfection! which is it? in this case, i think it's both. whether messiness is good depends on the type of messiness and the reason for the messiness. but i fear that repeating 'we should tolerate more messiness' is going to lead to a lot of disappointment, because i don't think it means what a lot of people might hope it means.post-script: you know what else sucks? two years ago i said that high-powered studies were the solution to everything. now it's high-powered pre-registered studies. what's next, high-powered pre-registered blindfolded underwater studies? perhaps. the goal posts keep moving, because we keep learning more about the problems and the solutions. i'm still obsessed with power, but also coming to the realization that pre-registration is going to be a big part of the new way we do science. it's a painful realization for someone who uses mostly existing data that i know too well to be able to pre-register anything with. i will work these feelings out in a future blogpost.* in fact, it would be perfect for the 'most damning result' section.** i can take 'em. i was on my high school wrestling team.*** i'm not saying that if you pre-register, you can't deviate from the pre-registered analysis plan. but if you do, those findings are now exploratory and the meta-analysis is not longer bias-free. and a bias-full meta-analysis is maybe completely useless. ******** who wants tequila?
is this what it sounds like when the doves cry?
there are so many (very good) stories already about the RP:P, it's easy to feel like we're overreacting. but there is a lot at stake. some people feel that the reputation of our field is at stake. i don't share that view. i trust the public to recognize that this conversation about our methods is healthy and normal for science. the public accepts that science progresses slowly - we waited over 40 years for the higgs boson, and, according to wikipedia, we're still not sure we found it. i don't think we're going to look that bad if psychologists, as a field, ask the public for some patience while we improve our methods. if anything, i think what makes us look bad is when psychology studies are reported in a way that is clearly miscalibrated, that makes us sound much more confident than scientists have any right to be when just starting out investigating a new topic.
what i think is at stake is not the reputation of our field, but our commitment to trying out these new practices and seeing how our results look. in the press release, Gilbert is quoted as saying that the RP:P paper led to changes in policy at many scientific journals. that's not my impression. my impression is that the changes that happened came before the RP:P was published. i also haven't seen a lot of big changes. what i've seen is one or two journals make a big change, and a bunch make small changes. Gilbert's comment seems to imply that he thinks these changes should be rolled back. even if you don't accept that the RP:P is informative, it seems very strange to me to conclude that the replicability debate is over, and that journals should roll back their policy changes. if that's what's at stake, this is an incredibly important discussion.
it's tempting to stop there and just say, whatever we all believe about the RP:P, let's focus on the future and making things better. but i feel compelled to explain why Gilbert et al.'s critique of the RP:P leaves me unsatisfied, because if i did find it very compelling, i would want to know other people's reasons for not being compelled.
let me be clear. although i continue to believe the RP:P provides some evidence that our studies are not as replicable as i would like (with a wide margin of error around that estimate), i think it's pretty reasonable to draw different conclusions. specifically, i think many of the replication studies in the RP:P, like many of the original studies, were underpowered. for this and other reasons, i agree the RP:P was flawed, and the replicability rate was suppressed by these flaws. but i'm super uncomfortable with jumping from that conclusion to the one that there is no replicability problem and the discussion is over.
if you don't think the RP:P was informative, what has to come next is a proposal about what would be a more ideal test. it's not enough to say this replication project was flawed and that replication project was flawed, so let's not use replications to test our findings. the replicability of psychological science has to be open to empirical assessment. if not the RP:P and the RRRs and the Many Labs and the hugely-powered, careful replications of key effects, then what? i would really like to know what empirical evidence would convince the critics that we do have a replication problem.
i imagine one answer someone could give is that the empirical basis for their beliefs is all the conceptual replications that exist. the problem with this is that it answers 'why do you believe we don't have a replicability problem?' but it doesn't answer 'what would convince you that we do?'. to protect ourselves against confirmation bias, we have to be able to state what would falsify our beliefs. the existence of successful conceptual replications isn't an answer to the question i'm looking for, because if the conceptual replications fail, we aren't likely to take that as evidence that we have a problem. maybe we shouldn't, that's ok, but we have to be able to name what would make us change our minds, and commit ourselves to that evidence, regardless of outcome.
i understand why people want the flexibility to interpret the results of a study after the fact, with freedom to use reasoning and critical thinking to consider possibilities that they didn't consider ahead of time. but we are human, we know that we are very very good at abusing reasoning and critical thinking in the service of confirmation bias. so we have to tie our hands to some extent. we have to pre-commit to accepting certain results. we have to be able to design a replication study that we ourselves feel comfortable committing to as a valid test. and, if we really don't want to accept the results that we had pre-committed to accepting, we can always run a new pre-registered study to demonstrate that our first pre-registered study was misguided.
indeed, the pre-commitment to accepting the results as informative is what greatly increases the informational value of the RP:P and RRRs and Many Labs projects. they are like conceptual replications of each other, but unlike most conceptual replications, they used pre-registration and open data & materials to tie their hands and reduce the opportunity to exploit flexibility in data collection and analysis. i trust these conceptual replications in large part because the process was so constrained and so transparent. i am on board with conceptual replications when they are conducted in a way that would force the authors to accept that any outcome should be treated as informative, success or failure.
after-the-fact scrutiny of the results is useful, but it has to be taken as less reliable, and more susceptible to human bias, than conclusions that were constrained by pre-registration. we can use these post-hoc explanations as a starting point for follow-up studies, but they are not an end point. if you believe the RP:P, or any other piece of evidence, is uninformative, it's great to point out the flaws, but then you also have to tell us what an informative study would look like, and that answer has to be something that could make you reconsider your beliefs.
think of it as an invitation. if Gilbert et al. laid out what empirical demonstration would convince them that the old way of doing things was problematic, i'm pretty sure the field would spring into action to conduct those tests.
but to be honest, i'm not sure there is any empirical evidence that would convince them, and my goal is not to convince them anyway. my goal is to explain why i am dissatisfied with the post-hoc critiques of each new piece of evidence that comes out suggesting that we have a problem. it's not that i don't think the critics make some valid points. it's that i don't see the critics taking the necessary next steps. they aren't making any risky predictions, or opening themselves up to possible disconfirmation.
turning that question around on myself, what empirical evidence would convince me that published social/personality psych studies are as replicable as we would like, that the old practices are sound? my answer is: if people start pre-registering their study design and analysis plan, test similar questions, and the results look the same (same distribution of p-values, same effect sizes) as they have in the past. that would definitely convince me, and i would accept the arguments of those who want to go back to doing things the old way.
i would love a similarly concrete answer from the Gilbert et al. crowd. but maybe i'm just too demanding.
it's the end of the world as we know it... and i feel fine
what, me worry?several people have told me recently that they are incredibly depressed about the news from psychology (and medicine, and political science, and economics, and biology...). which has made me wonder, why am i completely fine?
let's be real. one reason i'm not upset is because i have tenure (and a job that i love). my heart goes out to everyone trying to navigate this brave new world without job security.
but even some of my tenured friends are depressed. so what's my deal? maybe it's just my sunny disposition.* more likely, i think i am particularly good at focusing on the right counterfactuals.
when we learn that the evidence for one of the most well-known effects in our field (e.g., ego depletion, but it will happen to other effects as well, so pick your favorite effect and insert it here) is extremely shaky, it's natural to be sad and to think of the most obvious counterfactual: i wish the effect was true.
it's easy to feel like it was true until those goddamn replicators came along and destroyed it. but the reality is, if it's not true, it was never true. so, to me, the more useful counterfactual is: what if we had never discovered the flaws and continued to be pretty certain it was true?
the same goes for our research practices. it might be tempting, when we're told that social/personality psychology research actually requires not 60 but more like 600 participants per study,** that the world just got a little bit uglier. it might be easy to think of the counterfactual: i wish i could still do informative research with 60 participants.
but we never could. we just thought we could, but we were always wrong. so, the more appropriate counterfactual is: what if we had never discovered that our practices were flawed and we went on believing we could run informative studies with 60 people?
when i think of these more relevant counterfactuals (what if we hadn't learned any of this 'bad news' and kept doing things as we had been?), i am ecstatic. i am so glad we are figuring this out. i am incredibly proud of our field for doing the hard and unpleasant work of questioning and correcting ourselves. isn't it amazing that we are figuring all this out and we're doing something about it? i feel lucky to be a part of this process, and excited that, going forward, our research is going to be more informative.
also, i feel like it is much more exciting to be living through this period in the history of psychology than what i had originally signed up for. when i became a researcher, i thought that, at best, i'd contribute a few findings to the literature and then i would die and the world would be exactly the same.*** but now i get to be part of something that i think will change the course of psychology. my role is tiny, but i still think the impact of participating in these changes will be bigger than anything i had imagined when i signed up for this gig. and that makes me happy.
sometimes i worry that my not being upset about all this comes across as glee in the face of other people's sadness. it's not. it's just that i can't for a second wish that we hadn't learned these lessons, that we were still in the dark. of course some of the changes are going to be painful, and that's a blog post for another day. but the alternative is not a world where everything was easy and our effects were robust. that was never the reality. the only relevant counterfactual is the one where we remain ignorant.
so if you miss the old days when we could be blissfully ignorant and just keep doing our research the old way, i can't relate. i feel so grateful that we now know what we were doing wrong, and have a chance to do better. i don't wish we could go back to doing cheap and easy studies, because we never should've been doing them in the first place. it turns out we were not being frugal, we were being incredibly wasteful.
do i think we're stupid for having thought we could do research on the cheap? no. people had been screaming and shouting about statistical power for a while, but we needed the missing piece of the puzzle, p-hacking, to fully understand why power was such a big problem. until p-hacking, the main danger of low power was type II error (false negatives), and if we were still producing plenty of significant results, then why should we worry about power? now we know why, and we are changing our practices. but until we knew about p-hacking and QRPs, it made sense to trust our significant results. thank god we know now.
* ok, maybe not sunny, but at least like sitting under a nice shady tree on a warm day. with a light sweater. or, as my grad students put it, 'very hard to annoy.' (possibly the nicest thing anyone has said about me.)** relax. i just picked a number that starts with 6. it's more like 550.
*** except the 1999 carleton college badminton barber bonanza. i was pretty key to that.
it could be worse.
and now for something a little more uplifting
if you read my blog, you might think everything is shit and we might as well go drink whiskey and play euchre. that's definitely plan b. but for now, i'm sticking with plan a: SIPS.* you should come, too.
Society for the Improvement of Psychological Science (SIPS)**
Inaugural Meeting
June 6-8th, 2016
Center for Open Science, Charlottesville, VASIPS is a new group created to bring together scholars working to improve methods and practices in psychological science. The aim of the inaugural meeting is to generate ideas, goals, and actionable plans to improve psychological science, including:
-Improving the training and research practices in psychological science
-Improving institutional practices to incentivize better scientific practices (e.g., journals, societies, departments, and universities)
-Conducting meta-science, empirical tests of reforms, and critical self-evaluation
-Outreach within and outside psychology (including attention to diversity)
The meeting will be a dynamic agenda of very brief presentations, open discussion, break-out work, and action planning. We have a draft agenda here.
If you're interested in coming, sign up here!Because of practical constraints, registration is limited to approximately 60 participants for the inaugural meeting. Beyond the organizing team, registration will be on a first-come, first-served basis and we will create a wait-list should interest exceed capacity.
* not mutually exclusive with plan b.
** you know i'm serious because i'm using capitals.***
*** don't get used to it.
plan c
fifty million frenchmen can eat it
the struggle is real.
this blog post is an attempt to lay out my reasoning about why i think it's safe to conclude that p-hacking is a big problem, and false positives are a big problem, as clearly and bluntly as i can.
there have been grumblings online about a new Registered Replication Report (RRR) about to come out showing that the meta-analytic result of 20 pre-registered replications of an ego depletion study is pretty much zero.
it might seem like jumping the gun to write a blog post about it before it’s come out. that's because it is jumping the gun. but i’m doing it anyway, because i think the most important conclusion is not about ego depletion. the most important conclusion is that we need to accept that 50 million frenchmen can be wrong.
throughout the last few years, when i have talked to people,* one of the most strongly and frequently expressed reasons i’ve heard for not panicking is that it seems impossible that p-hacking is so rampant that even a phenomenon shown in 50 or 100 studies (e.g., ego depletion) could be a false positive. if a paradigm has been used over and over again, and dozens of papers have shown the effect, then it can’t all be a house of cards. the idea was that even though there were some problems, there was no reason to panic because of these conceptual replications. (also, meta-analysis.)
i wrote a blog post about this a while ago, pointing out the problem with this logic. i argued that if we test a few bricks in the wall (studies), and many of them turn out to be weak (fail to replicate), we can’t point to the rest of the wall as evidence that everything is ok. if even a small sample of bricks (studies) is tested and a high proportion turn out to be weak (fail to replicate), we need to consider the possibility that the wall is not sound.
the only way out of that conclusion is if we think the sample of bricks (studies) tested was biased (i.e., we only tried to replicate studies that had obvious signs of weakness), or that the process of testing the bricks (replicating studies) was flawed. i think the reproducibility project ruled out the first possibility by selecting a mostly-representative sample of studies published in our top journals. i think the RRR rules out the second possibility by having a very careful pre-registration process that involves both believers and skeptics and a multi-site replication effort. it’s a one-two punch. we can no longer be confident that an effect is real just because there are many studies showing the effect. or because there is a meta-analysis showing the effect. the existence of the wall (a large literature full of conceptual replications) can no longer be a source of comfort.
to me, the RRR is another exit door slamming shut, pushing us to the conclusion that p-hacking is rampant and can produce mountains of false positives. if we can no longer trust meta-analysis, or trust the existence of 100 studies in top journals showing similar effects, there is nowhere to retreat.
i can understand why we didn’t panic when the ‘false positive psychology’ paper came out. who knows how many of those p-hacking strategies people use, much less in what combination.
i can understand why we didn’t panic when the QRP paper came out. who knows whether people did these things just once or many times in their lives. and i think we can all agree that, whatever our beliefs about the prevalence of p-hacking are, we could design a survey that would produce responses consistent with those beliefs. which of the two QRP papers you think is more valid is predicted by your a priori beliefs about the prevalence of QRPs.**
i can even almost understand why not everyone panicked when the reproducibility project (RP:P) came out because, as we all teach our students, we never put much stock in single studies anyway.
ok, wait, no. i’m lying. i can’t understand why not everyone was worried about the RP:P results. i have resisted writing a blog post about the RP:P but i can’t hold it in anymore. here it is:
---
so, it seems to me that the only way to not be worried about what the RP:P results are telling us is to reject the idea that the replicability rate in the RP:P tells us something about the (inverse of the) false positive rate.
there is definitely noise and error (e.g., hidden moderators) that affect replication results, but there is still something we can learn from the RP:P, even if there's uncertainty around that conclusion. in the RP:P, 74.5% (41 out of 55) of social/personality studies failed to replicate by the “replication study reaches p < .05” standard.# let's assume that half of those failed replications were due to noise or error. ## that's still 20 out of 55 studies that failed to replicate, or 36%. so, i think it's reasonable to conclude that the RP:P suggests that a good estimate of the false positive rate in social/personality psych is 36-75%. that's a wide confidence interval, but it's still way more precise than anything we had before. and it's way way higher than we would like.
# if you prefer a different metric for deciding what counts as a failed replication, here is the same calculation applied to the most generous metric of replication success: if we use “meta-analytic p < .05” (which i think is far far too generous because it gives the non-pre-registered original study as much credibility as the pre-registered second study), 27 out of 55 studies successfully replicated. that’s 51% that didn’t replicate, even by this very generous standard. now let’s generously assume half of those failed replications were due to noise or error, and we’re still left with a 25% false positive rate as the lower bound of our estimate. best case scenario.
## note that i’m generously### assuming all noise/error leads to underestimating the effect. as sanjay has noted, noise and error (and hidden moderators) can lead to overestimating effects, too.
### so much generosity.
---
ok, back to the original blog post.
so, i can almost understand if people weren’t worried until now. but the RRR makes it very hard for me to imagine being ok with the state of the field.*** even if you look at your own practices and those of everyone you know, and you don't see much p-hacking going on, the evidence is becoming overwhelming that p-hacking is happening a lot. my guess is that the reason people can't reconcile that with the practices they see happening in their labs and their friends' labs is that we're not very good at recognizing p-hacking when it's happening, much less after the fact. we can't rely on our intuitions about p-hacking. we have to face the facts. and, in my view, the facts are starting to look pretty damning.
in conclusion, fifty million frenchmen can definitely be wrong. so can fifty JPSP papers.
(this situation reminds me a bit of the situation described in the big short, about the mortgage bubble - few economists and bankers thought it was possible that there was this huge crisis looming, partly because everyone thought that if there was this fundamental problem, it would be obvious to the experts.**** sometimes, things that seem super solid are actually way more shaky than almost anybody realizes.)
i know there are very smart people that i respect a lot who disagree with me. i very much welcome comments/responses.
ps: as gloomy as this post sounds, i am, still, optimistic. also stubborn.
* introvert gone wild.
** p < .000007
*** in case you were worried about running out of things to panic about when lying awake at 3 a.m.
**** i'm not sure why paramount hasn't called to acquire the rights to my blog. mark ruffalo already told me he'd play uri simonsohn.
meuh.
give the gift of data
beautiful things happen when people make their data publicly available.
from 1895 to 1903, one anonymous man kept track of his nocturnal emissions every day.* in 1904, he published an analysis of seasonal effects in his eight years of data. because few statistical techniques had been invented for analyzing these types of data, the author based his conclusions mainly on a visual examination of the data (see top panel of the figure below). he concluded that his nocturnal emissions were higher in the spring and summer than in fall and winter.
the author also did something else impressive: he shared the raw data with the world (in numerical form, not the actual biological specimens). in 2012, widaman and helm** decided to reanalyze the data with six different modern quantitative techniques. i won't get into the details of their analyses,*** but even just rescaling the y-axis and adding 95% confidence intervals to the monthly averages (bottom panel below****) shows just how weak the evidence was for the author's conclusion that there were seasonal patterns in the data. indeed, widaman and helm's re-analyses did not show much evidence of any monthly or seasonal patterns at all.
Figure 3 from Widaman & Helm (2012)the authors describe their results as anticlimactic,***** because their main conclusion is a null effect for seasonality. however, i would argue that the ability to reanalyze 100+ year-old data is quite titillating. it is a striking demonstration of the great strides we've made in quantitative methods in the last 110 years, and a vivid example of how data sharing enables self-correction in science.
sometimes, sharing your data can feel like giving away your most intimate possession, a piece of yourself. in anonymous' case, that was kind of true. but if you try it a couple times, you soon realize that, most of the time, sharing your data is not a form of self-exposure - nothing bad happens, no one is pointing and laughing. most of the time, no one is going to pay any attention at all. but if they do, chances are, something good will come out of it.
when it comes to data sharing, there's no such thing as TMI. anonymous was brave enough to do it back in 1904, with his most intimate data. i think it's time for the rest of us to be as open and transparent as anonymous.******
give science a present this holiday season, share your data.
* that's what i call within-person data.** full disclosure: helm is my boyfriend. in case it's not obvious, this paper played a critical role in our courtship.
*** because i don't understand them.
**** july 1897 was an exciting month for anonymous.
***** in case this makes you think it's not a serious paper, i'll have you know there is one equation in there with six different greek letters.
****** or, you know, maybe a tiny bit less open.
references
Anonymous (1904). Nocturnal emissions. American Journal of Psychology, 15, 104-107.
Widaman, K. F., & Helm, J. L. (2012). Nocturnal emissions: A failure to replicate. American Journal of Psychology, 125, 39-50.
most damning result
reviewer 2 is not buying it.
i've had a blog post stuck in my head* for a few months now, and the new post on datacolada.org is finally spurring me to write it.
i'll admit it. i'm reviewer 2. i'm the one who doesn't quite believe that you're telling the whole story, or that things are as simple and neat as you make them out to be. that's because we've all been taught to pretty things up (pp. 8-10). it sucks for authors that reviewers are super skeptical, but it's the price we are now paying for all those years of following bem's advice. there are going to be more and more reviewer 2s out there. i'm pretty sure most people have already run into the skeptical reviewer who suspects you had other measures, other data, or that you didn't plan those covariates a priori. if you haven't yet, it's just a matter of time.
there's nothing personal about it. the reason reviewer 2 thinks that you're telling the best possible story you could extract from your data is because why wouldn't you? that's the game we were taught to play, those were the rules until now. so when reviewer 2 asks you to disclose all flexibility in your data analyses, to share your data, to describe all your measures, etc., she isn't accusing you of anything other than having been trained in a pre-revolution era. it's not rude, it's just healthy skepticism.
when i read a paper, i assume the authors are showing the most beautiful result they could get, unless they give me a reason to think otherwise. the fantastic thing about pre-registration is that it's an almost foolproof way of convincing readers and reviewers that you're being honest about what was planned and what wasn't. if you can pre-register, that is definitely what you should do, for all the reasons described in the datacolada post and more.**
what if you can't pre-register? what if the data are already collected? or what if you pre-register one thing, but then you get a new idea about what you can do with the data after you collect them? in other words, when you want to publish something that was in fact exploratory, what's the best you can do to assuage reviewer 2 (and all the skeptical readers)?
first, acknowledge that any solution is second best to pre-registration. as someone who has never pre-registered anything and possibly never will,*** i think it's important to admit this. the fact that some research is really hard to pre-register doesn't mean that it wouldn't be better if it was pre-registered. just like even though i will probably continue to use college student participants, i don't pretend that my research wouldn't be better if i used a more diverse sample. i am perfectly comfortable acknowledging that a certain practice is ideal, and that i lack the resolve to do it. it's called hypocrisy and i'm sticking with it.
second, be transparent. if you want to convince readers that you're not selling them a touched-up version of your results, show them the flaws.**** tell the reader what other analyses you tried, what other studies you ran, how the results look with different decision rules about outliers and covariates, etc. in other words, write paper two in mickey's tale of two papers.
what if being totally transparent takes up too much space? one solution is to put some stuff in an appendix or supplementary materials. i'm not against that, but i don't think it's a good enough solution because most readers won't notice it, so it still ends up letting authors get away with pretending the touched-up version is the whole truth. so, i came up with an additional solution, a new subsection for every results section: your most damning result.
i am pretty enamored of this 'most damning result' idea. i would love to make it a standard section of every paper. i think it would help readers evaluate the strength of your evidence - if you can still convince readers that you have strong evidence for your conclusion, even after showing them the most damning result, that's pretty impressive. like getting a date after posting your best and worst selfie.
i think the "most damning result" section would also encourage us to be more skeptical of our own results. probably most of us have no problem writing off our most damning result as not informative because, in retrospect, of course we shouldn't have expected that manipulation to work/that item to make sense/that cover story to be convincing. but having to at least write up that result like any other might give us some pause. if we look at touched-up data too long, we start to forget what real data look like, and being forced to stare at the blemishes could do us some good.
bottom line:
if you can pre-register, pre-register.
if you can't, be transparent, and show us that you're telling the whole story. like by highlighting your ugliest, most damning result.
* i think it's responsible for my recurring ear infection.
** here's another: if you can get your pre-registered study to work with 25 people per condition,***** i won't harass you about your sample size.
*** i know, i'm a bad, bad person.
**** of course, if showing all the flaws makes your results inconclusive, it's a sign that you should probably go do some more research before writing anything up. transparency is not a silver bullet - if the data don't give a clear answer, transparency won't save you (and will, in fact, make it harder for you to publish your result. but that's a good thing. for science. not for you.)
***** good luck with that.
the nice reviewer. (don't try to pet him.)
Guest Post: A Tale of Two Papers
A Tale of Two Papers
By Michael Inzlicht
Change is afoot in psychology.
After years of bickering on social media and handwringing about whether our field is or is not in serious trouble, some consensus is emerging. Although we might not agree on the severity of our problems, almost no one doubts that our field needs improvement. And we’re now seeing the field take real steps toward that, with new editors stepping in with mandates for genuine and powerful change.
As an Associate Editor at a journal I’m very proud of, I have lived through some of this change. While the standards at the Journal of Experimental Psychology: General have always been high, the standards are more stringent now than when I began. Some people interpret this change as a turn toward conservatism, of valuing safe over creative work. While I appreciate this perspective, I disagree. Instead, I see this as a turn toward transparency, as a turn toward robustness. I still value creative research—and demand it of myself and of authors with whom I work, but now I also value transparency, which allows for robustness.
Transparency is critical, yet has been lacking in the past. Transparency not only offers an honest reflection of how hard science is, it allows for cumulative science, with others that follow being able to truly build on and extend past work, knowing precisely what steps to take and what pitfalls to avoid. It is amazing that for so long we have been writing papers backassward: We were taught to look at data first and then derive hypotheses after the fact, with our resulting papers seeming perfect for far too long. The result of all these “perfect” papers is that our field produces findings that cannot be replicated. If, however, our work and our data were transparent, if we revealed all their ugly flaws, we might actually be able to build on them and create a robust science.
To illustrate, let me tell you the story of two papers I edited. The first paper contained 7 experiments, often with outliers removed and covariates included, and reported effect sizes in the medium to large range all supporting the main hypothesis. The second paper contained 18 experiments, didn’t exclude anyone or add any covariates, and reported small effect sizes that sometimes were contrary to the hypothesis. The first paper found 7 out of 7 significant results; the second paper contained 2 significant effects out of 18.
These were not two papers. They were two versions of the same paper.
The first was emblematic of the old way of doing business, with 7 studies that were scrubbed clean to be near-perfect. The second is emblematic of the new ways we are trying to do business, with studies that were raw, unvarnished, and true.
This paper, authored by the very brave Mirjam Tuk, Kuangjie Zang, and Steven Sweldens titled, “The propagation of self-control: self-control in one domain simultaneously improves self-control in other domains” (volume 144, pages, 639-654) is now my favorite as editor. I say this not necessarily because of the topic area (which is fascinating in its own right and deserving of its own blogpost), but because it is a model of transparency and a template for the kinds of things we should be seeing more of in our top journals.
Despite high marks from reviewers enthralled with the topic, I decided to reject the first paper because of signs of non-robustness. One intrepid reviewer analyzed the paper using my friend Uli Schimmack’s Incredibility Index and found it to be excessively significant. Papers are excessively significant when they contain multiple studies and there is a discrepancy between the expected number of significant results (as determined by power) and the actual number of significant results. Excessive significance, or so-called incredible results, undermines the trustworthiness of reported results and intimates that something else might be going on.
In the case of the first paper, that something else was a rather large file drawer.
Not everyone considers file drawering studies to be a questionable research practice; however, almost everyone agrees that they are very problematic for the field. File drawering studies warps our sense of how real, robust, and large an effect is. Because researchers file drawer papers—be that because they choose to or because the review process leads them to—there is a growing sense that we cannot fully trust the published record. Even meta-analyses, seen as the best way of establishing the veracity of an effect, have become suspect because of things like the file drawer. Garbage in, garbage out, right? We are now witnessing a meta-analysis boon, with researchers re-doing old meta-analyses but adding sophisticated techniques to detect and correct for bias, chiefly the problems caused by file drawering.
So: file drawering is bad. And that first paper was guilty of doing it. But, almost everyone is guilty of doing it. I know I am.
After I rejected the first paper based on it appearing too good to be true, the authors came back to me saying it indeed was too good to be true, that they had a large file drawer. But here was the rub: when the authors included all studies, when they emptied the file drawer, the meta-analytic effect was robust, albeit small. Better still, the meta-analytic effect held when including all participants (some of whom legitimately could be dropped) and excluding all covariates.
I invited the authors to submit this second paper and after a few rounds of revision, the paper was accepted. Their meta-analytic effect, unlike almost all others in the field, is based on a body of data that is not warped by the file drawer. Clearly, we now know that all data warrant independent replication; but to me, these data are trustworthy. I mean, only two of the studies exceeded the magic p<.05 line and a couple of the studies were in the wrong direction altogether. Check out their forest plot. This is what real data look like. The data are not always pretty, they have warts, but they are real.
This paper is a model of transparency and leads to better studies on this topic in the future. Because the unvarnished effect size from this study is quite small at d=0.22, it means that future follow-up studies will need 652 participants in a 2-cell between subject design to achieve 80% power. Yes, this is a huge number. But, you know: research is hard.
If we want a robust science, we need to run high-powered studies. But the only way to correctly calculate power is with a realistic estimate of the underlying effect size. This second paper allows for this, even if the usual caveats of independent replication apply. Power is more than just sample size, however, and some of you might be relieved to hear that 80% power can be achieved with 166 participants in a simple pre-post repeated measured design[1].
I am a huge fan of this second paper. I love all my children, but I would be lying if I said that this wasn’t my favorite as editor. I love it because it is transparent; and because it is transparent, it allows for a robust science. This push for transparency, of revealing our warts, is exactly what the field needs.
Many of the new standards emerging from our field are means to ensure transparency. Scientists can still pursue creative, even bold ideas, but should do so while being more sincere, more circumspect about the fruits of their effort, even when they reveal imperfections.
Science is hard. While it can sometimes reveal true beauty, it can only do so when we allow it to be ugly too.
[1] This assumes a .5 correlation between pre and post measures. Sample size will go down (up) with higher (lower) correlations between pre-post measures.
super power
lazy dog.hi there. i'm here to lecture you about power again. it's what i do for fun.collecting data is hard. large samples take time, and resources. i am sympathetic to the view that it's sometimes ok to have small samples.but if you're doing a typical social/personality lab experiment, or a correlational study using questionnaire measures, then it's probably not ok. for those types of studies, adequate power should be a basic requirement for publishing your work in a good journal.*when i hear people push back against the call for larger samples because they are sticking up for people who use hard-to-collect data, i scratch my head. those people are exactly why i think we absolutely need to increase the sample size of typical social/personality studies. if some of our colleagues are busting their asses measuring cortisol four times a day for weeks, or coding couples' behavior as they discuss marital problems, and even they can get samples of 100 or 200, then the least the rest of us can do is get a couple hundred undergrads to come to our labs for an hour.when i see a simple lab/questionnaire study that has a smaller sample size than many super-hard-to-collect studies, it makes me sad.today, i want to introduce you to two of my favorite researchers who do some of the hardest research i know of. i picked them because they study super important questions with incredibly rigorous methods. they are the people i think of when someone with a simple study is claiming that it would just be too hard to collect 80 more undergrad participants. they agreed to let me write about them,** but all of the opinions expressed in this post are mine alone.case study onekristina olson studies gender nonconformity and transgender children's development (among other things). this research is almost guaranteed to lead to important insights about the development of identity, gender stereotypes, and of course the understudied population of transgender and gender nonconforming individuals. but recruiting a sample of transgender children, and following them over time, is hard.***here's what they do. when a family signs up for the study and meets the inclusion criteria, they make plans for a few members of the research team to drive or fly out to where the family lives, go to their home, and conduct several hours of tasks with various family members. for every. single. family.their first paper (Olson, Key, & Eaton, 2015) included a sample of 32 socially transitioned transgender kids, 18 cisgender siblings, and 32 matched controls. that's a total of sample size of 82 participants. this dataset took one year to collect, including four flights and two roadtrips. and their sample size is LARGER than the median sample size for social/personality research published in psych science (Fraley & Vazire, 2014).but they didn't stop there. their team is currently expanding this dataset, aiming for 200+ socially transitioned transgender kids, some gender nonconforming kids, a matched control for each kid, and a sibling control when possible. their total sample size will likely be around 750, and they're well on their way. they have families from 22 different states, and their waitlist includes families from 17 more states.wait, it gets better. in addition to recruiting this incredible sample, they plan to follow them longitudinally, assess them in person every 2-3 years, and conduct questionnaire and interview measures more regularly. also they will cure world hunger. (that last part was a joke. it kind of all sounds unbelievable to me, so i just wanted to make sure you knew.)one interesting moral of this story is that the psych science paper with a meager sample size of 82 is what made the larger, 700+ person study possible. it sparked interest in the topic, and gave olson a stronger foundation for recruiting new participants, securing funding, and investing more of her own time and effort into this project. this is a perfect example of when publishing a small sample study is obviously a good call.case study twojennifer tackett studies environmental and dispositional factors that predict aggression and self-control/disinhibition (among other things). her research focuses on childhood and adolescence, and uncovers important biological and environmental processes that influence the development of personality traits and personality disorders. her research also involves recruiting families and following them longitudinally, and measuring everything you can imagine about them.one of her recent papers (Tackett, Herzhoff, Kushner, & Rule, in press) came from a dataset including over 300 families that were assessed four times over four years. two of these assessments involved a two and half hour in-lab assessment of the child, mom, and dad, and included questionnaire measures, videotaping the kids in 15 different situations and coding their behavior, clinical interviews (also coded and transcribed), and genotyping (plus some other measures).pretty damn impressive.but then tackett decided that wasn't good enough, and decided to do the study again, but with a more ethnically and economically diverse sample, more behavioral tasks (including a parent-child interaction), and more psychophysiological measures. each session lasts three and a half to four hours in the laboratory, which means that the families get a meal, transportation to and from the lab, and babysitting for siblings. and she's collected data from 350 families in this second study (that's over 700 families in the two studies combined).just thinking about running that study exhausts me. thinking about trying to recruit a community sample like that makes me want to curl up in a ball. can you imagine recruiting 350 families from a broad range of backgrounds and convincing them to come to your lab for 3.5 hours to do a bunch of tasks and spit in some vials? now imagine arranging meals, parking/transportation, and babysitting for all of them. now imagine trying to recruit and manage a team of insanely dedicated grad students and RAs who are willing to spend their weekends running the study. without paid staff. good news: you don't have to, because while we are watching netflix and eating peanut butter out of a jar, jennifer tackett is doing it for us.conclusionwhen i talked to both olson and tackett about this blog post, both insisted that they did not want to be held up as exemplars, that they felt that their research had important limitations. so let me be clear, this conclusion i am about to draw is mine and definitely, absolutely, positively not theirs:the rest of us suck.i'm not arguing all of us should do this. this cannot be the new standard. we're not all superhuman. these researchers (and their formidable grad students and RAs) are and always will be outliers, and they should get medals and many many grants and maybe you should consider their labs when doing your estate planning. but my point is if they can do this, most of us can run 80-100 people per condition in our simple studies with undergrads.researchers like tackett and olson could use the fact that their data are super hard to collect as an excuse not to collect large samples. but they don't. so the rest of us should suck it up and do the right thing. next time you think about stopping data collection at 35 people per condition because it's too hard to get twice that many, just be glad you don't have to fly to bakersfield**** to get your next participant. then collect more data.****** and if you're doing an mTurk study, "adequate power" = at least 95% (why take a 1 in 5, or even 1 in 10, chance of a false negative if it costs you $18 to slash your type II error rate? is your research hypothesis worth less to you than a bottle of napa valley chardonnay?******)** 'agreed' = tried to convince me that they weren't the best role models but didn't threaten to sue me if i went ahead.*** kind of like the ironman is hard. but harder.**** sorry bakersfield.***** also, sequential analysis, within-person studies, bayes, openness, truth, utopia. did i miss anything?****** while i'm hating on california, i hate oaky chardonnays. also, i just heard on the radio that the utility companies want us to know that the dirt-like odor and flavor of our tap water is a purely aesthetic issue, and will continue as long as the drought does. that oaky chardonnay is starting to sound pretty tasty now, actually. (who am i kidding, i love california. i would drink dirt for california any day of the week. but not oaky chardonnay.)
does psychology have a public relations problem?*
near holland, mi
some people worry that having a loud and public debate about the reproducibility of psychology findings may be detrimental to our public image. in this blog post, i make the bold argument that not only is this not what will happen, but if we have a public relations problem it's the opposite: we sometimes come across as too naive, not skeptical enough of our own preliminary results.why do i believe the replicability discussion is not going to cause harm to our reputation?i'm not generally known for my deep respect for the average person, but i do think people understand the basic concept of science - that we are getting closer and closer to the truth, but that all current knowledge is incomplete and subject to revision. in his essay The Relativity of Wrong, Asimov makes the point that science is all about becoming less and less wrong. undergoing the kind of critical self-examination psychology is currently in the midst of is a normal part of science.[for a fascinating example, see kuo's march 2014 discovery, at a 'five sigma' level of confidence (i.e., p < .0000003), that the universe expanded rapidly immediately after the big bang ('chaotic inflation theory'). when the discovery was made, the guy who had come up with the theory thirty years before (andrei linde) said "If this is true, ..." which seemed super modest to me at the time. fast forward to january 2015, and it turns out the evidence they thought they had was wrong. welcome to science. these errors don't mean we aren't hurtling towards the truth, they mean that the path is not a perfect, smooth, straight line.]i think we are successfully communicating to the world that we are having this discussion because we care about truth. the very fact that we are debating these issues, and not sweeping them under the rug, shows that we value getting it right more than portraying a glossy, airbrushed version of our science to the public.it's not news to anyone that we are a young science, and that we tackle extremely noisy phenomena. if we tried to pretend that our theories are complete, that we can predict complex human behavior with high precision, that our methods are perfect, no one would believe us anyway. if we show that we are getting better and better, and willing to revise our theories and methods when we get new, better ones, people will take us more seriously, not less.what do i think is the bigger threat to our public image?in my view, the bigger threats to people taking us seriously are the exaggerated headlines and conclusions often appearing in the media. when a preliminary study suggests an association between two variables, and that gets presented in the media as a definite and deterministic relationship, we look bad. to some extent, this is not our fault. we can be careful in our journal articles, and even in our conversations with the media, and they can still blow our claims out of proportion. but we do have some responsibility to try to correct this.first, we can be more careful about what we do write in our articles. a laundry list of caveats in the limitations section does not give us free reign to make wild speculations everywhere else in the paper. speculations have their place, but they should be clearly marked as such. and it's our responsibility to put the important limitations of our research front and center. (also, if you're tempted to say that there is 'a gene for X' or that 'brain region X lights up', read this.) (thanks sanjay.)second, we can resist the hype that reporters try to put in our mouths. some of my conversations with journalists go something like this:them: "so you're saying that people don't know themselves at all?"me: "no"them: "but people are really really bad at knowing themselves?"
me: "no"them: "but wouldn't you say that, in general, people are astonishingly bad at knowing themselves?"me: "no"**third, we can do a better job of helping science writers and readers evaluate psychological science. to that end, here are some tips for science writers and readers of science writing. when a study claims that doing X will make you smarter/happier/more popular, ask these questions:1. have there been multiple studies, ideally by different researchers, all showing the same thing?2. did the studies have a lot of participants (for social/personality psychology, at least 200***)?3. are the researchers actually studying the phenomenon they are drawing conclusions about? if not, is the phenomenon they are studying at least similar to the one they are attempting to draw conclusions about?4. if the researchers are drawing conclusions about what causes what, did they at least conduct an experiment or look at changes over time?****5. if the study is an experiment, can you rule out all likely confounds? are you pretty sure that nothing besides the experimental manipulation could have caused the differences between the control and experimental conditions?6. did the researchers use appropriate measures to quantify the variables they are studying? are they better than any other (feasible) measures you can think of? (e.g., if the researchers used self-reports, do you trust that people can report on this honestly and accurately?)7. is the statistical evidence strong? if the researchers used p-values, are they well below .05 (say, smaller than .01)?if you answered no to any of these questions, you may be reading about a preliminary study. it's totally fine to get excited about it, and if you're a science journalist, you can even write about it. just think about it (and write about it) as a thought-provoking possibility that needs to be followed-up on, not a fact.indeed, even if the answers to all the above questions are yes, it's a good idea to remind yourself (and your readers) that all psychological research findings are probabilistic - at best they tell us that variable X tends to be associated with variable Y (or, tends to cause an increase/decrease in Y), but there are many exceptions, and many other things that influence variable Y. also, remember chaotic inflation theory.knowledge ain't cheap. it should be hard, and slow, to accumulate enough evidence for an effect that we should be confident we understand what's going on, especially when we're talking about human beings interacting with each other. when the media coverage makes it look like we're discovering incredible, counterintuitive new things that deterministically predict important outcomes every other week, the public starts to think we're either dumb, or we think they're dumb. if we still don't know for sure, after years of nutrition research, whether coffee is good for you or not, how could we know for sure after one study with 45 college students whether reading about X, thinking about Y, or watching Z is going to improve your social relationships, motivation, or happiness?i don't mean to put all the blame on the media. it's partly on us to insist that journalists convey the uncertainty in our results. it's less sexy, but i suspect the public will find the more circumscribed conclusions more palatable, and will take us more seriously if they believe that we are well-calibrated. we sound more scientific, not less, when we say "this is our best guess so far, but new evidence could suggest otherwise".also, there are many good science journalists out there, conveying the messiness***** and noisiness in our research in a way that still captures the public's interest in, and respect for, what we do. let's talk to them, and hang up on the ones that want to make us look like hucksters.** there are any number of****** popular science articles about self-knowledge that were not written because of my stubbornness. i think the world will find a way to live without them.*** more or less arbitrary. but gives researchers 80% power to detect effects of r = .20 or greater .**** i don't really want to get into all the subtleties of causal reasoning (mostly because i don't think i understand them well enough). how about a massive oversimplification instead? yes? ok: in social/personality research, i believe even experiments and longitudinal designs typically don't give us super conclusive evidence about what actually causes what in the world. in the best case, they tell us that X is one of the (probably many) things that can have a causal effect on Y. probably best to keep causal claims tentative.***** ok, this article still glosses over a lot of important issues (especially causality), but it's better than most!****** at LEAST two.
cosmic microwave background radiation (photo credit: nasa)
submit your papers to SPPS!
remember how i like to tell journal editors how to do their job? well.....
i am super excited to be starting my stint as editor in chief of Social Psychological and Personality Science, a journal that belongs to a consortium of four organizations (SPSP, SESP, EASP, and ARP), is co-sponsored by two more (AASP and SASP), and is published by SAGE.
as always, my blog posts reflect my own views and not those of SPPS, SAGE, or any other organization.
i'm excited to take on this role for several reasons. first, Allen McConnell, and before him Vincent Yzerbyt, built up this new journal into one of the top outlets for short reports in social/personality psychology. the journal now receives almost 600 submissions in a typical year, publishes eight volumes per year, has a circulation of over 7,700, and has its first impact factor (2.56*). not bad for a six-year-old journal.
second, i kind of love this journal. i've been an associate editor there for a few years, and have learned so much from reading the papers and reviews that have crossed my desk. it's amazing how much cool research can be packed into a 5,000 word article. it seems like people are sending some of their best work to us. i would like you to continue this excellent practice please. thank you.
third, i have an awesome team of associate editors:
Wiebke Bleidorn, UC Davis, USA
Lorne Campbell, University of Western Ontario, Canada
Jesse Graham, University of Southern California, USA
Dominique Muller, University Grenoble Alpes, France
Nickola C. Overall, University of Auckland, New Zealand
Kate A. Ratliff, University of Florida, USA
Joseph P. Simmons, University of Pennsylvania, USA
Gerben A. van Kleef, University of Amsterdam, The Netherlands
Gregory D. Webster, University of Florida, USA
Tessa V. West, New York University, USAfourth, it's an exciting time to be taking the helm. lots of ideas about how to improve the scientific and peer-review process are floating around. each journal is taking its own approach to navigating this fast-changing landscape. here are some things we are doing at SPPS:
some other stats about SPPS that might be of interest:
- Upon submission, authors will be asked to confirm that they reported the following in the text of their manuscript:
- How sample size was determined for each study, and a discussion of statistical power.
- All data exclusions (e.g., dropped outliers) and how decisions about data exclusions were made.
- All measures or conditions for variables of interest to the research question(s), whether they were included in the analyses or not.
- Exact p-values, effect sizes, and 95% confidence intervals for all key results, or an explanation of why this is not possible.
- Tables, table notes, figures, and figure captions will not count towards the 5,000 word limit.
- Tables and Figures should be embedded in the manuscript upon submission, not appended at the end of the manuscript.
- The name of the handling editor (or ‘action editor’) will be printed at the end of each article.
- SPPS now accepts submissions of replication studies.
i would love to hear your thoughts about these changes, and the journal in general. and i sincerely hope you'll consider submitting your work here. all your favorite researchers are doing it. (actually, that's kind of true. check out this month's table of contents. or last month's. or the month before that.)
- Acceptance rate: about 15-20%
- Average time from submission to decision: 39 days (47 if we exclude desk rejections)
- Average time from acceptance to online publication: about 20 days
- Average time from acceptance to print publication: about 4 months
finally, i hope you all appreciate the increased use of capital letters in this post.**
* i know i'm not supposed to care. i wish that i didn't care. but i do care.
** don't get used to it.
yeah.
why p = .048 should be rare (and why this feels counterintuitive)
sometimes i read a paper with three studies, and the key results have p-values of .04, .03 and .045. and i feel like a jerk for not believing the results. sometimes i am skeptical even when i see a single p-value of .04.* is that fair?
mickey inzlicht asked a similar question on twitter a few weeks ago, and daniel lakens wrote an excellent blog post in response. i just sat on my ass. this is the highly-non-technical result of all that ass-sitting.
we are used to thinking about the null distribution. and for good reason. the rationale behind Null Hypothesis Significance Testing (NHST) is that we are testing the probability that our observed effect actually came from the distribution of effects you would expect if the true effect is zero (null). so NHST starts with the assumption that we are living in the world where the real effect is zero, and the expected distribution of results when we do studies looks something like this:**
when we do a study and we get an effect, we want to know whether that effect (e.g., a difference between means, or a correlation) is unlikely to come from this distribution, i.e., if it is far enough in the tail of the distribution that the probability that it came from this distribution is low (less than 5%, to be precise). so the 'goal' is to get an extreme effect.
if we think about it this way, it makes sense that it is hard, or rare, to get an effect in the tail of the distribution, and that p-values just below .05 should be common.
but of course, we don't believe that our effect actually comes from this distribution. our aim is to reject that null hypothesis. we actually believe that our effect comes from another distribution, one around a non-zero true effect. let's call this distribution the H1 distribution.
if our observed effect really does come from the H1 distribution, we should often observe effects that are far out in the tail of the null distribution, not barely past the p < .05 threshold. if we're actually sampling from H1, we should get lots of effects more extreme than the p < .05 threshold.
this is more true to the extent that:
-the true effect size (around which H1 is centered) is far from zero
-each distribution is narrow (i.e., the study has high power/precision, i.e., has a large sample (for between-subjects designs), low error, etc.)so p-values close to .05 should be rare. p-values closer to 0 should be more common. that’s the basic idea behind the p-curve.
how much more common? according to simonsohn, nelson, & simmons (2014), if a study has 46% power (probably pretty realistic, maybe a tad optimistic), the ratio of p-values below .01 to p-values between .04 and .05 should still be 6:1. daniel lakens’s blog post also provides some useful stats about the probabilities of various combinations of p-values. basically, p-values close to .05 should be rare.
so why aren’t they?
maybe we’re really good at power planning! maybe we know exactly what size our effect will be! and we can run exactly enough subjects to get our p-value just below .05!
if you believe that, you haven’t been reading my blog.
why is that unlikely? first, if we knew the exact size of our effect, we wouldn’t need to keep studying it. we don’t know. second, even if you did know the exact size of the true effect, why would you power your study to be only just barely significant? maybe you like to live dangerously, but i prefer to sleep at night.
ok, so hopefully i’ve convinced you that p-values close to .05 should be rare, and that perfect power planning can’t be the reason we see them all the time. so why do we see them all the time?
it is tempting to look for more complicated answers, but i think the answer is simple: p-hacking. and file drawers.
what should we do with this information? first, it’s ok to be skeptical of a paper with a couple studies with high p-values. it’s even ok to be skeptical of a single study with a high p-value. that doesn’t make you a jerk.***
but more importantly, p-values close to .05 can be a signal to ourselves that we may be unintentionally p-hacking. see mickey's blog post.
ps: i know that if i only became a bayesian, none of this would matter and i would never feel lonely and my dog would never barf in my office during office hours. but by now it should be abundantly clear that my role in this whole thing is to think like someone who does not have advanced stats training. (it turns that this is pretty easy to do if you do, in fact, lack advanced stats training.) also, the dog barfing in office hours has its pluses.
* sometimes just the letter p sets me off.
** yes, i instagrammed my graph. what's your point?
*** it doesn’t make you not a jerk. only extensive personality testing by a trained expert can tell you that. i offer discounts for friends and family.
Guest Post: Excuses for Data Peeking
Excuses for Data Peeking
Guest Post by Don Moore and Liz Tenney
Good research practice says you pre-specify your sample size and you wait until the data are in before you analyze them. Is it ever okay to peek early? This question stimulated one of the more interesting discussions that we (Liz Tenney and Don Moore) had in our lab group at Berkeley. Here’s the way the debate unfolded.
Don staked out what he thought was the methodological high ground and argued that we shouldn’t peek. He pointed out the perils of peeking:
If the early results look encouraging then you might be tempted to stop early and declare victory. However, we all know why this is a mortal sin: selecting your sample size conditional on obtaining the hypothesized result increases the chance of a false positive result. However, if you don’t stop but the effect weakens in the full sample, you will be haunted by the counterfactual that you might have stopped. The part of you that might consider using part of the sample won’t be able to help wondering about some hidden moderator. Maybe the students were more stressed out and distracted as we approached the end of the semester? Maybe the waning sunlight affected participants’ circadian rhythms? So peeking tempts you to sin and runs the risk of leaving you full of regret.
Liz agreed about these risks but pointed out that if you peek at the data very early, with n = 10 or n = 20 per cell, you could get information from peeking and would not be tempted to stop the study there with such a small sample, because, thank goodness, bigger sample sizes are becoming the norm.* If you peek and the results look good directionally, you should keep going to run the full, planned sample.
Alternatively, it is possible that the data fool you. If the results look bad when you peek, but your hypothesis is actually true, it might be vindicated in the full sample. This is the tragedy of a false negative. If you give up on the hypothesis too early, you’ll never know if it might have been supported because you didn’t give it a fighting chance.
But then Liz made an excellent point: If your study is going to fail, then it makes sense to abort it. Let’s say you have the hypothesis that Utahns eat more kale than do Californians. So you’re rooting for the Utahns as you start your big study with a planned sample size of 1000, composed of 500 Utahns and 500 Californians. If you find a difference going in the opposite direction or if you find no difference (a null result), the results aren’t that interesting (because they wouldn’t be surprising to anyone) and aren’t going to be publishable.**
If you peek at the data early and see that the Utahns are eating less kale (i.e., a result in the opposite direction from that hypothesized), wouldn’t it be smart to scrap the study? After all, how likely is it that the Utahns wind up eating more kale in the full sample if they start out with a trend in the opposite direction after collecting 10 or 20 in each cell? This question formed the crux of our debate. The answer depends, of course, on whether your hypothesis is true and how big the effect size actually is.
The figure below*** shows the probability of obtaining a nominal difference in means between conditions that goes in the OPPOSITE direction from the true population difference, given effects of various sizes (from d = .1 to 1) and samples of different sizes (from 5 to 50 per cell).
The probability of obtaining results that go in the opposite direction of the true population difference go to zero as your sample size goes up and your effect size gets larger. If the true effect size is d = .1 and you peek after collecting 10 per cell, the probability of being fooled is around 40%. That seemed unacceptably high to Don, so he thought he had made his point: don’t peek, you probably shouldn’t abort the study early, even if you get a trend in the wrong direction. But Liz looked at the same set of numbers and saw that if d = .5 and you peek after collecting 20 per cell, the probability of being fooled is only around 5%. She saw that as acceptably low, and she thought she had made her point: peek because if the results are in the opposite direction, you can abort early. Liz claimed that with an expected medium effect of d = .5 in a two-cell design, powered at 80%, alpha = .05, two-tailed, you would only need to collect about 30% of your sample (40 participants out of 128) before you had a pretty good idea of whether you should jump ship.**** Even with a smaller d = .3, under the same study parameters, you’d only need to collect 23% of your total sample (80 participants out of 352) to have less than 10% chance of being fooled.
But the truth, as both Liz and Don acknowledge, is actually a bit more complicated. How bad is it to waste the resources on collecting the full sample, only to have to throw it all away at the end? If the study is expensive to run, if the hypothesis isn’t all that exciting to begin with*****, and the opportunity costs are high****** then aborting early and giving up wouldn’t be so terrible. To pick another example, if in your clinical trial your participants are getting sicker instead of getting better, you should end the study early. If in your MTURK sample your participants are reporting slightly lower values on a Likert scale, and the cost to continue is low, you might want to see the study through.
However, there are good reasons you might want to stick with it. If there are other studies with more scientific promise that you could be investing your resources in, you would probably be working on them first anyway. And truth be told, many of us find ourselves doing research out on the thin ice at the edge of our scientific understanding where the effect sizes are small. A Cohen’s d of .3 or smaller is probably pretty common for the effects we’re looking at. If those are both true then the risk of peeking is greater.
Liz and Don both declared victory.
*Unless through your peeking you discover you made a computer coding error, or you have worded things in a confusing way, or you catch some other major issue. Catching mistakes is a benefit of early peeking.
**Only publishing results that support your hypothesis is a bigger, also important issue, but one we won’t tackle here. A premise of our debate was that in this particular case, the results would only see the light of day if they came out a certain way.
***Thanks to Ryan Goh for helping us crunch numbers that went into this graph. The simulations were based off of male and female height means and standard deviations obtained from the US Census Bureau website.
****There is a danger here that a researcher would be tempted to abandon the study only partly—by tweaking the materials and then starting data collection over. We would caution that this approach could increase the chance of a false positive. By jumping ship, we really mean giving up on the study completely.
*****Even a positive result wouldn’t be good enough to stand a chance at getting published in the Global Journal of Psychology and Counseling, no matter how high a publication fee you paid.
******You have so many good ideas. Some of the other ones might actually get published in journals that people read.
both ways is the only way i want it*
it’s halftime in america. or at least it’s halftime in game five of the clippers-rockets series. although i had planned to root for the clippers, mostly because they managed to eliminate the spurs and the holder of the best nickname ever,** i am finding it hard to resist james harden’s mohawk/beard combo. clippers or rockets? so hard to choose.
this is a common problem for me. i like to have it both ways. i am a personality psychologist and a social psychologist. i’m an atheist who is baptized and had my first communion, but if anyone in iran asks, i’m muslim. i think eating meat is mostly wrong but i just ate a bbq cheeseburger with pepperjack cheese. i am vast. i contain multitudes. (and now also a bbq cheeseburger).
so i’m here to plead for two things that might, on the face of it, seem contradictory: more civility and more criticism.
more and more of our scientific dialogue is happening in informal settings such as blogs and social media. this has many upsides. but as eli finkel pointed out on the facebook ISCON page, it's important to think about the norms for these interactions.
more civilityin our formal outlets (i.e., journals), i think our field is unique in its extremely civil tone, even in critiques, failures to replicate, etc., and also in the tone of our reviews. from what i gather from people in other fields, we are a really respectful bunch. i like that about us.
the informal outlets are more of a mixed bag. i see a lot of very dispassionate, sober analysis. but i also see some snarkiness, especially on twitter. my twitter activity has mostly dwindled to werner twertzog retweets because the intellectual discussions feel a bit like a dodgeball drill. maybe that’s fine – we all need a place to let off steam. but if we want social media to be a place for serious intellectual discussion, we have to treat each other as competent and honest actors. if we don’t have that foundation, why bother debating?
i hesitate to even bring up the civility issue, because i worry that it is my own anxiety that makes me scared of participating in these more heated discussions. but i am using my expert interpersonal perception skills to infer a pattern in who does and does not participate,*** and i don’t think i’m alone. if we want to draw more people into the conversation, civility is key.
more criticism
by civility, i don't mean agreement. the best intellectual debates are between people who vehemently disagree, but have great respect for each other. we need to separate politeness from agreement. in some domains of life it might be impolite for me not to believe you, but in science skepticism is vital - it is not a sign of disrespect. we need to stop being so sensitive about critiquing each other’s research. it should not take guts to question a phenomenon, to say you think a theory or hypothesis is wrong, to critique the methods or conclusions drawn by your colleagues. disagreement should be commonplace. we all read things we don’t believe. i sometimes read six impossible things before breakfast.****
and yet i find myself biting my tongue because of how criticism is perceived in our field. saying you don’t believe a finding, or a theory, is often seen as rude. that needs to change. we need to be able to debate, disagree, critique, and then go to happy hour like civilized people. i might be totally fooling myself, but if you wanted to buy me a beer and trash one of my papers, i think i would be ok with that.******
conclusion
we need to draw a sharp distinction between scientific criticism and the more personal kind. i think we all know where that line is, because we observe it very well in professional contexts. we know how to disagree without disrespecting. it's practically our job description.
we also know that being skeptical is not impolite - it is the heart and soul of the scientific process. we teach it to our students and we practice it all the time. but for some reason once a paper is in print we act like it's blasphemy to be critical of it. let's get over that. no finding should be sacred.
update
oh my god the clippers are dying. i hope big baby doesn’t start crying.
werner herzog
* title shamelessly lifted from maile meloy’s excellent short story collection. which i read all of. and i don’t even like short stories. i like my stories long.** what’s a girl gotta do to get called the big fundamental? (also, tim duncan has a social psychology publication with mark leary. true fact.) (also, i’m partial to doc rivers, but i do think the whole father-son combo is kind of weird.)
*** i'm a professional. don't try this at home.
**** when you live on the west coast, there is a lot of stuff to catch up on when you wake up in the morning. like the psych science table of contents.*****
***** sorry psych science. you don’t deserve that, not any more than many other journals. plus you've done a lot of good things. please don’t reject the manuscript that we just submitted to your very fine journal.
****** especially if it’s a west coast IPA.
Guest Post: Check Yourself before you Wreck Yourself
Check yourself before you wreck yourself
by Michael Inzlicht
Things have gone sideways in social psychology.
And, I am not sure this is necessarily a recent trend. The rot in my chosen field might have been festering for a very long time. I say this because not a day goes by when I do not hear about one or another of our cherished findings falling under disrepute. You have all heard about the depressingly high number of failed replications, be they one-offs or large scale coordinated lab attempts. You have also, no doubt, heard about problems with publication bias, low statistical power, and the widespread use of questionable research practices. Taken individually, none of these might be terribly upsetting; together they are an unmitigated disaster.
Perhaps more upsetting than the real problems facing my beloved social psychology is the level of denial I see. I have heard suggestions that things aren’t so bad. “Yes, there are a few effects that are probably not real,” say the defenders, “but we have made real and extraordinary breakthroughs and most of the stuff we study is rock solid”. I have also heard that every twenty to thirty years our field goes through its routine hand-wringing, but that it always passes. Many people are waiting for our current hand-wringing to pass. Finally, I have heard whispers and intimations that all of this is a way for personality psychologists, who are some of our most vocal critics, to get even with social psychologists, who seemed (for a time, at least) to have won the person versus situation debate.
This denial needs to stop.
This denial stops us from reflecting on our own work and asking what we have done to contribute to the problem. It stops us from trying to make things truly better. Now is the time to stop making excuses and to take a long look in the mirror.
Starting with me.
Have I contributed to our current crisis? It is painful to wonder if my research is part of the problem, but such introspection is vital to recognizing our troubles and, critically, rectifying our field.
Despite how painful it has been to watch my troubled field, I have been listening to our critics; I have been open to change. After years of hearing critiques of the field by my friend, colleague, and pub-adversary, Uli Schimmack, I have made active attempts to get better. For example, I have tried to collect larger samples; I have tried to use more statistically powerful designs (e.g., within-subject and repeated-measures); and I have tried to look at neighboring fields to get a richer understanding of my phenomena of interest. By not denying our troubles, I have changed how I conduct research.
So: Is there some way to quantify the quality of my work? Is there some way to see if changes to my research practice made a difference?
Looking to my H-index or number of citations is pointless, as these might say more about the popularity of my work and less about the quality of my science. Instead, it might be better to look toward metrics that might say something about the presence of bias and that can ascertain if my work has informational value.
Recent years have seen a few such metrics emerge. The most well-known of these is the p-curve, developed by Uri Simonsohn, Leif Nelson, and Joe Simmons, which can assess the evidential value of a number of findings. There is also the Replicability-Index (R-Index), developed by Uli Schimmack, which tracks the replicability of findings, with higher values suggesting a higher probability that replications will be successful. Uli has developed a second metric, the Test for Insufficient Variance (TIVA), which can also be used to detect the use of questionable research practices.
More than a little afraid of what I would discover, I analyzed my research output using all three of the above metrics. I decided to separately examine my first ten empirical papers and then my last ten empirical papers to see if my research output has increased in scientific quality. I suspected that it would. So, what did my analyses reveal?
First, a quick word or two on how I conducted my analyses. I decided to examine the one statistic per study that corresponded to the main hypothesis of interest. This was typically an interaction effect, but not always. If the main hypothesis corresponded to a crossover interaction, I would typically examine the two simple effects. Now because each typical study contains multiple statistics, often testing various parts of the main hypothesis, it was not always easy to pick which statistic to include. I typically selected the first relevant statistic to appear in the paper and I tried my best to be unbiased; but I would not be surprised if different people came up with somewhat different results. Finally, my life was made very easy by using Felix Schönbrodt’s amazing online p-checker app. I cannot recommend this app enough, and see it as essential tool for reviewers and editors wanting to improve the quality of our science.
So how did I do?
The statistics for my first ten papers are a mixed bag, but mostly unwelcome (see Table above). You can also recreate the analyses here. First, the good news: My p-curve analysis revealed no evidence of intense p-hacking and strong evidence that my first ten papers contain evidential value. This is where the good news ends, however.
While a positive p-curve result indicates that there is informational value to my studies, this does not necessarily mean they are devoid of bias. One’s work can have evidential value, can be informative, but still contain biased results. Do the results of my first ten papers contain bias? Sadly: yes. The results of the TIVA indicate that my results are not variable enough, suggesting that questionable research practices might have influenced things. While many of these practices are the norm in psychology, we now know that they are suspect and can lead to faulty conclusions. The R-index for my first 10 papers is also worryingly low, clocking in just under 40%[1]. Although there are no strict cutoffs, an R-index less than 50% raises doubts about the empirical support for the underlying conclusions in these papers. Unfortunately, my R-index is low and suggests that some of my results might not replicate. Unfortunately, I am in good company here. Uli Schimmack has analyzed a representative set of findings in social psychology and found that they achieved an R-index of 43%. He also reanalyzed a set of psychological studies appearing in Science, finding that they achieved a painfully low R-index of 33.9%. These low values raise concerns about the empirical support for the underlying articles
Looking more closely at my first ten papers reveals that the state of my (initial) science leaves much to desire. My median sample size was 54 participants, and that is not counting the number of participants that remained after excluding the occasional outlier. The most common design in my first ten papers was a 2x2 between-subject design, which means that on average I had a measly 13 to 14 participants per cell. I’m embarrassed to admit it, but in one study I collected only 28 participants in a two-cell between-subject design…but get this: it also included an individual difference variable as a moderator. Ouff!
Two of the three analyses revealed that questionable research practices might have influenced my initial results. These research practices are the norm and all too common, yet we now know how damaging they can be to our cumulative science. We now know, for example, that too much flexibility in data collection, too many experimenter degrees of freedom, allows practically anything to appear significant, even when it is not. If I look back at my early days and try to honestly assess where I went wrong, I cannot come up with any easy answers. I was slavishly focused on statistical significance and probably ran the fewest number of participants that I could. Perhaps, I too selectively reported the results confirming my hypotheses, overlooking those that didn’t. I most definitely ran underpowered studies. But to be honest, I simply do not know the precise reason that these analyses came out as they did. In that sense, I wonder if my bias was unconscious, with me fooling me.
I can do better. And it is clear from my most recent papers that once I stopped denying our problems, I have in fact done better.
Also on Table 1 are the analyses of my last ten papers, which you can recreate here. There are clear signs of improvement on every metric of scientific integrity. My R-index improved to a respectable 52.7%. While an R-index of 80% or higher is ideal (corresponding to Cohen’s recommendation that researchers achieve 80% power), an R-index between 50% and 80% should be considered trustworthy. I also saw significant improvements in my p-curve, TIVA, and median sample sizes (all Zs > 2.69, all ps < .007). And, yes, I just derived statistics on statistics of statistics, my first ever meta-meta-analysis.
While the p-curve of my last ten studies reveals that my papers continue to have informational value, the TIVA suggests that these same papers did not take advantage of questionable research practices. I’m getting better!
I’ve already mentioned the steps I took to improve, but it is worth repeating. The most telling part of my analyses reveals that the median sample size of my studies increased twofold. I think this one simple step can remedy many (though, not all) of our troubles. Instead of rushing through data collection until some arbitrary cutoff is reached (e.g., 20 per cell rule of thumb), I now routinely conduct power analyses with honest and conservative effect size estimates. As statistical power is not derived by sample size alone, but also by experimental design (among other things), I have made a concerted effort to use better designs, and as a result, my science has become more robust. Inspired by the elegance and replicability of cognitive psychology, where within-subject and repeated-measures paradigms are the norm, four of my last ten papers included within-subject factors, and six of my last ten papers used repeated measures dependent variables. These designs are extremely effective at reducing random error and between-person variability and thus produce results you can take to the bank. These designs have their limitations, and they cannot be used to address all questions, but I am convinced they can and should be used more often. And with a little imagination, I have been using them with great success. Most critical, perhaps, is that I have tried to be as transparent as possible—to others as well as to myself—by reporting results as fully as possible, even when they are unflattering.
I have been fortunate to have colleagues who have pushed me to become a better scientist (Thank you Uli! Thank you Liz!). Although I might have been reluctant at times, I have made concerted efforts to improve, and it is satisfying to see that I have. I haven’t stopped trying to get better, with my lab slowly embracing open science (see here, for example) and even experimenting with pre-registration. I am pushing to collect yet bigger samples.
I found the experience of checking myself to be humbling, yet gratifying. The results of my analyses suggest that at the outset of my career, I made mistakes, I skipped corners. But, by being open to the critiques of my beloved field—and by not denying our problems or making excuses or blaming outsiders—I have improved.
I no doubt continue to make mistakes, but hopefully I’m cutting fewer corners, and hopefully I’m helping improve social psychology. And as a quick reminder about how badly we need to improve, how badly we all need to check ourselves, when I perform the same set of analyses on papers that cross my desk, it doesn't take me long to flag papers that, to put it kindly, are simply non-robust. Many of these papers come from our elite universities, published in of our elite journals, with results sometimes appearing in elite newspapers. We can do better.
[1] To re-create the results for the R-index on p-checker, one needs to select “omit nearly significant results.”
on flukiness
this idea keeps popping up: if you conduct a replication study and get a null result, you need to explain why the original study found a significant effect and you didn't.
what's wrong with this idea? a few things.first, it seems to discount the possibility that the original finding was a fluke - a false positive that made it look like there is an effect when in fact there isn't. here's an analogy:null hypothesis: my coin is fairresearch hypothesis: my coin is weighted
original study: i flip the coin 20 times and get 15 heads (p = .041)
replication study: i flip the coin another 20 times and get 10 heads (p = 1.0)
do i need to explain why i got 15 heads the first time?maybe. or maybe the first study was just a fluke. that happens sometimes (4.1% of the time, to be exact).
what if the replication study was: i flip the same coin 100 times and get 50 heads? now isn't the evidence pretty strong that the null is true, and the original study was just a fluke? when an original study had low power and the replication study had lots more power, we are kind of in this situation. the most likely explanation is not that there is a meaningful difference between the original study and the replication study that needs to be explained, the most likely explanation is that the original study was a fluke, and in fact there is no effect.am i implying the original result was p-hacked??no. if we assume that people sometimes p-hack, then a false positive becomes an even more likely explanation, but if you're uncomfortable with this assumption, that's ok.* it's still pretty plausible that a small, underpowered study that got a barely-significant result was a false positive, even without p-hacking.where does the idea that we need to explain the original result come from? i think one source is that we underestimate randomness. we think that a significant result is almost like an existence proof (this doesn't help). that is, we think that if someone found an effect once, the effect really happened, just like if someone saw a black swan once, black swans must really exist. this has been debunked many times, but i think even when people rationally understand why the analogy doesn't hold, we have a hard time shaking the gut feeling that it happened once so it must have been true. but that's like saying that if i got 15/20 heads once, then the coin must have been weighted, at least at that time.
why doesn't the analogy hold? because there is error in our studies, but not in the sighting of the black swan, or in any other existence proof. the better analogy would be if someone reported that they are 95% sure that they saw a black swan, but it's possible it was actually a really dirty white swan,** they can't be totally sure. (well, there are other reasons the analogy doesn't hold up, so don't think about it too hard, but that's the main one).but isn't it possible the original result was real and there is a theoretical explanation for the different result in the replication study (i.e., a moderator)?absolutely. and it's also possible the replication is a type II error and the effect is real and robust. but my point is that it's also possible - very, very possible - that the original result was a fluke and there never was any effect. how do we decide among these explanations (no real effect, a robust real effect, a real effect with boundary conditions/moderators)? we don't have to decide once and for all - we shouldn't treat a single study (original or replication) as definitive anyway. but we should weigh the evidence: if the replication study has no known flaws, was well-powered, and the original study had low power and found a barely-significant effect, then i would lean towards believing that the original was a fluke. if the replication study is demonstrably flawed and/or underpowered, then it should be discounted or weighed less. if both studies were well-powered and rigorous, then we should look for moderators, and test them. (actually, i'm totally happy with looking for moderators and testing them anyway, as long as someone else does the work.)my point is simply that when the original study was relatively modest to begin with, and the replication study is rigorous, there is no need for any other explanation than 'hey, look, that original finding may have been a false positive.' i'm not saying let's decide 100% for sure it was a false positive and never study the phenomenon again. i'm just saying let's not make the replicators come up with a theoretical explanation for the difference between their results and the original results. we would be overfitting the data to a more complicated model when a simple one suffices: flukes happen. (for a nice demonstration of just how flukey p-values are, watch this.)in a recent blog post, sam schwarzkopf made an argument related to the one i'm trying to debunk here. in it, he wrote: "the only underlying theory replicators put forth is that the original findings were spurious and potentially due to publication bias, p-hacking and/or questionable research practices. This seems mostly unfalsifiable."it's absolutely falsifiable: run a well-powered, pre-registered direct replication of the original study, and get a significant result.
in a comment on that blog post, uli schimmack put it best:'[...] often a simple explanation for failed replications is that the original studies capitalized on chance/sampling error. There is no moderator to be found. It is just random noise and you cannot replicate random noise. There is simply no empirical, experimental solution to find out why a particular study at one particular moment in history produced a particular result.'it seems like that should be all that needs to be said. i feel like this entire blog post should be unnecessary.*** but this idea that those who fail to replicate a finding always need to explain the original result keeps coming up, and i thinks it's harmful. i think it's harmful because it confuses people about how probability and error work. and it's harmful because it puts yet another burden on replicators, who are already, to my mind, taking way too much shit from way too many people for doing something that should be seen as a standard part of science, and is actually a huge service to the field.**** let's stop beating up on them, and let's stop asking them to come up with theoretical explanations for what might very well be statistical noise.* it's not really ok.** get your shit together, swan.*** except for the underwater hippo.
**** if you think doing a replication is a quick and easy shortcut to fame, glory, and highly cited publications, call me. we need to talk.this is not a hippo.
Guest Post: Not Nutting Up or Shutting Up
Not nutting up or shutting up:
Notes on the demographic disconnect in our field’s best practices conversationAlison Ledgerwood, Elizabeth Haines, and Kate Ratliff
A few weeks ago, two of us chaired a symposium on best practices at SPSP focusing on concrete steps that researchers can take right now to maximize the information they get from the work that they do. Before starting, we paused briefly to ask a couple simple questions about the field’s ongoing conversation on these issues. Our goal was to take a step back for a moment and consider both who is doing the talking as well as how are we talking about these issues.
Apparently our brief pause sounded strident to some ears, and precipitated an email debate that was ultimately publicized on two blogs. The thing is, the issues we originally wanted to raise seemed to be getting a little lost in translation. And somehow, despite the absolute best of intentions of the two people having the (cordial, reasonable, interesting) debate, we had become literally invisible in the conversation that was taking place. So we thought maybe we would chime in, and Simine graciously allowed us to guest blog.*
As we said in our symposium, a conversation about where the field as a whole is going should involve the field as a whole. And yet, when we look at the demographics of the voices involved in the conversation on best practices and the demographics of the field, it’s clear that there’s a disconnect.**
For instance, the SPSP membership is about 56% female. Yet when you look at the speakers in best practices symposia at SPSP over the last four years, only about 30% are female. (That number drops to 23% if you remove our symposium.) Or take the ISCON Facebook page, where so many of these conversations take place. The membership is 40% female. The percent of comments on best practice-related posts made in the past year by females? 10%. You can see similar disconnects if you look at race/ethnicity, power, or institution type.
So here’s an interesting puzzle for the social scientist in all of us: Why is there this disconnect?
One might wonder: Is it intentional discrimination?*** As far as we know, there is zero evidence that anybody is sitting around intentionally excluding groups of people from the conversation. For instance, there happens to be an unofficial email list of more than 30 scholars in our field devoted to discussing issues in best practices as they arise. This email list is 100% white and 100% male.
But we find it extremely hard to imagine that any of them are actively trying to keep women or minorities off their listserv. In fact, we strongly suspect that most of them haven’t noticed the demographic composition of the listserv, and might be a little surprised if they did. Another example: That “Nut Up or Shut Up” blog post on replication last year was obviously not intended to make women feel alienated. (And nonetheless, it did.)
Here’s a much more compelling explanation for the demographic disconnect, and one that social psychological research can actually tell us a lot about: We suspect that a number of forces have combined to create unintentional barriers that limit the extent to which some scholars are involved and visible in the best practices conversation. For instance:
1. The question of whether to chime in (and whether that’s even a question)
Over the last few years, we’ve asked a number of scholars with strong methodological backgrounds to contribute to the best practices conversation, in talks or in articles, and we’ve heard many other people talk aloud about the prospect of chiming in.
Here’s what men tend to say: “Yes! I have a lot of opinions about this. Let me tell you about them.”
But here’s what women tend to say instead: “Hmm…I do have a lot of opinions about this…but I’m not sure I’m the most qualified person to be talking about them.”
In other words, whereas some scholars seem to see a logical link between having an opinion and wanting to express it, others seem to grapple with an extra question in the middle of that process: What gives me the legitimacy to say this?
These anecdotes shouldn’t be surprising to researchers who study topics like power and gender: We know that power tends to move people away from a deliberative mindset (e.g., “Should I say something? What are the pros and cons?”) and toward an implemental, action-oriented mindset (“I should do this now;” see Magee et al., 2005, for a review). And we know that in general, men are more likely to speak up, talk for longer, and interrupt more than women are (Dovidio et al., 1988; Hall & Sandler, 1982; Holmes, 1992; Zimmerman & West, 1975).
Certain demographics, then, are less likely to chime in because they get stalled somewhere halfway through a deliberative mindset. But notice what else this means. If people who feel higher in power are likely to jump in frequently and talk a lot, another unintentional consequence may be a crowding out of the less frequent voices. One of the people reacting to our symposium recently asked: Do the validity of arguments depend on the demographics of people making them? Definitely not. But if the airtime given to arguments depends on the demographics of people making them, maybe we should be paying attention to that.
2. How we’re talking.
The tone of the discussion and debates about best practices so far has been remarkable in its tenor and volume. Conference symposia, papers, and especially less formal posts on blogs and social media tend to be heated, polarizing, and often angry or moralizing. We challenge you to think of another topic in our field that could generate 289 comments in response to a single post on the ISCON Facebook page. Once again, social psychological research can tell us a lot about how this tone may be feeding into the demographic disconnect.
For instance, decades of research demonstrate what happens to women who act in powerful, directive, and authoritative ways. They are targets for backlash—the social, economic, and evaluative penalties of not being nice, kind, accommodating, and likable (e.g., Eagly & Karau, 2002; Heilman, 1983; Rudman, Moss-Racusin, & Nauts, 2010). Moreover, participating in conversations that get angry can be uniquely costly for women: When women (vs. men) appear angry, they are more likely to be written off as an “angry person” or “out of control”—in fact, whereas perceived anger tends to undermine women’s authority, men’s anger actually increases their perceived status (Brescoll & Uhlmann 2008). And when women are seen as talking too much, they are viewed as less competent and less suited for leadership than men who use the floor too much (Brescoll, 2011).
In other words, competitive discourse tends to hurt women and help men, in terms of how they are perceived by others. Women may therefore lean back from heated public discourse because they understand—consciously or unconsciously—the professional and social consequences of talking too much or talking with the intensity that has become de rigueur in the fields’ conversations about best practices.****
More broadly, framing debates in terms of who is a “real” or “good” scientist—a good group member—may tend to particularly discourage peripheral (vs. central) group members from expressing their views. Research suggests that peripheral group members tend to be more concerned about demonstrating that they are good, prototypical group members, and they may feel especially threatened by the prospect of being told that they are not (e.g., Jetten et al., 2006; Klein et al., 2007). In other words, to the extent that things like gender, ethnicity, power, and institution type correlate with how central or peripheral people feel as scientists or members of the field, the tone of the conversation may make the prospect of participating in the conversation feel more threatening or costly to some people more than others.
Of course, there are other reasons for the demographic disconnect, too. Feel free to chime in. Yep, that’s right. We mean you.
Why should we care?
Right now, our discipline is grappling with a number of complex and important issues on the best practices front. We are at a time when we need a lot of good and creative ideas. Diversity helps us achieve that goal—just look at all the research linking demographic diversity to improved decision-making (Smith et al., 2006), particularly in areas where innovation is important (Richard et al., 2003).
Including more voices also improves our chances of developing field-wide standards that are applicable and effective across a wide range of research areas. For instance, we don’t want just experimental social cognition researchers developing a set of field-wide standards based on their experience with running two-condition reaction time studies—we can’t assume that the same standards will apply equally well across areas like political psychology and social neuroscience and relationships research. The more voices we include in the discussion, the better our chances of incorporating a wide variety of researchers’ experiences with the complexities, nuances, and realities of data collection in their own subfields.
So what should we do?
Already have a spot at the table? Share it. Look around you for people with good ideas who might be hovering in the periphery, and invite them to pull up a chair (as an author, speaker, guest blogger, you name it). Be aware of when the airtime given to arguments tracks the demographics of people making them more than their expertise or thoughtfulness, and consider whether you could steer the conversation toward the latter more than the former.Meanwhile, if you’re waiting silently, deliberating whether to weigh in: Say something. Think about how long you’ve been conducting research. You know something about research practices. You may not have conducted a dozen simulation studies, but neither have most people contributing to the current conversation...they’re basing their opinions on a consideration of their own experience in the field. So if you’re sitting there with thoughtful opinions, speak up! Our field is striving toward a better science—toward identifying and honing research practices that maximize the knowledge we get from the work that we do. That’s not an easy task. We need all hands on deck.
*Prerequisite: A shared, profound fondness for footnotes?
**Note that we haven’t said anything about WHY there’s a disconnect yet, nor did we say anything about it during our symposium. There can be all sorts of reasons for disconnects. We’ll get to some possible reasons for this one soon. But in the meantime, if you feel like we’re attacking you? Calm down. This is about as vanilla as suggesting that the demographics of cat owners don’t match the demographics of society as a whole. We’re just mentioning some descriptive statistics.***We actually never wondered this, but apparently some people see the pie charts above as not just wondering it but claiming it, so let’s tackle this idea head on.
****Other women may jump into heated public discourse after a year or three of restraining themselves because they think maybe, just maybe, if they write this post, something will start to change.
lady problems
california. not relevant. just nice.below is a joint blog entry with lee jussim. the title, pictures, and post-script are mine, the rest is posted on both of our blogs.
joint introduction
recently, two friends and scholars who are working together on scientific integrity stuff had a very sane and civil email discussion about gender representation in the scientific integrity debate and in STEM more generally. at the end of the discussion, neither had convinced the other but they decided it was still an interesting and informative discussion, and they decided to post it on their blogs for the world to see. you're welcome, world.
you can find simine’s blog here:
and
Lee’s here:
Lee Jussim and Simine Vazire
3. 1. 15 Lee:
I recently went to a social psychology symposium that I thought was quite good with one partial exception. The first 3-4min of the talk was on how women have been underrepresented in the scientific integrity/best practices in science discussions so far. The speaker had good data that was pretty convincing to me that this is true (gender of authors, gender of presenters, etc.). Almost whenever I hear this sort of thing, without any substantive followup as to how this bears on the substance of the actual arguments, it grates on me. It comes across (to me) as implying, without explicitly stating or providing a shred of scientific/intellectual justification, that there is something wrong with the arguments to date because the demographic distribution of the people making those arguments does not meet some ideal distribution (the ideal could be "in society" or in "the profession" or "attending SPSP" or whatever).
As I have written before, I strongly support taking active steps to increase diversity in all sorts of contexts. Also, I thought her actual, scientific perspective brings something important to the table. Still, the only thing the evidence about gender representation did for me was to elevate the superficial importance of the CYA efforts on diversity -- to increase the appearance of diversity to undercut such (in my view, underhanded and non-substantive) comments. It is hard for me not to see this as political posturing rather than substantive -- at least, she presented nothing substantive about this as if "everyone knows" that an unequal distribution was itself evidence of a problem (if unrepresentative distributions constitute such a case, I have four papers on the lopsided political distribution of social psychologists that she would find appalling assuming she applies the same standard that "underrepresentation is inherent evidence of a problem.") Most social psychologists will wiggle around this by saying the standard only applies to underrepresentation of groups they care about, not groups they do not care about or do not like. Which kinda makes social psychologists, IDK, like everyone else in the world. And which, to me, is classic double standards evidence of how severe the political problems are in the field, but I am now digressing.
3.3.15 Simine:
That's interesting that you think the presenter/people are implying that a lack of diversity undermines the arguments/evidence. I would never have thought of that, and I suspect that wasn't the intention at all. I think it is important to point out these imbalances because they are so difficult to fix and can lead to a lot of disadvantage, and I think scientific integrity is a topic with especially extreme gender imbalance, which is really really troubling because I think (hope) the discussion about scientific integrity is the discussion about the future of our field, and if some groups are systematically underrepresented in that discussion, there is a risk they will be systematically disadvantaged in the new system that the discussion produces. If anyone says that the lack of representation undermines the quality of the arguments (without presenting any evidence), then I think that is completely wrong. But I think it is a very good thing that the presenter talked about this.
3.3.15 Lee:
If it is not relevant to the scientific argument, then the main alternative would seem to be that it is raising the spectre of hostility/discrimination/disadvantage, in this case, without a hint of evidence (other than the distribution inequality itself). The idea that distributions alone represent discrimination is odd, as can be seen by the gender representation of SPSP. Put differently, the if "unequal distributions means discrimination/disadvantage" then one would be compelled to conclude that early career men are being subjected to nasty discrimination in social psych. People who really believe that should be going wild advocating for a massive program to increase the representation of men in the field*. It is not all that lopsided yet, but the trajectory is headed for massive lopsidedness. If the trajectory was in the other direction, people would be loudly objecting. How do I know? That SPSP talk would be a good place to start as anecdotal evidence...
I really do not like disputing anything with you, Simine, because I hold your views in such high regard. At the same time, I also do not like these sorts of implicit importation of agendas without evidence, and the double standards (in context of the distribution of SPSP) they imply. Is there really a shred of evidence that women are being rejected at higher rates for these events? Dismissed as irrelevant at any higher rate than anyone else? Maybe there is, but it was not presented. Absent evidence of either scientific relevance or disadvantage/discrimination, I am still left feeling like it was a section of the talk filled with insinuations without evidence, and wondering, what was the point?
* Demographic distribution of SPSP.
3.4.15 Simine:
First, I want to say that I really really appreciate that we can have an open conversation about this, Lee, and I too very much respect your point of view and have no doubt that we share the same underlying values. And also, I enjoy disagreeing with you because it's rare to find someone I can disagree with and know that they will hear me out and even if we don't end up agreeing, I will be glad we had the conversation.
I have two reactions to your last email. The first is that you're right, I do think that underrepresentation of a group that has traditionally/historically been disadvantaged is an indirect sign of discrimination. (So it's not imbalance alone, but imbalance that tracks demographics or other lines that are associated with discrimination at the societal level.) I admit this is making some assumptions, but I think they are not crazy assumptions. And this is also why I don't get worked up about imbalances in the other direction - fewer men than women doesn't bother me too much because there have not been many barriers to entry that disproportionally affect men. (Also, the imbalance at the junior level is unlikely to trickle up very fast because of the leaky pipeline for women). So I admit that I do see imbalance/underrepresentation as a sign that there is some (probably unintentional) disadvantage or hardship that disproportionally affects women. I'm not sure I think it's necessarily discrimination, I don't think it has to be, it could be a lot of other small things that are not anyone's fault but still cause women to be at a disadvantage. I could tell you at least a dozen stories from my own experiences, all of which do not involve a 'bad guy', but constitute a disadvantage/obstacle nonetheless, and were, I believe, related to my gender. And I think you know me well enough to know I'm not the kind of person who sees sexism everywhere. (And, just to be clear, I also recognize that I have been extremely lucky in general and have had a lot of opportunities, and that some of those were because of my gender (i.e., affirmative action)).
My other reaction is that even if I give up on that point, and I agree, for the sake of argument, that imbalance does not provide any evidence of discrimination (or hardship or whatever you want to call it), I still think the imbalance itself is something we should try to fix. This is also related to the fact that women have historically (and still today) been at a disadvantage, so I think it is important to try to help make sure women are included in the most important 'clubs', whether that be informal conversations or formal organizations. If women are not at the table, even if it's for totally benign reasons, this is likely to lead to more disparity in outcomes, and I believe there is likely to be an interactive effect with sexism/discrimination such that a disadvantage, even one that had nothing to do with discrimination, will get compounded if the group being disadvantaged is also a group that suffers from discrimination at the societal level. Again, I am making a lot of assumptions, but I think they are reasonable (based on some empirical evidence, mostly descriptive stats from big surveys, and also from personal experience, which, while it might not be the best source of evidence, is really hard to ignore :)). Also, I think this second point is really important because many times when people are pointing out a dearth of women, they are doing so for this second reason (even if they also agree with the first), and I hope people won't assume that when we talk about gender we do so only because of concerns about overt discrimination. I'm not sure that the insinuations you are seeing are there, and I hope we could separate my first point from my second such that even people who disagree with me about the first would consider the second as a potentially good reason to care about gender (especially when the imbalance is in the women < men direction).
I bet you won't agree with me on many of these points, but I am curious if they at least make sense to you. And hopefully they will help explain why we see things differently. I would love to talk to you about this more sometime, it's really interesting for me to hear your thoughts!
3.5.15 Lee:
Hey, Simine, while I do not find the inequality of distributions that tracks historical discrimination argument as suggesting ongoing discrimination at all persuasive (absent other independent evidence of ongoing discrimination)*, I actually do agree with and find your second set of points persuasive. First, when direct evidence is hard to come by, I have no problem admitting anecdotal evidence**.
Second, disproportions risk creating the appearance of hostility or not being welcome -- and that can become self-fulfilling. Thus, taking proactive steps to make sure people -- especially people who might be wondering whether people "like them" are welcome*** -- is usually something I do support.
* I recently reviewed a paper for Proceedings of the National Academy of Sciences. It was essentially an audit-style experiment of faculty hiring in STEM fields. As an experiment, it was able to hold stuff constant and manipulate stuff. Women emerged with a 2:1 advantage in hiring. I'd give the reference, but I cannot do it till it is published, which, as of right now, I cannot be sure that it is. This stands in stark contrast to the Moss-Racusin/Dovidio study, which was about 1/5 the sample size, and did find modest favoritism for male applicants for a (relatively low grade, at least compared to faculty) lab manager position.
** On anecdotal information. Here are two true anecdotes. About 3 years ago, a student of mine submitted a proposal for a small grant from an internal Rutgers source to study how people's politics can lead to biased views of science. It was rejected on the entirely scientific grounds that the theory and importance was unclear, and there were problems with the method. When I reviewed the proposal, I saw no such problems, but I did see this:“The field of psychology is dominated by liberals (Redding, 2001), and this political homogeneity can be problematic… In fact, content analysis of all the articles published in American Psychologist during the 1990s revealed that 97% had liberal themes (Redding, 2001). Furthermore, recent research suggests many social psychologists would blatantly discriminate based on politics (Inbar & Lammers, 2012)…”
"Oh, crap," I thought, "of course they rejected it -- it is telling them that this proposed project is on them, and that it proposes to catch them in the act of distorting science with their politics." Of course, I might have been wrong, those substantive comments might have been completely sincere and bona fide. How to distinguish the two? Run an anecdotal experiment -- anecdotal, because it is one shot (no random assignment of lots of people to conditions), but still an experiment (change one thing and see what happens!).
I instructed the student to delete the "offending" paragraph, and replace it with this:
“… Science has a long and checkered history of periodically being used and exploited as a tool to advance nefarious rightwing political agendas (e.g., social Darwinism; Nazi eliminationist practices; Herrnstein & Murray's (1994) claims about genetic bases of race differences in intelligence).”
We changed nothing else. The proposal was funded. Slamming Nazis, and Herrnstein & Murray, apparently, made the theory clear and cogent, and the methods valid and sound.
Anecdote 2. A collaborator of mine has been on a roll undoing the field's view of conservatives as incompetent and immoral. One avenue has been to show that, in areas where others have found cons more biased than libs, if you tweak the methods just a little bit, you get similar levels of bias, and conditions under which libs are more biased than cons. We had collaborated on a paper that demonstrated the latter (lib bias > con bias). It said so. We could not get it published.
He then had a brilliant stroke of insight. He reframed the paper, and, though the info is there in the tables and figures, took out all mention of lib bias > con bias from the text.
That version got published.
These are anecdotes. In the anthropology and qualitative social science literatures, they are called "lived experience."
*** See the references for my paper on a group very unwelcome in the social sciences, and for which there is massive inequality in distribution.
3.6.15 Lee:
Simine,
That paper I reviewed and referred to as being embargoed was just accepted. Although no one piece of research is ever definitive on these issues, or, probably, on any issue, I would say it is a must-read for anyone arguing that sexism is a major problem in STEM. It is also listed in the references.
To paraphrase and partially quote from the paper:
The 2:1 bias found in favor of hiring women for faculty positions means that it is a propitious time for a woman to be entering STEM fields.
A final note on "disadvantage." As you know, I am now in Australia for a conference. There is a pretty famous White guy who could not come because he has two small kids at home. A former student and ongoing collaborator of mine had a sabbatical. He wanted to spend it in Europe, with one of his his other collaborators. He couldn't because he has two small kids at home. For most of my career, I felt unable to do things like I am doing now (spending a year on sabbatical at Stanford, attending conferences and giving talks all over the world) because of the exact same thing. My sabbaticals were staybatticals. I turned down an invitation to Stanford’s Center for Advanced Study in the Behavioral Sciences (CASBS) early on because of that. I was not willing to uproot my family to search for jobs all over the country, probably depressing my income compared to those capable of doing such searches. I always thought of these decisions as "choices" not "disadvantages." However, if I wanted to use the rhetoric of disadvantage -- applied in this case to social psychologists with and without small children, and with or without strong geographically based family ties -- it would not be difficult. Years ago Bella DePaulo had a couple of articles about prejudice against single people. I am on a dissertation committee about prejudice against married people who choose not to have children.
When you carve out all the potential ways we are all "disadvantaged" there are very very few people left over as advantaged. I am not claiming an equivalence between all forms of "disadvantage" -- but I am claiming we have lost our way. Life's challenges and "disadvantage" are not the same thing, and, of course, those challenges are different for different people.
None of this is a case against proactively seeking diversity, whether for our conference or anything else. None of this argues against taking people's unique life circumstances into account in trying to accommodate and include them in things valuable and important. Nor am I denying the existence of bona fide, ongoing discrimination against certain groups. I do, however, reject the notion that unequal distributions are, by themselves, even suggestive of discrimination. Of course discrimination can produce unequal distributions, but it can also produce equal distributions (discrimination against a group that is in fact more highly qualified can produce an equal distribution*) and unequal distributions can result from many processes other than ongoing discrimination in the present.
Lee
P.S. Simine -- perhaps we should consider publishing an edited version of this back n forth? It is one of the few sane & civil discussions** of these issues I have ever seen, and the idea that we can disagree on some of these issues respectfully is probably something that is worth somehow making public...?
* Admissions of Asian students to college has long been below what would be predicted on the basis of their SAT scores. So, even though Asian students get degrees at much higher levels than any other group, including Whites, this discrepancy between achievement and admission rates sure smells like bias to me.
** Plus, like you taught Funder, you have taught me to love footnotes.
Recommended Readings For Anyone Interested in Any of These Issues
DePaulo, B. M., & Morris, W. L. (2006). The unrecognized prejudice and discrimination against singles. Current Directions in Psychological Science, 15, 251-254.
Duarte, J. L., Crawford, J. T., Stern, C., Haidt, J., Jussim, L., & Tetlock, P. (In press). Political diversity will improve social and personality psychological science. Behavioral and Brain Sciences.
Jussim, L. (2012). Liberal privilege in academic psychology and the social sciences: Commentary on Inbar & Lammers (2012). Perspectives on Psychological Science, 7, 504-507.
Moss-Racusin, C. et al (2012). Science faculty’s subtle gender biases favor male students. Proceedings of the National Academy of Sciences, 109, 16474-16479.
Williams, W. M., & Ceci, S. J. (In press). National hiring experiments reveal 2-to-1 faculty preference for women on STEM tenure track. Proceedings of the National Academy of Sciences.
END EMAIL CONVERSATION
postscript by simine (on simine's blog only):
you don't have to be a supergeniusto do the animal thingyou don't have to be a supermodelor, in the words of the inimitable ani difranco:i've been wanting to blog about the underrepresentation of women in the scientific integrity discussion for a while. i hesitated because i couldn't figure out exactly what i wanted to say. at the same time that lee and i were having this discussion, i heard someone on npr quote roxane gay: "i'd rather be a bad feminist than no feminist at all". that helped me decide to post this - i don't have any answers, but i think it's an important enough topic to try to fumble through the discussion anyway.i hope more people will join in the scientific integrity discussion. don't wait for someone to anoint you an expert on scientific integrity. none of us are. all of us say some stupid shit sometimes.* join us.to open your face up and sing
*have you read facebook?
supergenius.
this is what p-hacking looks like
keri russell plotting her next QRP
i am teaching a seminar called 'oh you like that finding do you? well it's probably FALSE.'
the students are a little bit shell-shocked.
i am having the time of my life.
the hardest part is explaining how perfectly smart, truth-seeking scientists can repeatedly do idiotic things. happily, i have plenty of examples from my own life.
just this week, i almost p-hacked. the details are a little dry, but i think it's worth telling this tale of mundane p-hackery, because this is what p-hacking looks like in the wild. it is not sinister and dark like a good episode of the americans. it is super boring like the stories you read to your children at night.* in fact, if you run out of bedtime material, read them this.
once upon a time, there were some personality researchers who wanted to measure personality.
they collected self-reports and six peer reports from each of their participants.
they wrote a paper examining how the big five personality traits correlate with friendship satisfaction.
because the paper was due right when the data came in, and organizing the peer reports takes time, they only used self-reports of personality (bad personality researchers!).
the reviewers and editor said 'dude, we know you have peer reports. and you shouldn't just correlate self-reports (of personality) with self-reports (of friendship satisfaction).'**
so they compiled the data and ran the analyses with the peer reports of personality. not surprisingly, all the correlations with (self-reported) friendship satisfaction were weaker. still significant (yay large samples), and with a similar pattern as the self-reports, but weaker.
they started off by reporting all of their results, with the self-reports and the peer reports side by side.
then their tables got unwieldy, and they decided that they should just aggregate the self- and peer-reports into a composite measure for each personality trait. because obviously that is the best measure of personality. and it would make their tables easier to read.***
so now, gentle reader, we have a self-report and (up to) six peer reports for each participant. we want to aggregate them. what are we to do?
option a is to first aggregate the six peer reports, and then average that composite with the self-report, in which case the self-report is weighted as much as ALL the peer reports put together.
option b is to just average all seven reports (self-report and six peer reports) all at once, in which case the self-report is weighted only as much as any single peer report.****
pop quiz hotshot: which one will give us "better" results?
option a would give us bigger effects. and many many researchers in this area***** would say option a is the better option because self-reports are special and should be weighted more than each individual peer report. so our protagonists would have cover if they went with option a.
option b, however, is what our protagonists have typically done in the past. partly because they are convention-busting mavericks and partly because they believe that the self is not that special - each of your close friends knows about as much about your personality as you do yourself.
so the noble researchers sat there and, for several minutes, contemplated what to do. 'contemplated' is the wrong word. it was immediately obvious to them what they should do. they even said 'this is p-hacking, what we're contemplating doing right now.' out loud. and still, they sat there for another minute. squirming.
then they did the right thing because that's what protagonists do and would i really be telling you this story if we p-hacked? come on.
the moral of the story is: not p-hacking is HARD. side effects include redness, swelling, and deep, deep frustration. and self-righteousness.
what's that? your kids are still awake? here is a fun little postscript:
another thing the reviewers said was 'look, i know your analyses were exploratory, but anyone with half a brain would have predicted finding ABC. stop dicking around and, in the introduction, tell your readers about how all existing personality theory and research would predict ABC.'
what are our protagonists to do? they did not predict ABC, but mostly because they were too damn lazy to make specific predictions (bad bad researchers!). they did not want to lie in the introduction. but they also agreed that only a dumbass would not predict ABC.
in a brilliant stroke of genius, they decided to write: 'previous theory and research on A and B and C would clearly suggest that ABC. although this would be a reasonable prediction, we did not actually make this prediction a priori' and they lived happily ever after.
the end.******
* i am extrapolating from the stories my dog makes me read at night. i hear parents love it when you compare their kids to dogs.
** this is called getting a taste of your own medicine. your kids will appreciate this lesson someday.
*** of course we are putting the data and disaggregated results on OSF. we're not total idiots.
**** the younger kids might get confused here. just write out the formulas for them.
***** almost all seven of them.
****** everything in this story is true (you know, without the dicking around and the dumbass bits), but in our defense, our paper (and methods) were a little more sophisticated than described here.
sweet dreams.
modus tollens bitches
i am not going to say anything original in this post.
my philosophy of science friends tell me that i should abandon popper. before i do, i'd like to have one last fling with him.
i am teaching two research methods classes this quarter, so i'm teaching, you know, how scientists are objective and our theories are falsifiable and we seek to disconfirm them. (hello psc41 students!)
today's topic: modus tollens and falsifiability.for those of you who don't remember day 2 of research methods, here is modus tollens (aka, denying by denying, a valid form of logical argument):if p then q~qtherefore ~pwhy is modus tollens important for empirical research? i am cribbing from meehl 1978 here (who himself was stealing from others, including of course popper), but basically, it's how we make our theories falsifiable. we make risky predictions of the sort:if my theory is correct, then i should observe a certain pattern of data.then we collect data.if the data show what we predicted, then our theory is left standing. of course it doesn't prove that our theory is correct, that would be affirming the consequent, and we all know that's a logical fallacy.* but we do have a tendency to say that the data are consistent with our theory (true), and even that the data support our theory (maybe less warranted, depending on how risky the prediction was - if a lot of other theories are also consistent with the data, then i'm not so sure the data really support our theory). basically, if the data come out as predicted, we celebrate and claim to have found evidence for our theory.the interesting part comes when the data don't come out as predicted. according to modus tollens, if we find ~q (not the predicted results), we should conclude ~p (our theory was wrong). let's put it this way:if my theory is right (T) then i should observe these data (D)
i observe data that are not those i predicted (~D)therefore, my theory is wrong (~T)
that's how it should work, and that's what makes our theories falsifiable.
but, as meehl pointed out, what we are really testing is something like the following:if my theory (T) is right, my hypothesis (H) is appropriately derived from my theory, and my methods (M) are sound, then i should observe these data (D)
or:
if T.H.M ---> Dso now what happens when we observe ~D? well, we no longer have to conclude ~T, because we have two other options: ~H and ~M. that is, instead of concluding that our theory is wrong, we can throw the hypothesis or the methods under the bus.
~M is especially tempting, because all it requires is to say that there was something wrong with the methods. it's a very easy escape route. blame the subject pool. blame the weather. blame the research assistants.**as etienne lebel and kurt peters pointed out, also echoing meehl, in psychology we are often quite happy to throw our methods under the bus. that's because we are measuring things that are very difficult to measure, and also because we probably don't pay enough attention to the validity of our methods. in a way, there is a perverse incentive to gloss over the methods - the less well-defined and rigorous they are, the easier it is to escape the ~T conclusion by escaping through the ~M door. if we bolster our methods, and think of all the possible pitfalls ahead of time, we block that escape route and are left in the uncomfortable position of risking falsifying our hypothesis or our theory. god forbid.***this is a problem for (at least) two reasons.1. when a researcher is conducting original research, it is too easy to ignore null results by attributing them to a quirk in the methods. there is always some rationale for sticking a null result in the file drawer, so long as you can point to some potential flaw in the method.2. when a researcher is conducting a replication of someone else's work, it is too easy to ignore null results by attributing them to a quirk in the method. it's even easier here, because you don't even need to point to a specific potential flaw in the method - you can just say that the researcher lacked expertise, or did not have the tacit knowledge necessary.what can we do?1. improve our methods. if we really want our theories to be falsifiable, we need to close the ~M loophole. think ahead of potential flaws in your design, and only run the study once you are fairly certain that you will believe the results, even if they are null. of course we can't anticipate every flaw, and some results legitimately belong in the file drawer. but we can shrink the loophole by using well-validated measures, a large sample, thinking ahead of potential moderators (and pre-registering them when possible).2. don't throw the methods under the bus. consider the possibility that the theory or hypothesis is wrong. make it your new year's resolution: i will not write off a null finding or a failed replication unless there is a glaring error -- one that my enemy's mother would agree is an error.3. when publishing your original results, provide enough detail about all important aspects of the method so that if someone attempts a replication following the procedure you describe, you cannot blame their result on poor methods. the onus is on us as original researchers to specify the things that matter. of course some are too obvious to specify (e.g., make sure your experimenter is wearing clothes), but these are covered by the 'glaring error that your enemy's mother would agree with' clause in #2. if you don't think another researcher in your field could competently carry out a replication with the information you publish, that's on you. if your procedure requires special expertise to carry out, specify what expertise. if you can't, your theory is not falsifiable.still 3. of course, sometimes a failed replication makes you think of a potential moderator you hadn't thought of before. good for you. i don't really care, unless you are willing to go out and test that moderator. show me a confirmatory study in which the moderator does its moderation thing, and i will believe you. otherwise, your modified theory is as good as any untested theory.
but never mind, popper is dead. of course i know**** that naive falsification can't be right because of the problem of compensatory adjustments and the probabilistic nature of science. but still, aiming to make our theories more falsifiable***** seems like a good idea to me.
* you're a logical fallacy.** don't blame the research assistants.*** this week i managed to convince 94% of my class that they should switch doors in the monty hall problem. this is not actually related to my blog post, but i wanted to tell you about it anyway. to be fair, 27% thought it was the right strategy at the beginning of class, so i only get credit for 67%. still, i am pretty pleased with myself.**** thank you wikipedia.***** i can feel the philosophers cringing.
why do the black swans get all the attention?
happy festivus (and the meaning of life)
i have been re-watching the west wing*, and i think i just discovered the meaning of life. in the episode i just saw, president bartlett comes back from church and complains about the pastor** getting ephesians wrong. the point of ephesians, he says, is:
be subject to one another.
i am not religious***, and a quick glance at the rest of ephesians 5 tells me that i probably don't have much in common with paul****. but the phrase resonated with me. and it reminded me of a passage i read years ago:
'You fight your superficiality, your shallowness, so as to try to come at people without unreal expectations, without an overload of bias or hope or arrogance, as untanklike as you can be, sans cannon and machine guns and steel plating half a foot thick; you come at them unmenacingly on your own ten toes instead of tearing up the turf with your caterpillar treads [...]'
-philip roth, american pastoral*****
it applies to scientific inquiry - it means subjecting ourselves to different opinions, different perspectives, disagreement. and actually being open to changing. it means more than just discussing or interacting with one another. it means considering, for a moment, that we might be wrong.it also applies to life. there is a lot to be said for toughness, resilience. but taken too far, one becomes impermeable. and the best things in life come from being permeable.
so that is my big insight for the year.
be untanklike.
be subject to one another.
happy festivus.
* for the third time.
**priest? minister? i googled 'what do you call the guy who gives the sermon at church?' but that did not help.
*** can you tell?
**** 'let there be no filthiness nor foolish talk nor crude joking.' no foolish talk? no thank you.
***** p. 35. guess how many pages i read?
why i am optimistic
i was recently engaged in a totally civil disagreement on facebook* about whether things are changing in our field in response to all of the concerns about the robustness of our scientific findings, and the integrity of our methods. i said i was optimistic. someone asked me if i could elaborate on why i am optimistic. this is for you, brent roberts.1. the obvious reasonsthe fact that the open science framework exists, that the reproducibility project exists, that social psychology did a special issue on replications, that there have been workshops on reproducibility at NSF, that SPSP had a taskforce on best practices, that APS had a taskforce that led to changes in submission guidelines and procedures at psych science, that psych science commissioned and published an article and tutorials on effect estimation approaches by geoff cumming, that perspectives coordinates and publishes registered replication reports, and that every conference i've been to for at least the last two years has included some presentations on replicability. everyone already knows all of this, but it's easy to take it for granted. i'm pretty sure that if i had told someone in 2007 that all of this would happen in the next seven years, they would not have believed me.2. i am not a pariahi am shocked and amazed that people still tolerate me after i spew my not-always-entirely-well-thought-out opinions on my blog. when i started blogging i never thought that 1,500 people would see my posts. i'd like to think this has something to do with my biting sense of humor, but i know it's because there is a huge amount of interest in this topic. other blogs, like sanjay srivastava's and daniel lakens's get tons of readers, i'm sure. betsy levy paluck has 1,500 followers on twitter. dorothy bishop has 16,000.** think about it, people are choosing to spend their free time reading about these things. lots of people. that blows my mind.also, brian nosek, who has arguably done more for The Cause than anyone else in social/personality psychology, is almost universally well-liked. one indicator of this is that brian is the winner of this year's spsp prize for distinguished service to the field of social/personality psychology. perhaps there are people out there who hate his guts, but no one dares say it in public because his reputation is that strong. if nothing else, i am extremely proud of my field for recognizing the value of everything he has done.3. key peoplethere are several constituencies whose opinions i think are especially relevant for whether or not optimism is warranted.
the first is the decision-makers.as suggested by my last post, i think among the people with the most power to implement change are journal editors. and the people who select the journal editors get to choose who wields this power. which brings me to a little story.
recently i was at a reproducibility workshop at the center for open science. in one of the breakout sessions, a group of us were discussing what journals and societies can do to increase the reproducibility of social science research. several editors in chief were there, as well as several executive officers of Very Large professional social science organizations. at one point, one of the executive officers said that it would be difficult to make large-scale changes across all of the association's journals because not all editors would be on board, even with policies that have broad support from members. at this point, i said something i have been thinking for a long time: if a society claims to value X, they should appoint editors who share that value. for example, if a society explicitly values theoretically-rich research, they should select editors in chief for their journals who share that value and will implement it. so if a society values particular research practices, it should make agreement with that value a litmus test for selecting editors in chief. what happened next made me extremely hopeful: the executive officer said that it might make sense to ask candidates for editor in chief of the association's journals to submit a statement on reproducibility. i'm still waiting to see if that will happen. if it does, it will make me very happy. note that i'm not saying every society must value reproducibility (though i think they should), i'm just saying that if they do explicitly claim to value it, then it makes sense that they should care what their editors in chief think about the issue (and even then, it makes sense that they would be willing to accept a range of views, but some views, i.e., those explicitly in tension with the society's values, should probably be dealbreakers).the second constituency that matters a lot: the youths. i may be wrong, but my strong impression is that there is very broad support for change among junior scientists in our field. in fact, i have been amazed and heartened by grad students' interest in these issues. i am also surprised at how many of the prospective graduate students who contact me mention this issue. these are people who have not yet started their phd. and they are looking for mentors who care about these issues. in my opinion, this interest among early career researchers is going to trickle up. if our graduate students and junior colleagues want to talk about these issues, we are going to talk about these issues. to all of you early career people out there, keep the pressure on. i don't mean be aggressive about it or dogmatic - in fact, if you just joined the field i would encourage you to read and listen a lot before making up your mind - but keep bringing it up, keep asking questions, keep expressing your concern and interest about these issues. your voice matters more than you think, and it gives the rest of us hope.why it matters that things are changing
why does it matter if optimism is warranted? i think it matters for a few reasons.first, if things are changing and most people are on board with these changes, that means that writing and talking about these issues, spending some of your research time on them, is not career suicide. it means it's ok for graduate students to spend some (probably not a lot, but some) of their time on this. my limited experience so far suggests that they will still get good jobs. i could be wrong, and this is a dangerous conclusion to come to if i am wrong, but i don't think i am.***second, and this is perhaps the most important point i want to make in this post, if what we're doing is working, we don't need to change our approach.so far, i can say that i agree with and stand by most of the efforts that have been made in the name of scientific integrity/reproducibility etc. i am extremely proud of the people who are carefully, rigorously carrying out replication studies (and i don't like it one bit when their motives are called into question - these people are doing a gigantic service to the field and are being incredibly diligent and careful. most of them are the model of scientific rigor).****i read almost all of the discussions on the facebook ISCON page, and while there are occasionally remarks (on both 'sides') that i find unnecessarily personal or emotional, i find them to be overwhelmingly calm, reasonable, and fruitful. i wish there were more women participating in those discussions, but overall i am proud of the level of discourse there and elsewhere.what i like about the conversation that is happening is that it's about the science. it is not about the people behind the science, or their motives. those things should not come into it. if you want to vent about specific people and their motives, start a gchat group with your friends. but don't publicly accuse someone of having bad motives, or lying, or whatever.
this goes for criticisms of original research and criticisms of replication studies: don't tell me what you think the authors' motives were, tell me what's wrong with the science.the flip side of this is that it is completely fair game to criticize people's methods, results, and interpretation of their results. that is not personal. that is not bullying. that is just science, and in fact it's good science. science without critical evaluation is not science. the fact that it sometimes happens post-publication does not make it unscientific or personal. and a personal attack in response to purely scientific criticism is bullying.here's a good rule: critical comments about research should take the same tone as reviewers' comments during the peer review process. we don't tolerate personal attacks there, and we shouldn't in our public discourse. but we should tolerate, and even encourage, skepticism and criticism of the science. i know that's a nuanced position, but i'm sticking with it.i am afraid that pessimism is going to drive people i mostly agree with to do things i don't like in the name of a cause i care deeply about. i'm afraid people are going to get impatient, think that our current approach isn't working, and start getting emotional or personal. besides the fact that that would be wrong and unscientific, it would also backfire. it will not convince anyone and may turn people off who are still deciding who to listen to.so if you are one of those people who want things to change and are getting impatient, i hope you will give the current approach a little more time. every month a new journal adopts new disclosure guidelines, data sharing policies, methodological standards, etc. i know that people have been talking about reforming our research practices for decades, and for a long time nothing changed. but things are changing now. so many things.* i know, i didn't think those existed either.** i don't know much about dorothy bishop, in fact i know almost nothing about her, but i really like her blog posts. i also like the funny animal pictures she retweets.*** but i might be.**** yes, i'm talking about david johnson, felix cheung, brent donnellan, and lots of others.
open letter to editors
dear editors,
i love journals. i love editors. i love editors of journals. that's why i want to help. we need more quality control in our journals, and you are the ones who can do it.
in july 2012, in a comment to a blog post, chris fraley wrote 'What we might need, in other words, is a formal “consumer reports” for our leading journals.' i was so excited by this idea that i wrote to him and told him it was the best actionable idea that has come out of the replicability discussion.* fast forward 27 months, and our paper, 'N-Pact Factor: Evaluating the quality of empirical journals with respect to sample size and statistical power' is out.
you can read the actual paper here, and you can read chris's blog post about it here.
the N-Pact Factor (NF) has many flaws. it is not a perfect measure of quality. but it is better than nothing (or better than the Impact Factor alone). if you want to improve on the NF, please do. we hope you will. i would love to see a world in which there are many indices of journal quality, all tapping into various facets of quality. in my ideal world, i could look up not only the average sample size of studies published in a journal, but also how often they publish studies with multiple methods, field studies, non-student samples, longitudinal designs, actual behavior, replications, open data, preregistration, author disclosures, etc. but if i can know only one thing about the studies published in a journal, it would be their sample size.
so this is a call to editors to please please please pay attention to sample size when evaluating a paper.
why am i so obsessed with sample size? in short, because it is shorthand for 'amount of information'. and ultimately, that's what we're in the business of producing: information/knowledge. larger samples lead to more accurate conclusions. they reduce type II error, and indirectly reduce the proportion of published findings that are likely to be type I errors (read the paper for full explanation). in short, large samples mean fewer mistakes. *** (also, for a great summary of the problem with small sample sizes, see my favorite spsp talk ever.)
below i address some potential objections to this obsession with sample size, and make some controversial statements. i speak only for myself here.
how much is enough? one problem with saying 'we need bigger samples' is that it's hard to know how much is enough. i agree this is a problem. it has an easy solution: 200.
let's start with 200. it might turn out not to be enough****. but there are some good reasons to start there. first of all, a sample size of 200 gives you 80% power to detect an effect size of r = .20 (d = .40), which is about the average published effect size in social and personality psychology*****. also, this. and sanjay's 2013 arp talk. (you had to be there******).
what about power analyses? you don't need power analyses. if there is an effect, it will probably be somewhere in the ballpark of all the other effects in social and personality psych. so just go with 200 (or 250. or 300.). my view on power analyses is we can do one for the entire field, and i just did it, and it told me we need 200 people (or 100 per condition). there, we're done with power analyses. (yes, there are exceptions, blah blah blah.)
but i can't afford to run 100 participants per condition. then don't do the study.
but seriously, if you are doing a typical social/personality study, that is, you are running college student participants in a lab and they are doing some stuff on computers and maybe talking a little bit and eating some chocolate, you can run 100 people per condition. if you can't, i'm sorry. i feel bad that you can't. but that doesn't mean we should publish your paper.
if you are not doing a typical social/personality study, that is, if you have non-college-student participants*******, or you are using intensive or expensive methods, or studying something rare, etc., then we should absolutely take that into consideration and be flexible about sample size. each editor needs to weigh the value of the evidence, which includes how important it is and how hard it would be to collect more data.
my recommendation for editors: if a study has fewer than 200 people, the authors should give a convincing reason why it should be published anyway. there are many potentially good reasons, and editors need to use their judgment all the time anyway. make this part of your evaluation.
but this will take forever! yup. there will be fewer papers to read. and review. and for the media to sensationalize. what a terrible, terrible loss.
in my view, increasing sample sizes kills two birds with one stone. (actually i think it kills about eighteen birds. in the good sense.) it improves the quality of our published findings, and it address the looming crisis of The Dearth of Reviewers********.
in conclusion. every time i see a paper with a simple design and 36 participants, i die a little inside. the point of the NF is to encourage editors (and reviewers, and ultimately authors) to place more value on sample size. don't tolerate small samples, unless there is a good reason to. let's push ourselves and each other to do the hard work of doubling or tripling our samples. it is painful. it sucks. it's not fun. but we can't go on like this. and i think in the end we may actually come to enjoy living in a world where there are only 228 new papers to read each month. all of us can help make this happen, but you, editors, can do the most.
xoxo,
simine
* i also wrote 'it wouldn't be hard to do, right?' i crack myself up.
** not literally 'we'. thank you yuchen and ashley.
*** sample sizes do not decrease all kinds of mistakes (i.e., systematic error is not reduced), but they reduce random error and they don't increase other kinds of error.
**** i hope to one day look as foolish as simmons, nelson, and simonsohn now look for their n = 20 recommendation in 2011.
***** note that this means that half of all published effects are smaller than this. which means you should probably assume the effect you are studying is smaller than this. which means 200 people is probably not enough. i am starting to feel foolish already.
****** happily, you can attend the 2015 arp conference. i am almost positive sanjay will be there.
******* i don't mean mturk. if you are using plain old mturkers, you better not have fewer than 200 people or i'll get really mad.
******* soon to be a major motion picture in which editors go insane and start stalking people who submit 42 manuscripts a year and turn down all review requests. starring mindy kaling.
mindy kaling is not impressed.
Guest Post by John Doris
with permission, an excerpt from john doris's forthcoming book. i wanted to post this because i think it is an excellent summary of the current crisis in social/personality psychology from a well-informed 'outsider'. this will be review for many psychologists, but should be useful for new researchers, and for those outside of psychology who want some background/analysis from someone who is not in the trenches but has read and thought a lot about this literature and these issues.
art by melissa dominiak
Doris, J. M. 2015. Talking to Our Selves: Reflection, Ignorance, and Agency. Oxford:Oxford University Press.
[…] At this writing, social psychology is being shaken by charges that many published findings, including numerous iconic findings, do not replicate when tested by independent investigators. Cyberspace is thick with skirmishes between Replicators, who broadcast the failed replications, and Finders, who insist that their findings are real. Viewed from a safe distance, it’s all good fun, the sort of academic kerfuffle that makes for diverting reads in those corners of the media where academic kerfuffles get covered. Unfortunately, I’m not at safe distance. Here, as elsewhere, I’ve repurposed psychological findings in philosophical argumentation. Awkward for me, if the findings are false.
Popular reports on RepliGate link the controversy to notorious cases of scientific misconduct (Bartlett 2013, Yong 2012). But the big problem’s not a few cheaters, or even a few more than a few cheaters; if the Replicators are right, the infirmity results from standard practice in experimental psychology. Consider the disciplinary norm for “statistical significance,” a “p-value” of .05 or less. When a finding is reported with this value, it means that if the experiment were run 100 times, and the phenomenon is not real –i.e., if the null hypothesis is true -- the finding would appear “by chance” five times or fewer. The .05 is supposed to warrant confidence that the effect in question is real, but it also entails that “about 5% of the time when researchers test a null hypothesis that is true (i.e., when they look for a difference that does not exist) they will end up with a statistically significance difference” (Pashler and Harris 2012: 531).
This depiction perhaps underestimates the appearance of false positives in the published literature. Some grounds for suspecting this are dauntingly technical, and involve complex models requiring contestable assumptions (Pashler and Harris 2012: 531-2; cf. Ioannidis 2005). Others -- the most prominent being the Publication Bias and File Drawer Effect -- are more easily grasped. The Publication Bias is a pervasive tendency for psychology journals to publish only effects that are novel and statistically significant, with the result that most investigators, unless they strike gold every time, are likely to have a File Drawer full of failures to find significant effects, very often in the immediate vicinity of those effects their publications indicate exist. In the worst case, assuming a significance threshold of .05, a researcher who only investigated phenomena that did not exist could have 95 failed experiments for each 5 successful ones they publish. If all researchers were worst case, every published finding could be a false positive capitalizing on chance, an alarming circumstance concealed in vast File Drawers of failure (Simonsohn 2012: 597).
It’s probably not exactly like this, but neither is it completely unlike this. Experimenters start with a protocol, and if it doesn’t produce the hoped for result, they tweak, and re-tweak, the protocol in order to find their finding. Along the way, some failed experiments that ought be treated as non-occurrences of an effect are instead treated as imperfectly designed pilot studies and interred in File Drawers. No surprise then, if there’s considerable numbers of unpublished failures haunting many published success.
That much might be good faith science; there’s a fine line between refining a protocol and the not-quite-kosher-not-quite-misconduct “questionable research practice” of “selectively reporting” only findings congenial to one’s research program in the papers one submits for publication. But other tricks of the trade, informally known as “p-hacking,” are rather shadier. For example, there’s “optional stopping,” the practice of continually checking results and halting an experiment as soon as p < .05 is reached, which substantially increases the probability that significant findings are a function of chance; during data collection, the p-value fluctuates, and even where there is no effect it will sometimes reach .05. Such practices are likely common: in an anonymous survey of some 2000 psychologists, over 50% admitted selective reporting, and over 20% admitted optional stopping (John et al. 2012: Figure 1).
All this considered, one might predict that many published findings will fail to replicate (capitalizing on chance is a chancy business). Unfortunately, it’s hard to know exactly what’s going on: the Publication Bias militates against the appearance of failed replications, but it also militates against the appearance of successful replications, because journals typically require novel findings (though failed replications may now be getting published more frequently; e.g., Pashler et al. 2013). The 64,000-dollar question concerns what molders in all those File Drawers: how many interesting but non-significant failed replications, and how many significant but boring successful replications?
The Replicators suspect there’s a lot of failure moldering, and if that’s right, tribulation is inevitable, when the published work gets checked. In fact, on a website serving as a repository for replication attempts, psychfiledrawer.org, there seems to be a fair bit of just that. Among the hardest hit are some studies most congenial to the arguments I’m making here, those involving subliminal “priming” of behavior. [1] The classic is an experiment by Bargh and colleagues (1996: 236-7) that found exposure to words invoking elderly stereotypes (e.g., grey, wrinkle, Florida) resulted in healthy young people walking more slowly, without being able to attribute their performance to the “semantic prime.”
The Infirm Words Effect was much celebrated; I celebrated it myself (Doris 2009: 57; Merritt et al 2010: 374), since I thought the existence of such quirky influences on behavior undermined philosophically standard accounts of agency. My merrymaking was truncated, however, for along came a failed replication by Doyen and associates (2012), joining another failure to replicate by Pashler and colleagues (2011) that appeared on psychfiledrawer.org. In response, Bargh (2012) noted two replications of Infirm Words appearing in a top journal (Cesario et al. 2006; Hull et al. 2002), together with two replications on science television (!). Moreover, the effect has appeared in a range of domains: exposure to material associated with stereotypes of the elderly may also slow decision making (Dijksterhuis et al. 2001, Kawakami et al. 2002), decrease reading speed (Loersch and Payne, forthcoming), and weaken memory performance (Dijksterhuis et al. 2000; Levy 1996).
My hunch is that the priming studies have themselves been victims of a prejudice, the Incredulity Bias, which presumes that if a study reports a surprising finding, there must be something fishy. On the other hand, there’s also a Surprising Effect Bias, where editors favor incredible findings, in hopes of garnering attention for their journal. (Less cynically: if scientific findings weren’t surprising, why would we need experiments and publications?) In a perfect world, unlikely findings would be both published and scrutinized -- and maybe that world’s not so far from the world we have. Still, the evidence appears to be badly mixed; can any conclusion – save that we’ve got a mess on our hands – be safely drawn?
Hard to say, partly because the feuding parties feud about what should be counted as successful replication. In a direct replication, the new investigator tries to exactly copy what the original investigator did. Variation in time, place, and resources means the copies won’t be identical (the Pashler et al. attempted replication of Infirm Words is listed as “fairly exact” on psychfiledrawer.org), but if they’re close enough, they may bear quite closely on the question of whether the original finding can be trusted. Conceptual replications, on the other hand, don’t exactly follow the original; they aim to extend the effect, by testing whether the process at issue obtains in other domains. Thus, the extension of Infirm Words from walking to reading might be counted as a conceptual replication, suggesting not only that the original finding was tracking something real, but also that the finding may generalize beyond the original domain.
For the academic psychologist, conceptual replications are more professionally advantageous, since they, unlike direct replications, meet the novelty standard for publication. The problem with conceptual replication is that it ain’t replication; if a study’s different enough from the original to count as novel, its finding could be real while the original finding is not (or vice versa). Moreover, failed conceptual replications may be even more likely to end in File Drawers than failed direct replications. If an investigator fails to conceptually replicate a study with which he is sympathetic, he may be tempted to treat the failure as an injudiciously large deviation from the original study instead of failure to replicate, meaning a (non) finding that ought to raise questions about the original study doesn’t get treated as doing so. At the same time, successful conceptual replications will frequently be deemed publication worthy, so the practice of conceptual replication may be biased in favor of validating previously published effects (Pashler and Harris 2012: 533).
This doesn’t mean conceptual replications say nothing about the reality and generality of an effect; when a cluster of conceptual replications accrues around a finding, it should increase confidence that there’s something real around which the cluster is accruing. Of course, if you’re in sympathy with the Replicators, you might have doubts about many studies comprising the cluster. But something like the Publication Bias is arguably present in all scientific journals, so to require dismissing all published research where the bias is present “leads,” as one psychologist (himself a Replicator) put it, “to the absurd conclusion that all published scientific knowledge should be ignored” (Simonsohn 2012: 598).
Maybe the absurd conclusion isn’t so absurd. In a paper entitled “Why Most Published Research Findings Are False,” Ioannidis (2005) deployed statistical techniques to argue just that: the majority of science is not to be believed. Certainly, the tsuris goes beyond psychology. Scientific medicine and genetics have both been unsettled by controversies akin to RepliGate (e.g., Begley and Ellis 2012), with the debate in genetics involving strikingly similar concerns about direct vs. conceptual replication (e.g., Kang et al. 2011, Munafò et al. 2009, Risch et al. 2009). Does this mean the findings of these fields should be ignored? Absolutely – the minute we have something better. No doubt there’s some trouble in genetics. But this doesn’t mean we’re better off consulting soothsayers about heredity than we are consulting geneticists. Likewise, there’s some trouble in medical research (which has been in a bit of a rut since identifying the importance of good hygiene in controlling infectious disease). But that doesn’t mean we’re better off consulting crystal healers about cancer than we are consulting oncologists.
And so too, there’s some trouble in psychology. But that doesn’t mean we’re better off trusting “common sense” than we are trusting the best available systematic study. There’s good, bad, and indifferent in psychology, just as in all of science, and the existence of the bad and indifferent shouldn’t dissuade us from figuring out what the good is. Doubtless, numbers of scientific findings should be discarded, but if all scientific findings were cast aside, we’d have a lot bigger problems than working out the right account of agency.
When the dust of RepliGate has settled, some currently venerated findings may be less venerated, and some currently commonplace research practices may be less commonplace. But we’re talking about an extensive and variegated field, and it’s not easy to predict what the casualties will be, and what forms retrenchment will take.
(Norms requiring larger sample sizes would make a good start; larger samples reduce random error, because more observations decrease the influence of random fluctuation, thus reducing the margin of error [Fraley and Vazire, forthcoming]). Priming effects range over perception, behavior and motivation, and in each domain, effects vary with both individual differences and situational variation (Loersch and Payne 2011). With this much diversity, indiscriminate statements like “priming studies have been challenged” don’t mean very much. Difficulty with one priming study does not entail difficulty for all studies in its domain, and difficulty with priming studies in one domain does not entail difficulty for all priming studies. Still less does difficulty with priming studies entail that all experimental psychology is in the soup.When facing conditions of scientific uncertainty, caution is in order. This is nothing new: scientific conditions are always conditions of uncertainty, and caution is always in order. (Where certainty obtains, what call is there for science?) There’s no science without a story, and if a large body of perplexing fact is to have theoretical utility, the theorist is forced to make editorial decisions that may require taking sides in scientific controversy. Conscientiously assuming this risk requires identifying the drift of a literature, or better yet, a range of literatures, where we find the strongest trends in results, even as we acknowledge the existence of results, and the possibility of future results, that don’t conform to these trends.
Reservations about the Publication Bias notwithstanding, patterns of conceptual replication are one important way to get the drift. Also important are meta-analyses, which average the results of a large number of studies, and allow more confident assessment of effects than is possible with a single study. For example, a meta-analysis of 167 priming studies (Cameron et al. 2012) found moderate relationships between priming manipulations and both behavioral measures and explicit attitude measures, while a meta-analysis of 65 priming effects in social cognition found effect sizes ranging from “small to medium” to “medium to large” (DeCoster and Claypool 2004: 9). These meta-analyses give reason to think priming effects are real, even as individual priming studies receive unflattering scrutiny.
In any event, the broader lesson remains: don’t lean too heavily on any one study, or one series of studies, in theory construction. All the more so, where there have been difficulties with replication. […]
[1] According to Cameron et al. (2012), “[p]riming involves presenting some stimulus with the aim of activating a particular, idea, category, or feeling and then measuring the effects of the prime on performance in some other task.” Not much in a name, methinks: this description looks to hold for a varietyof experimental manipulations.
on ferguson
i am not black. i don't live in st. louis (anymore). i've never been a cop. i have a lot less insight into the situation in ferguson than a lot of people i know. i hesitate to write a blog post about it - it feels presumptuous to think that i have something to add to the conversation. but it feels worse not to say anything.
some thoughts:
1. despite what a lot of white people think, we have a major racism problem in our country. the ferguson situation is a very stark reminder of this in many ways, but the problem is much bigger and deeper than that. still, it is a good opportunity to think and talk about these issues.
2. in the first few days after michael brown was killed, a lot of news stories mentioned what a good kid he was and that he was about to start college. that irritated me because it implied that being a 'good kid' is relevant to whether one has a right not to get murdered by the police. i don't care if you're lazy, mean, bigoted, smell bad, or work for an evil corporation. you have a right to your own life, at the very least, and to not being harrassed, threatened, and abused by the police. sure enough, when footage of michael brown shoving a store clerk came out, along with other evidence that he was less-than-perfect, people started talking about that. as if it was relevant. john oliver is right, it's irrelevant. if we continue to talk about victims of racism and police violence as if their moral purity is relevant to whether they deserve to be victimized, it reinforces the view that these fundamental rights don't apply to everyone - that some of us deserve life and freedom more than others.
3. the way the police reacted to the protests in ferguson is completely insane. i don't understand how anyone could not be completely appalled by the michael brown shooting itself, but if for some reason you weren't, then maybe the accounts of some cops' behavior during the protests will do the trick* (and this is leaving aside the use of tear gas, rubber bullets, and sonic cannons - which are a violation of our constitutional rights). the fact that police officers would act this way when they know that the whole world is watching makes me realize just how unaccountable some of them feel to the people they are supposed to serve. of course being a cop is a very very difficult job, but i'm not sure that's relevant. we need to find a way to recruit and train cops who can do this difficult job without violating people's basic human rights. there has to be a way. (some good suggestions here).
4. i think it is amazing that people aren't more angry. i don't necessarily think that anger is a productive reaction. i think peaceful protest is probably much better, much more likely to lead to change. but i think it is almost superhuman not to be angry.** imagine watching this happen in your neighborhood in broad daylight ten days after michael brown was killed (watch it with the sound on), and ask yourself how calm and peaceful you would feel. i can't imagine. the question is not why some people are angry, the question is how can anyone remain calm? i am in awe of all the peaceful protesters, activists, etc. i don't think many of us appreciate just how exceptional it is for all of these people to be resisting the urge to get mad, or give up.
i'm not sure what we can do. but i think talking about it, not shying away from facing this very serious problem in our society, is a first step. also, you can donate to the michael brown memorial fund here:
http://www.gofundme.com/justiceformikebrown
also, read this article:
http://www.theatlantic.com/features/archive/2014/05/the-case-for-reparations/361631/
"It is as though we have run up a credit-card bill and, having pledged to charge no more, remain befuddled that the balance does not disappear."
* there were almost not enough words in that clause to link to all the horrendous stories i wanted to link to.
** i am not known for my extreme emotions. some might even call me serene.
wash u students
the simpleminded & the muddleheaded
i have been sitting on this paul meehl gem for a few months now, ruminating on how it relates to our current situation:
"The two opposite errors to which psychologists, especially clinical psychologists, are tempted are the simpleminded and the muddleheaded (as Whitehead and Russell labeled each other in a famous dinner exchange). The simpleminded, due to their hypercriticality and superscientism and their acceptance of a variant of operationalist philosophy of science (that hardly any historian or logician of science has defended unqualifiedly for at least 30 years), tend to have a difficult time discovering anything interesting or exciting about the mind. The muddleheads, per contra, have a tendency to discover a lot of interesting things that are not so. I have never been able, despite my Minnesota “simpleminded” training, to decide between these two evils. At times it has seemed to me that the best solution is sort of like the political one, namely, we wait for clever muddleheads to cook up interesting possibilities and the task of the simpleminded contingent is then to sift the wheat from the chaff. But I do not really believe this, partly because I have become increasingly convinced that you cannot do the right kind of research on an interesting theoretical position if you are too simpleminded to enter into its frame of reference fully (see, e.g., Meehl, 1970b). One hardly knows how to choose between these two methodological sins." *here is what i have come up with (i am trying to fit what probably belongs in several separate blog posts into one because i think the points are interconnected. bear with me.)
1. another way to describe these groups is that the simpleminded are terrified of type I error while the muddleheaded are terrified of type II error. pick your paranoia.2. this seems like a pretty accurate (if caricatured) way to describe the two extremes of the scientific integrity debate. throughout the rest of the post, i am going to use these labels. i apologize to alexa tullett for speaking in dichotomies, and to all simpleminded and muddleheaded people out there for calling you (us) names.3. one thing that has struck me as i've observed discussions is that people at both extremes have incredibly strong intuitions about which is the bigger problem. the simpleminded are absolutely convinced that the literature is littered with false positives and that is the major threat to our field. the muddleheaded are equally certain that the proposed reforms would stifle scientific discovery and lead us to abandon ideas that are in fact correct and would significantly improve our understanding of human behavior. what is really striking is that not only are both sides full of conviction, they are so sure that they can't even believe someone would sincerely have the opposite intuition. i have seen people on both sides accuse those with opposite intuitions of being disingenuous - their own perception of reality seems so blatantly obvious that they think anyone who denies having that intuition is putting them on.**i'll admit, i am pretty far on the simpleminded side of the continuum, and i have sometimes caught myself completely flabbergasted at the vast distance between my own intuitions and others'.this is a terrible situation to be in. when we don't even believe that our colleagues sincerely hold the intuitions they profess to hold, we are at an impasse. when both sides believe that their (conflicting) assumptions are absurdly self-evident, attempts at reasoning with each other are futile.4. so what can we do?first, i think we need to give each other the benefit of the doubt that, at the very least, we all actually believe what we claim to believe, and that these beliefs do in fact feel intuitive to the belief holders. we should stop questioning each other's sincerity.second, we need to find a way out that does not rely on intuitions. happily, we are in the business of not relying on intuitions. we need empirical evidence. this leads us to the question:5. what empirical evidence would convince the simpleminded? what empirical evidence would convince the muddleheaded?this is, to me, the fundamental question that each side has to answer. we should make our beliefs/intuitions falsifiable, by making concrete empirical predictions about what the world would look like if our intuitions are correct.let me start with the simpleminded, since i feel more comfortable speaking for that side of the continuum.for me, a good test of the 'false-positives-are-everywhere' paranoia that characterizes the simpleminded is brian nosek and the open science framework's reproducibility project. this project aims to conduct close replications of published studies in some of the top journals in psychology. with a large enough sample size, this could give us an estimate of the proportion of published studies that are replicable. assuming this estimate has a decent amount of validity, it can provide a test of the simpleminded world view. what results would be consistent with simpleminded intuitions? it probably depends on which simpleminded person you're talking to. this simpleminded person would probably guess that the replication rate will be below 60%.*** call this my preregistration. if that is wrong, i will admit that the situation is not as dire as i thought, that my intuitions were wrong. of course even a 40% (or, for that matter, 20%) false positive rate should be alarming and would, in my view, justify some reforms. but some of the more drastic reforms proposed by the simpleminded rely on the assumption that the problem is Very Serious. so if the replication rate is much higher than 60%, we have to admit we were wrong, at least in the degree of our panic.so, now i come to a question i can't answer: what would the muddleheaded think is a fair test of their intuitions, and what evidence would cause them to reconsider those assumptions?i thought about writing an entirely separate blog post about this question, because it is one i've been wondering about for a while, but instead i will just add a heading and keep going:bricks in the wall vs. the wall itselfa common view i've heard in response to single instances of rigorous, conclusive failed replications is that the particular study that failed to replicate was just one brick in the wall, and the wall (i.e., the evidence for the broader phenomenon) is made up of hundreds of bricks (i.e., studies that show the predicted effect).that seems fair. pulling out a few bricks here and there does not do much to undermine the integrity of the entire wall, if it is made up of lots of bricks. but what if we sample 5% of all the bricks at random, and we find that most of them are faulty. then wouldn't we worry about the wall? it seems harsh to say that you can't test the wall by testing its bricks - there is no wall other than the bricks that make it up.**** so it seems like replication attempts - especially systematic ones like the reproducibility project - would be a good source of evidence about the soundness of the wall. i would be curious to know what replication rates the muddleheaded would consider consistent with their intuitions, and what results might lead them to rethink those intuitions.another approach is to look at statistical summaries of the literature that give us a clue about the degree of bias (e.g., the p-curve). i wonder whether the muddleheaded would consider these tools appropriate tests of their intuitions about the (non)-prevalence of false positives.
of course, the intuition that false positives are not a huge problem is only one part of the muddleheaded worldview. another important part is the belief that the proposed reforms would stifle discovery (i.e., increase type II error). it would be important to do an empirical test of the severity of this problem under various practices/policies as well. how would we go about doing that? i don't know.conclusion: because people at both ends of the continuum have such strong intuitions, i am not sure that talking at each other is going to get us anywhere (but i am still here typing, so obviously i have not lost all hope). we need empirical tests of: a) the prevalence of false positives in the published literature (which can be achieved by things like large-scale, systematic replication projects and by statistical estimates of bias like the p-curve), and b) the consequences of proposed reforms not only for false positive rates, but also for type II error ('misses').
* i left out the next part of the paragraph, which is this: "One thing I can say in favor of the simpleminded is that I have seen several cases of it get cured, by personal experience of psychoanalysis or by exposure to sufficiently bright, rational, and articulate intellects of opposite persuasion, or by just getting older, securer, and more “relaxed.” Simplemindedness is (not being correlated with stupidity among academicians) a curable condition. But I have, alas, never seen a muddlehead get well. I am inclined to believe that this condition has a hopeless prognosis." i left it out because i worry that it is unnecessarily inflammatory, but it's also an observation from one our field's great methodologists, so i am keeping it in the footnotes, as i hear humanists like to do with things they know they should cut but can't bring themselves to.** you know, like what abe thought god was doing.*** full disclosure: i was recently at a conference where some preliminary results from the reproducibility project were shared. i did not attend the presentation (in my defense, it was early in the morning and the chickens kept me up the night before*****). i did, however, hear about it from others. i can't remember what they told me, but it's entirely possible that i unconsciously absorbed this information and it is influencing my estimate. in fact, that seems pretty likely. especially if i end up being right.**** i am leaving aside the mortar for now.***** in their defense, their eggs were delicious.photo credits: rich lucas, me, & the chickens.
self-correction hurts
i know a thing or two about pain. just the other day, i was enjoying* a giants game when a foul ball landed on my knee. (i failed to catch the ball. this guy (above) ended up with it. i do, however, still have the bruise. i would post a picture of that but i don't do selfies.) some things in life just hurt. expecting them not to hurt is like expecting kale to taste good.
self-correction is one of those things. it is never going to be pleasant. whether it is in the domain of science or in the personal domain, self-improvement sucks. if we say that we should only self-correct if we can do it in a way that feels good for everyone, then we are giving up on the idea of self-correction. if we are that delicate, we are doomed.
there are more and less painful ways of correcting ourselves and each other. there are more and less strident ways to talk about replication. i am all for less stridence** and more humility. i think this is very important.
however, i don't think we can do self-correction without anyone's feelings getting hurt. i don't think it's going to feel good even if we all act as polite, humble, and professional as we absolutely ought to. as a field, we have to decide whether we can accept some amount of pain, whether we are willing to pay some price for the benefits of self-correction.
in the comments section of a recent chronicle of higher education piece on repligate, someone going by the name of bob sternberg*** wrote "we need replication but we do not need the acrimony that is going with it." i agree that we don't need some of the more personal and emotional discourse. but i think some amount of friction is unavoidable if we are going to put ourselves to the test. ideally, we will get to the point where the disagreement is all in good fun, and the pain is fleeting, and leaves fewer bruises. like a wiffle ball to the knee, perhaps. but in the meantime, i think we need to walk that very delicate line between being jerks and avoiding challenging each other altogether. and i think most of the time we are doing that quite well.
* it is possible that i was wrong and baseball is actually a real sport. just not as depicted on television.
** i'm not sure this is actually a word. it is, however, the name of my new fragrance.
*** i have no reason to think it was not actually bob sternberg but you never know. someone could be impersonating bob sternberg to make completely innocuous comments on the chronicle of higher ed website. crazier things have happened.****
**** actually, that might be crazier than anything that has happened ever.
another $*%#! blog post about repligate*
because i can't very well have a blog about scientific integrity and not blog about this incident, some thoughts:
1. there are assholes on both 'sides.'** of everything. there are assholes everywhere. there is probably one under your chair right now.2. most people on both sides are not assholes. in this case, none of the main characters are assholes. the record shows that those with significant roles in the special issue (original authors, replicators, editors) have tried very hard to be fair and balanced. this is impressive. i admire all of them for their dedication to the scientific process. they all got more than they bargained for, and we owe them our thanks. thank you.
3. this incident is a kind of projective test***. almost**** any reaction is justifiable - there is a lot to react to.4. i cannot help but see this incident through the lens of self-knowledge. this is an opportunity for self-reflection. it is tempting to go with our knee-jerk reactions, but what is more challenging, and will be more fruitful, is to force ourselves to acknowledge that the 'other side' has valid points. here are what i think are the most important points we on the side of All Things That Are Good and True***** would do well to consider.a. in this case, the original authors should have been guaranteed the chance to write a commentary from the start. i understand the reasons why they weren't, but i think this was a mistake. i am not saying this should always be the case going forward, but it should have been the case here.b. people care about their results. they believe in them and are passionate about the ideas behind them. when we talk about the possibility that those results or ideas are wrong, we should not take this lightly. it is not a small thing to the people who have invested a lot of time and energy in the research. (note: i am not talking about the johnson et al. paper - they did not take things lightly).c. there is currently not much daylight between saying an original finding was a false positive and accusing the original researchers of having p-hacked. in the current climate, they are almost the same thing (though of course they shouldn't be). and even though we all p-hacked******, it is very different to single out one finding for having been p-hacked than to say we all did it. there is no way around singling out individual findings if we are to pursue replications, and eventually we will need to find ways to make this process less personal, but for now, calling false positive on a finding is personal, and it is essentially the same as insinuating p-hacking. (again, not talking about the johnson et al. paper, they did not conclude that the original finding was a false positive).5. these observations lead to some difficult questions that i don't have the answers to:a. should people be passionate about their results? does the fact that we are personally invested in our results make us less scientific? is it plausible to ask people to be passionate about their questions but not their answers?
b. what should we do if we accumulate strong evidence that an original finding was a false positive? is there any way to publicly come to this conclusion without damaging the original authors' reputation?right now it is easy, and correct, to say that false positives are part of life and shouldn't harm anyone's reputation. and of course even if we eliminate all p-hacking, qrps, publication bias, and world hunger, there will still be completely innocent false positives. but as our techniques for detecting p-hacking get more and more sophisticated, and p-hacking becomes less and less forgivable, things are going to get dicier in terms of what counts as a personal attack, what should harm one's reputation, etc. it's naive to think that there are easy, black and white answers to these questions.personally, i favor some kind of amnesty for all papers published up until some date in the near past or future (like, for example, today. 6.3.14 - the day p-hacking became history*******). i don't mean amnesty from replication attempts, but amnesty from blame, or something like that. if we agree not to hold people accountable for normal p-hacking behavior in the past, maybe this will make things less treacherous for authors and replicators. if we agree to hold people accountable going forward, this would require making sure that all new researchers (and old ones) are educated about the new standards.
i also favor putting a lot of energy into de-stigmatizing past less-than-ideal behavior, and encouraging future good behavior. this means working to change the incentive structure. a lot. every day. e.g., every time you review a paper, think about what kinds of practices you are rewarding/punishing. it also means making heroes out of people who engage in adverserial collaborations to seek the truth about their own published work - e.g., this.6. dan gilbert should apologize.
* before this was a meme on twitter, it was a phrase used by john doris in his forthcoming book, which is where i stole it from. there are many other good things to steal there: Doris, J. M. (forthcoming). Talking to Our Selves: Reflection, Skepticism, and Agency. Oxford: Oxford University Press.
** trust me, i'm an expert on assholes.
*** comparing anything to a projective test is usually an insult in my book. i do not mean to demean this incident by comparing it to a form of assessment that mostly lacks any validity. sorry incident.
**** see #6.
***** too soon?
****** i p-hacked. sometimes when i am tired or hungry, i still catch myself p-hacking. this is why i sleep nine hours a day and always keep tostitos in my office.******* also my brother's birthday. navid, i hope you like null findings.
you had me at quintillion
i don't mean to pile on,* but want to share two quick thoughts about the jens förster case.
1.
first, i think the evidence is pretty overwhelming that the data in the paper in question are not real. in addition to the original report and LOWI committee's report, additional analyses by the data colada guys show that the pattern of results could never have happened without data manipulation.
this case is interesting in part because the evidence for fraud comes from statistical analyses of the published results, rather than from a whistleblower inside the lab, or a confession. this makes some people uncomfortable. i agree that concluding fraud based on probabilities could be problematic - this case makes us wonder what we would think if the odds were, say, 1 in 10,000. how small does the probability have to be for us to conclude fraud?
i don't know, but i know 1 in 508 quintillion is improbable enough. i agree it is worth thinking about what we should do with borderline cases, but it is also important to recognize that this is not a borderline case. 1 in 508,000,000,000,000,000,000 is at least 10 more zeros than i need in order to be certain.
2.
another interesting aspect of the förster case is his response. one line in particular jumped out at me:
'[i] never did something even vaguely related to questionable research practices.'
really? i can't go two hours without doing something vaguely related to questionable research practices.
this reminded me of a claim diederik stapel made in his response to the interim report on his fraudulent papers (october 31, 2011 - i can't find the english version online anywhere), in which he stated:
'i must emphasize that the errors i have made were not motivated by self-interest.'
why do these guys insist on completely undermining their credibility? being accused of fraud and responding by saying that, not only did you not commit fraud, but you've never done anything vaguely related to questionable research practices is like being accused of murder and responding by saying that not only did you not kill anyone, but you have never told a lie.
maybe förster isn't guilty. maybe someone else manipulated his data. (sometimes my dog likes to play around on spss.**) but it is fascinating that, when faced with a serious accusation like this, he makes a claim that makes it impossible to trust anything else he says. shouldn't a social psychologist know better?
* it's possible that i do mean to pile on. i'm not sure.
** the reason i suspect it was förster himself is that if an innocent person was presented with this evidence, i think their first reaction would not be defensiveness but 'holy shit. who is f*&$%ing with my data?'
when are two things really one thing?
the question came up a few weeks ago: if two measures are correlated .60, should they be aggregated into a single measure or kept separate? this might seem like a narrow question, but it raises some deep and complicated issues. at the heart of the matter is: when are two measures measuring the same construct?thinking about this reminded me of the jingle and jangle fallacies. i have not read the original papers (the jingle fallacy apparently dates back to edward thorndike, 1904, and the jangle fallacy to truman kelley, 1927. i have not tracked down either book/paper. sorry.) the jingle fallacy is calling two things by the same name that are actually different constructs. the jangle fallacy is using two different names for things that are actually the same construct.let's get specific. let's take the constructs 'neuroticism' and 'negative affect'. are these two the same thing? (maybe, maybe not).one obvious difference is that neuroticism is usually thought of as a stable personality trait, whereas negative affect is typically thought of as a momentary state. but either can be conceptualized either way. in our own studies, we* measure trait neuroticism and trait negative affect, and we also get people to report their state neuroticism and state negative affect several times a day for several weeks (using experience sampling methods). how do we decide whether neuroticism and negative affect are really a single construct (either at the trait or the state level)?i won't get into the specific results (because then we would not be able to publish them in a peer reviewed journal and get lots of fame and glory), but instead i will reflect on the factors that go into making this decision. there are two broad considerations relevant to deciding whether two measures are actually measuring the same construct: the empirical and the theoretical.1. the empirical part: how strongly are they correlated with each other?the strength of the correlation between the two measures is an important clue to whether they are measuring the same construct. but it is tough to know how strong of a correlation counts as strong enough to decide they are not distinct.the question i opened with - if two measures correlate .60 with each other, should they be aggregated? - is not a straightforward one. in fact, .60 is just about at the threshold of my own intuition about where the line is drawn between convergent and discriminant validity (or maybe it's more like .55, or .568). but how to interpret the .60 correlation depends on a few other things. perhaps most importantly, what is the reliability of each measure? if each measure has a reliability of .60, then the fact that they correlate .60 with each other suggests that they are highly similar, because a measure's reliability is, roughly speaking, the theoretical upper limit to how strongly it can correlate with other measures. this is because a measure's reliability is, in essence, its correlation with itself, so it should not correlate with another measure more strongly than it correlates with itself. (but this is in theory only, in reality measures regularly exhibit correlations with other measures that are stronger than their own reliability. partly because reliability estimates can be wrong, and partly because, as charcot said, 'theory is nice, but it doesn't prevent things from existing.') so if the correlation between two measures is high, and close to the reliabilities of the two measures, then that is pretty strong evidence that they are measuring the same construct.there is more to the empirical part. for example, do the two measures show similar patterns of association with other measures? do they covary not just across people, but within-person over time? when combined, do they load onto a single factor? but this is a blog so i am going to ignore those issues.2. the theoretical part: are there important conceptual distinctions between the two constructs?it is possible that, even if all empirical results point to the conclusion that your two measures are indeed getting at the same construct, you are persuaded by strong theoretical reasons to keep them separate. you are probably wrong, but you can try to make the case.i am a dust bowl empiricist at heart. i am inclined to believe the data over anyone's theory. so it will be difficult to convince me that two measures that don't pass the empirical test of distinctiveness are actually measuring distinct constructs.
however, if you have a strong theoretical argument why the two things you are attempting to measure are distinct constructs, i will entertain the possibility that your theory is right and your measures are crap. this is perhaps the best use i can think of for theory - sometimes it helps us figure out that we are using really bad measures.what are some examples of good theoretical reasons to maintain that two things are distinct constructs, even in the face of empirical evidence to the contrary? one is that you believe they arise from different processes. these could be developmental processes, cognitive processes, biological processes, etc. but you still need empirical evidence to back up the process difference, so this is not really a theoretical argument.another is that you can describe scenarios where the two constructs come apart (this is related to the 'different processes' reason - if they can come apart, there must be some difference in the processes that produce them). this is one purpose of thought experiments. for example, if you can paint a compelling picture of a time when a person would experience neuroticism but not negative affect, then you might be able to convince me that they are different constructs, even if, practically speaking, they almost never come apart. in my view, the scenario you depict has to be relatively plausible (philosophers and others may disagree). i don't care if neuroticism and negative affect could come apart on twin earth, i want to know if they could come apart in a situation human beings are liable to encounter here on this actual planet.but in the end, i think you have to back up your theoretical argument with empirical evidence. if you really believe your two things are different constructs, you need to be able to observe them coming apart in actual data. that might mean developing new, better measures, or tracking down rare cases or rare situations, or doing other difficult things, but until you do those things, you are not going to convince me based on theory alone.those are my half baked thoughts. many people have said much smarter things about construct validation. paul meehl, jane loevinger, campbell and fiske, cronbach... and also some people who are currently alive. i'm not sure why you are reading my blog instead of reading them. come to think of it, i'm not sure why i'm writing my blog instead of reading them.*by 'we' i mean my awesome graduate students and our fantastic research assistants.