who will watch the watchers?

 

[DISCLAIMER: The opinions expressed in my posts are personal opinions, and they do not reflect the editorial policy of Social Psychological and Personality Science or its sponsoring associations, which are responsible for setting editorial policy for the journal.] 

 

he makes it look easy.

 

i wasn't going to write a blog post about susan fiske's column.  many others have already raised excellent points about the earlier draft of her column, and about the tone discussion more generally.  but i have two points to make, and poor self-control.

 

point #1.

 

i have a complicated relationship with the tone issue.  on one hand, i hate the "if you can't stand the heat, get out of the kitchen" attitude.  being able to stand the heat is not a personal accomplishment.  it's often a consequence of privilege.  those who have been burnt many times before (by disadvantage, silencing, etc.) are less likely to want to hang out in a place with a ton of heat.  and we need to make our field welcoming to those people who refuse to tolerate bullshit.  we need more people like that, not fewer.

 

on the other hand, i don't think the self-appointed data police are the most egregious source of bullshit.  the vast majority of methodological freedom fighters* are doing something good for the field.  they are literally doing the things i teach my undergrad research methods students to do.  they are reading papers, critiquing the methods, double-checking the stats, evaluating the conclusions.  if more people want to volunteer their time to check our work, we should welcome them. even if they are robots.** i'd rather spend my energy trying to fix the other sources of bullshit that i think are much more pernicious - mainly the fact that many early career researchers receive inadequate support and are often taken advantage of in various ways.  we don't treat our young well, and we need to do better.***  this is not new to this era, and it doesn't get enough attention.

 

so, point #1: let's turn down the heat in the kitchen so that everyone feels welcome, but the source of the most heat is not the data police.  it's the same source that has been creating heat since the beginning of mankind: status/power/privilege/entitlement.  

 

point #2.

 

fiske's column places a great deal of emphasis on the importance of gatekeepers.  she criticizes some forms of communication for being "uncurated" and "unmoderated."  instead, she advocates for critics to make "their rebuttals and letters-to-the-editor subject to editorial oversight and peer review."  she praises changes made through APS because "APS innovates via expert consensus" and this way research is "judged through monitored channels."  she prefers these channels because they "offer continuing education, open discussion, and quality control."

 

full disclosure: i am one of those gatekeepers.****  i am also a board member of APS, and proud of some of the things the organization has done.  but i find this line of argument extremely problematic.*****  

 

the "expert consensus" and "quality control" that fiske trusts is a big part of what got us into this mess.  i think we can forgive our colleagues if they no longer feel that the gatekeepers are always wise arbiters of quality.

 

moreover, if a reader wants to critique a published paper, especially if she believes she has found a major flaw in the paper, she is implicitly critiquing the journal it was published in and it's expert judgment.  it seems unreasonable to me to require that she only be allowed to express that criticism if it meets with the approval of the journal's editor (and reviewers, who are likely to include the author whose work is being critiqued).

 

more generally, gatekeepers need to be held accountable, and open to criticism.  fiske seems to want to grant them more power and control over what views are allowed to be expressed.  in contrast, i would like there to be more avenues for people to express criticisms of the system.  what fiske calls "quality control" i would call a system that is responsive to some perverse incentives and that is designed to protect itself and its reputation.  

 

the healthiest thing we can do for the peer review system is to give its critics a voice.  we should expose the process and its flaws to the sunlight, including through uncensored post-publication peer review.  journals, editors, and societies should welcome open discussion of the current system, and post-publication critiques of their output.  we bemoan how hard it is to find reviewers before publication, and then smear the people who donate their time to post-publication review.  if we care about the quality of our published record, we should appreciate both kinds of feedback.

 

a while back on facebook, someone jokingly suggested a ratemyeditor.com website.  it wasn't a serious proposal, but it reflects a real concern: what are researchers supposed to do when they want to critique the system?  who will watch the watchers?  rather than concentrate even more power in the hands of the gatekeepers, i say let's open the doors to more voices.

 

* credit for this phrase goes to sanjay srivastava. 

 

** i'm not gonna name names, but a certain someone does walk a lot like a robot.  not saying who, just saying, would a non-robot come up with the phrase "methodological freedom fighter"?

*** i was lucky to have excellent support from my advisor and many others as an early career researcher, so i am mostly not speaking from personal experience.  except for that time a famous social psychologist walked up to me and grabbed my ass at my first SPSP.  true story.

 

**** in case it's not already super obvious, the views i express here do not necessarily reflect the views of any journal or society i am affiliated with. 

 

*****  

 

 

Comments

[Roger Giner-Sorolla]

Thank you Simine! I would add that if Susan's concerns are really about tone, a compromise would be to have an open peer review forum that is moderated only for tone.

But probably there are further lurking issues.

For example, having a file drawer of failed studies without an account for why they failed, not keeping careful records and data sets, not double checking statistics etc., all help you get ahead in publishing but are not really defensible as science. Being asked for these things by individuals where there is no general requirement to provide them in publishing may be seen as unfair. Conversely, if standards are not seen as changing fast enough, this spurs "vigilantes" to action.

[Samuel Kazungu Muramba]

Good article! Thanks for informing us.