status quoi? part iii

i wanted to write a series of blog posts featuring a few of the people i've met who are challenging the conventional wisdom and inspiring others.  but instead of telling you what i think of them, i wanted to give them a chance to share their insights in their own words.  i contacted a few early career researchers i've had the chance to get to know who have impressed me, and who are not affiliated with me or my lab (though i am collaborating on group projects with some of them).  there are many more role models than those featured here, and i encourage you to join me in amplifying them and their messages, however you can.

i asked each of these people "what are the blind spots in your field - what issues should we be tackling that we aren't paying enough attention to?"  here are their answers, in three parts, which i will post in three separate blog posts this week.

find part i of the series here (with a longer introduction)

find part ii of the series here

 

Part 3: Jackie Thompson, Joe Hilgard, Sophia Crüwell

Jackie Thompson

To me, failures of communication are the biggest blind spot in science. 
One aspect, notably, is science's focus on an outdated publishing model designed for an age when communication was slow, and relied on isolated scientists disseminating their findings on "sheets of pulped-up tree guts" (https://thehardestscience.com/2019/07/). We need to focus on new, more flexible ways to envision sharing of scientific inquiries-- for instance, making academic papers interactive and dynamic, or doing away with the idea of a "paper" altogether, and instead publishing individual elements of the research process (see the Octopus model by Alex Freeman; https://t.co/BPCarBGhZZ)

Another massive blindspot (almost literally) is a kind of self-focused myopia -- we spend loads of energy trying to reinvent the wheel, not communicating between fields and sub-fields, when some have already solved problems that other fields struggle massively with. (How many decades were preprints popular in physics before they started catching on in other fields?) Psychology put a name to the fundamental attribution error, but as a field we still fall prey to it every day. Many psychologists (myself included) scoff when we see non-academic ventures that fail to implement basic rules of good experimental practice -- e.g., businesses that send out poorly written surveys, or government departments that claim to show their interventions worked, despite not including any control groups. Yet, we turn around and tell our scientists to run their labs without any training in management; we try to change cultures and incentives without calling on any knowledge from sociology or economics; we try to map the future of science without delving into the history of science. We have so much expertise and wisdom at our fingertips from our colleagues just beyond the fences of other fields, yet we don't think to look beyond the fences for help from our neighbors; we happily restrict ourselves to the gardening tools we already have in our own backyards. Call this myopia, call it hypocrisy -- whatever the label, it's clear that this mindset results in a huge waste of time and effort. Interdisciplinary collaborations are incredibly valuable, but not valued (at least not by the insular academic ingroups they try to bridge.) The academic community needs to embrace the value of collaboration, and learn the humility to ask "who might know more about this than I do?"

 
Joe Hilgard

I study aggression, and I feel like my area is five years behind everybody else regarding the replication crisis. In 2011, psychology realized that its Type I error rate was not actually 5%, that several p values between .025 and .050 is a bad sign, and that we shouldn't see 95% of papers reporting statistically-significant results. Yet when I read aggression studies, I tend to see a lot of statistical significance, often with just-significant p-values. I worry sometimes that some of our research doesn't ask "is my hypothesis true?" but rather "how hard do we have to put our thumb on the scales to get the anticipated result?".

While we're still catching up with the last crisis, I think the next crisis will be measurement. We know very little about the reliability and validity of many popular measures in experiments about aggression. We've assumed our measurements are good because, in the past, we've usually gotten the statistical significance we expected -- maybe due to our elevated Type I error rates. Research from other fields indicates that the reliability of task measures is much too poor for between-subjects work. I think we've assumed a lot of our nomological network, and when we test those assumptions I don't think we'll like what we find.


Sophia Crüwell

I think we need to stop pushing the boundaries of our collective and individual ideals. We also need to stop thinking and arguing that some deviation from those ideals is acceptable or even good in certain circumstances, such as getting a career advantage – whether for ourselves or for others. Treating our ideals as optional in this way is the beginning of losing sight of why we (are being paid to) do research in the first place: to get closer to the truth and/or to improve people's lives. This goes for any scientist, really, but metascientists are in a particularly privileged position here: at least we know that potential future academic employers will be more likely to value e.g. openness and rigour over simple publication count. I believe that we have a responsibility to use this privilege and change the conversation with every decision and in all criticisms we can reasonably make.

However, we also really need to look into incentive structures and the lived experiences of actual scientists to figure out the specific underlying flaws in the system that we have to address in order to make any of the fabulous symptomatic solutions (e.g. preregistration, data sharing, publishing open access) worth each scientist's while. Sticking to your scientific and ethical ideals is incredibly difficult if it means having to fear for your livelihood. But just waving at "the incentives" and carrying on cannot be the solution either – we need to investigate the problems we have to deal with, and we need to try to deal with them.

Therefore, my appeals (to senior people in particular) are: please stick to your ideals to make it easier for everyone else to stick to them too. And if you are in a position to materially make it easier for people to stick to our ideals, please dare to make those decisions and have those conflicts.