Enter your Email:
Preview | Powered by FeedBlitz

« power grab: why i'm not that worried about false negatives | Main

Comments

Anonymous

I like your analogy, but i do wonder the following. You write:

"it dawned on me that journals are currently in much the same position as buyers of used cars. the sellers are the authors, and the information asymmetry is all the stuff the author knows that the editor and reviewers do not: what the authors predicted ahead of time, how they collected their data, what the raw data look like, what modifications to the data or analyses were made along the way (e.g., data exclusions, transformations) and why, what analyses or studies were conducted but not reported, etc. without all of this information, reviewers can only evaluate the final polished product, which is similar to a car buyer evaluating a used car based only on its outward appearance."

I wonder if this analogy can be improved. More specifically, if it makes sense to not only view the journal as the buyer, but also the eventual reader of the article. Car dealers (journals) only care about selling the cars (impact factor, etc.), they don't necessarily care about the quality of the car, just that it is being sold. Now, the actual buyers of the car from the dealer (reader of the article) have to actually drive it (cite it, build on it, etc.). They could potentially do care about quality! (and as the reward quality with citations, the journals will start to care).

In other words, to me the most important thing is not that the journal/reviewer can look under the hood, but that the buyer of the car can. Reasoning from there on, i think a possibly more accurate, and useful, analogy would be that the car dealer (journal) buys up cars, which the buyer (reader of the article) then buys from the dealer.

If you only let the car dealer look under the hood, but not the buyer of the car you're still left with the same problem, only on a different (and arguably more important) level.

If this makes any sense, it becomes clear that hiding things like pre-registration from the reader, and only using it on the journal/editor/reviewer level, could be considered to be a sub-optimal practice.

STeamTraen

It's interesting that Anonymous (comment above) interprets the analogy with the journal, not the researcher, in the role of the used car dealer. It would appear that there is scope for discussion here.

I'm reminded of a joke from the IT industry that was being passed round offices 30 years ago in the form of grainy photocopies (that was how office humour circulated back then) that probably dated back many more years:
"What's the difference between a used car salesman [sic, this was the 70s] and a computer salesman? The used car salesman knows when he's lying to you"

Anonymous

"It's interesting that Anonymous (comment above) interprets the analogy with the journal, not the researcher, in the role of the used car dealer. It would appear that there is scope for discussion here."

Perhaps i misinterpreted the analogy. In the piece i copy-pasted from the blog post it looks to me that it is stated that the buyer, who should have access to as much information as possible, is the journal/editor/reviewer, which in my opinion leaves out the most important (other) customer: the reader of the paper (member of the general public, fellow-scientist, etc.). I tried to make this clear by extending the car dealer/buyer- analogy.

Most importantly, and possibly relevant to real-world examples i think it could be very important to make the distinction between journal/reviewer/editor level and reader level, otherwise you are just left with the same problem on a different (and arguably more important) level.

It looks to me that making this distinction is already relevant for a discussion about transparency/good practices at two journals who are supposed to be all in favour of these issues (but please correct me if i am wrong):

1) At "Comprehensive Results in Social Psychology", the pre-registration information seems to not be available to the reader by means of the simple inclusion of a link to this information in the final paper, but this information seems to be handled only on an journal/editorial level.

2) At "Psychological Science" it seems to me that they have a demand at the submission level where you indicate that you confirm that you have disclosed methodological information in the paper, like all independent variables investigated (http://www.psychologicalscience.org/publications/psychological_science/ps-submissions). Now, the reader of the paper again has no direct access to this information. They could have authors include something like the "21 word solution" in the paper to actually make the reader aware of this important information.

If i am correct about this all, i reason that in both cases, transparency/good practices are handled only on the level of the journal/editor/reviewer, and not on the level of the final (reader of the) paper, for no apparent reason. To me, this is sub-optimal at best, and possibly setting a dangerous precedent at worst.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)