I’m starting to think about some of the talks I’ll be giving whilst on my travels later in the year. Questions relating to peer review inevitably crop up — about double-blind peer review, open peer review, or why the vast majority of journals don’t publish referee reports, and so on… Without boring you senseless with a long list of questions, below are three that sprang to mind earlier today. I realise this is hardly a controlled experiment and the sample size might not be huge — but I’m interested to see how the vote goes. I’ll leave the polls open for a little while and do a wrap-up post when I get the chance. If you have anything interesting to say about peer review — or any questions — then please comment!
A couple of notes:
1. These questions aren’t related to any impending change of policy at my day job — or indeed to how I do my day job. I’m just genuinely curious to see how the votes go.
2. If I never have to insert another poll into a WordPress post, it will be too soon. Talk about infuriating.
Chemical connections 
I now only referee for Open publications.
I’m not convinced by the arguments for non-anonymous refereeing. I think there would be a tendency for powerful/influential people to receive better referee reports for their work, even where this isn’t merited. Also there are some people in academia (we can all think of a few examples, I’m sure) who would just abuse the system and act vengefully against those who gave their work a negative report.
Stu, just an FYI: these polls are unreadable on my iPhone, but look fine in FFX.
Regarding question 1, for a non-anonymous system to work, I think that you’d have to make it mandatory instead of voluntary. The “volunteer” aspect enters in that you can elect not to submit your work to a particular journal or to referee for it. I think this is similar to how Warren Buffett says that the “top 1%” should pay higher taxes, but he refuses to cut a check unless everyone in his class does so.
I still stand by many of the ideas listed in my “If I Were the Editor of JACS…” post from 2007, especially when it comes to refereeing. If the referee reports and their authors were published, then people would be forced to be fair. If a referee were too soft on papers from “famous” chemists, then she’d get called out for it just as if she were too harsh on anyone.
That said, with Nature Chemistry now in the mix, I think that makes it more difficult for any one journal (e.g., JACS) to “go it alone” in terms of overhauling the present system. I bet a lot of people are too chicken to stand by their opinions, and correspondingly, would avoid submitting to—or refereeing for—the journal.
Also worth mentioning is that my stance on open access has softened considerably since 2007.
Hmm, not sure what I can do (if anything) with the polls – didn’t occur to check from my phone… WordPress and polls are not a good mix in my (incredibly limited) experience.
Your 2007 post is on my radar (again) — need to check it out once more, especially in light of the fact I’m going to be giving a talk that you’ll be at…!
The question of publishing referee reports is not straightforward, and not one I have the time to go into here. It’s not just a simple case of publishing them (from an editorial perspective, that is) — but that’s another post for another day.
And yes, any kind of development in the peer-review process that deviates significantly from the traditional approach would be a potentially risky step for any top-level journal. You don’t want to alienate the majority of your community when there are good alternatives around. That’s not to say the peer-review system can’t be improved (although I’m not sure I’ve seen a credible proposal yet that the majority of the chemistry community would buy into).
I’d be honored to have my old post ripped to shreds in front of my colleagues!
On the last question about whether the report would be “different” – it depends what you mean by different. Mine might be more polite (i.e. differently worded, fewer insults) if the authors knew who I was, but the substance would not change, nor my rating.
Enormous congratulations on inserting polls into the blog. I’m impressed.
My thoughts exactly,
you lying piece of putrid scatBarneyI deliberately kept the wording vague in the last question… I’m guessing that the substance of the vast majority of reports would not change if they were signed (I hope that would be the case), but that the tone might change. So I just left it as ‘different’ and am letting people interpret that as they see fit – and it is generating discussion too!
Several commentators here seem to possibly conflate open-access with non-anonymous peer review. Not at all the same: the biggest open-access journal in my field (philosophy) has anonymous review.
As a woman in a field that has a lot of gender imbalance, anonymous review functions for me in two crucial ways: it allows my work to be judged without knowledge of gender (and many of my referee reports have said “the author .. he”) and, perhaps more importantly, it shows people that my work is not accepted because of favoritism toward women — that is, it’s clear I was evaluated on a level playing field and not given preferential treatment for being female.
As a referee you’re not usually expert in every aspect of the paper you’re reviewing, and don’t have time to look everything up. So if you can’t be anonymous, you might choose to omit comments for fear of looking like an idiot.
Is not double-blind review the only good solution? Would this not eliminate all gender imbalance, if it exists in a certain field… Also, having worked as a biochemist at a Physics department I experienced that my publications where reviewed more critically as if I would have submitted the same from a Biochemistry Institute address. And, being last author more and more often it would just be fair to make the judgement just a little bit less dependent on the name and reputation of the last author…
The experiment of double-blind peer-review is there to be done, but in reality I wonder how practical it is. The answer could be that it is, but I do wonder. For example, a lot of introductions to papers have sentences along the lines of ‘We recently reported (ref. 1)…’ – and so we would need to get authors to ensure that the paper wasn’t written this way. I also wonder whether referees would be able to guess who wrote the paper anyway. They might not be 100% sure, but chemistry has very specific (and somewhat) niche areas and the identities of the authors might be obvious in a lot of cases. I also don’t know of any journals that use double-blind peer-review – but that does not mean there are not any…
These are interesting questions, though as some have already mentioned, more complex than they may seem. Q1 raises the interesting point of when you would say yes or no – fear of reprisal, or worry of developing a negative reputation I think would play a significant part. When submitting a negative review I believe people would be more likely to withhold their details. This reasoning may also be a factor in Q2 – should you read a paper and think it unworthy of publication – rather than associating your name with it, you may just send it back ‘unreviewed’. Q3 – As mentioned above, I think tone is the key factor that would change, and this may be a positive thing. More constructive criticism, especially for those early in their careers would be most useful (rather than just requesting citations for themselves!).
I think an interesting option would be to publish names of referees on accepted papers (possibly without the report), but allow referees to remain anonymous should they turn down a paper. This would allow an individual to be critical without fear of reprisal, and would ensure individuals refereed fairly when accepting articles. There are holes in this I know, but I think an interesting option.
Also, from a young academics perspective (of which I am not), submitting anonymously may be a big help. As Patricia says, being reviewed on a level playing field is not always easy to come by, and breaking that (imaginary?) barrier into JACS and Angewandte may be easier, should people not know you are new to the game. I would omit ‘we recently published’ with pleasure if I could submit anonymously.
Pingback: Altmetrics, Traditional Metrics, and more - Weekly Twitter Activity 2012-03-16 | Michael Habib | Nudging Serendipity
Pingback: I’m still here | Chemical connections
Maybe it makes sense to have a mix of anonymous / non-anonymous review. For instance reviewers could be named (maybe with some statistics on their score) on the journal issue, or on the journal (per year).
Both systems have their pros and cons.
Anonymous reviews can be less biased, but can also be more superficial.
Non-anonymous reviews are problematic if somebody knows the authors in some form, or if there could be some conflict of interest. For small communities, this is a real problem.
There is also another fact to consider: while it is true and good that reviewing is a commitment… if people had to have their review public, these would require more time. Will there be enough people able to handle the amount of papers produced ?
An anonymous review system compensate this a bit, as if a reviewer can provide a a review which is not completely sound, the editor can take this into account when weighting the whole reviews set.