My first glance at peer review
Monday, May 21, 2007
I’m still not over my sinus infection and am feeling pretty low. The Seattle trip will probably be postponed. So I’ve been trying to use what time I can think and focus to read. One important thing on my list of priorities is to act as a referee for a journal. This is the first time I’ve been asked to do such a task.
To explain : Peer review is one of the many (one hopes) quality controls we use in science. In short, during at least one step in the process the paper will be reviewed anonymously by other scientists, usually within the same field of study as the paper. The referees are given the option to make various recommendations, raise questions for the authors, and in general provide an expert opinion about the work in the paper. The editors of the journal provide the comments to the authors and give them a chance to respond. Depending upon the severity of any objections, this process can go on for a long time. My own personal record is having a manuscript hung up in this process for about a year. Ultimately the editors have the final decision.
It is by far and away not a perfect system. The objectivity of the reviewer is not certain, the process can delay the publication of results (especially difficult during direct competition to publish) and it can place additional burdens and barriers to the dissemination of good science. In the worst cases, those can become rather extreme. Many scientists in fact consider peer review a rather broken tool. But despite its flaws, we try to make the best of it. There are alternatives proposed which largely rely on open review by a community of scientists or, in some cases, leaving it largely to each scientist to make up his or her own mind.
It’s a rather huge responsibility and one that I take as a rather serious, sombre task. So much so, that I do not wish to reveal the subject or even journal for the paper that is in question before me. My first task as a newly minted referee was to consult with more senior scientists about how to be a referee. Most of them said what I was fairly certain of already. Be honest, take notes, read it at least twice, be timely, look up similar papers, etc... The journal also provides some rather explicit guidelines which I feel as though I’ve memorized.
My first couple of readings of the paper were generally positive, though a little flat. It’s a good subject, something interesting has been demonstrated, though not hugely different from what has been shown in the past. But during the extended process of review I’ve begun coming up with rather harsh questions. One of the most troubling question is whether or not the material is sufficiently different from other work to warrant publication in this journal. That’s especially relevant as the author has published very similar results in other journals previously.
My first reading of the paper was somewhat in a vacuum, but once I’d pulled up the relevant publications that have come before I have some context. So perhaps the best thing I can ask at the moment is for the author to demonstrate in the manuscript what is truly new in this paper, to bring it out, to say something new. That little request (even innocuously phrased) may in fact be asking a great deal.
So I’m faced with the prospect of presenting some rather critical responses to the editors and it’s left me with an uneasy feeling. I’d by far and away prefer one of two papers. It would be easy if it was so brilliant and dazzling (or heck, even just really good with no errors) that my only comment is, “damn that’s good work. I wish I’d of done it!” It would also be easy if the paper was utter trash or nonsense. However, this paper neither. It isn’t trash and won’t win a Nobel prize.
At the same time, I know exactly what it’s like on the other side. It’s a terrible feeling when someone tears up your carefully built hard work. I know exactly how much work can be involved in going from an idea to seeing that idea through to fruition in a journal. So I’ve found myself trying to word my comments in as constructive a way as possible while still presenting them soundly. After all, the authors can disagree with me. If they think I’ve really missed it, they can just go to the editor directly. They’ve also got the option of requesting other referees.
While writing this I’ve been trying to think of the feeling I’ve been having and I just put my finger on it. This is very similar, though perhaps greater, than when I was first a grading homework for students. It was difficult to mark the answers objectively and fairly the first time I found one of my favorite students making mistakes. I knew he had worked hard and tried, but he just didn’t understand. This is a similar feeling though stronger for I know how much more work can go into a journal publication rather than just a homework, lab assignment, or exam. This can be further exacerbated when one realizes that publications are often equated with “success” and funding.
Something old, something new
Sunday, May 20, 2007
I suppose this won’t be very different from my previous web-journals that I keep during experiments. I’ll try to post periodically about interesting news from the lab. If you’re reading this for the first time, this is mainly for family and friends that have asked me what it’s like to be an experimental scientist and physicist. What you can find here: a bit of insight into the day-to-day activities of a scientist, things that interest me, and a bit from my life. Hopefully you’ll find some of it interesting. At some point I’ll try to expand on the choice of name for the blog. Frankly it has nothing to do with the photo I’ve got at the top of the main page.
I’ve decided to merge everything into a single blog and to update it more often than previously. Why? I’m not sure. Perhaps it’s just that I’m sick today (still) and this seems like a fun way to occupy some of my time. Perhaps it’s just a nice thought to have everything in one place. It also gives me the chance to do some updating for the website and make a few stylistic changes.
So I’m supposed to go to Seattle this week, but I’m not sure if that’s going to happen with my head so full viruses and snot. The german word for this is “Schleimkopf” : literally slime-head. And I think I’ve got enough for 10 of them.
I decided to stick with the iWeb software because, well, I’m lazy about it. Right now it works reasonably well for what I want to do and is very easy to use. I wish it had the ability to categorize posts and sort things. I also wish it was easy to edit the “themes” in an easy fashion. But nonetheless it does most everything I desire with minimum fuss.