ISCA 2009 Reviewer Guidelines
Luiz André Barroso, Program
Chair
As an ISCA reviewer you are in a
position to
make a
positive impact on the health and vibrancy of our technical community. Here
are some of the most important things to keep in mind as you read a submission
and write your feedback:
A reviewer is a
mentor
Giving authors
valuable feedback on their research is a key part of your service to our
community. That is,
it
is as important as your role in helping us select the best
papers
for the
conference. Write your review as
if you were trying to give a friend-colleague candid, constructive advice
about their
project.
A review is
not about how smart you are. We
know you are smart; that's why we asked you to be a reviewer. A review is
about just two
things: helping the program
committee make sound decisions about the final program and helping the authors
improve their
work.
Think first of the positive aspects of the work, and show
the appropriate praise and enthusiasm for them. When you point out
shortcomings, follow each of them up with constructive suggestions for
improvement. Be critical of the work while being courteous and considerate to
the authors. Remind yourself that the final program will be filled with great
albeit imperfect articles.
Take
confidentiality seriously
You are being entrusted an unpublished manuscript to review. The efforts
around those papers may not have been publicly disclosed yet, and the authors
have not given us any permissions to do so on their behalf. You cannot
distribute any manuscripts you are reviewing to others. If you feel that a
colleague's opinion would be critical to the article, ask the Program
Committee Chairman to invite your colleague as an additional reviewer.
Timeliness =
Respect
Authors have worked very hard to produce their submission,
and they have made their deadline. Nothing shows more disrespect to their
efforts than a late review or a rushed/cryptic one. Our selection process has
historically benefited greatly from the authors' opportunity to rebut reviews
before a final decision is reached. A late review deprives them of that
opportunity. Please plan ahead your time budget for reviews, especially if you
are reviewing more than one submission.
There's no single
formula for a great
paper
One feature of a great program is balance between papers that
make significant contributions to established areas and papers that venture into
truly new territory. Both categories of papers are expected to display
originality and technical soundness. It is generally easier to conduct rigorous
quantitative evaluation methods when working within established areas, so we do
expect to see excellent quantitative evaluations in those papers.
However, if you are excited
about an article which carries a bold new idea, you are allowed to recommend
acceptance even if the evaluation methodology used has some shortcomings.
Authors of those papers are still expected to make their best effort to
objectively assess the value of their ideas.
ISCA should be a "Big
Tent"
Computer Science has grown tremendously in breadth of
disciplines over the past 40 years, and is continuing to evolve. If we choose to
stick to a static and narrow definition of what makes a legitimate ISCA paper,
computer architecture is bound to decrease in relevance as a discipline over
time. Instead, ISCA should be inclusive of work in areas that can broaden
the boundaries of our field, so that it continues to have a great impact on
our industry and on society at large.
Although there will be cases where it will be fair to
question whether a paper belongs in ISCA, think twice before using that argument
to reject a submission. If we think our reviewers will have the expertise to
appreciate the technical merits of a paper and our attendees will enjoy learning
about it, we should consider accepting it.
Further reading:
If you haven't reviewed research papers in a while, I'd recommend
reading Alan Jay
Smith's The
Task of the Referee (IEEE Computer, 4/90). Mark Hill has
also written a great set
of guidelines
for reviewers of ISCA'05. Thanks to Mark, Kathryn McKinley, Steve Keckler
and Brad Calder for letting me steal material from their guidelines.