SODA 2004 - Scoring instructions

ACM-SIAM Symposium on Discrete Algorithms (SODA 2004), January 2004, New Orleans

Scope of the conference (from the call for papers)

Themes and application areas include, but are not limited to, the following topics:

*Combinatorics and other aspects of Discrete Mathematics such as:

Combinatorial Structures
Discrete Optimization
Discrete Probability
Graph Drawing
Graphs and Networks
Mathematical Programming
Number Theory
Algebra, and Random Structures

*other aspects of Computer Science such as:

Communication Networks
Computational Geometry
Computer Graphics and Computer Vision
Computer Systems
Cryptography and Security
Data Compression
Databases and Information Retrieval
Distributed Algorithms
Experimental Algorithmics
Machine Learning
On-line Problems
Quantum Computing
Pattern Matching
Robotics
Symbolic Computation, and the World Wide Web

*and applications in the Sciences and Business such as: Biology, Physics, and Finance.

The program committee encourages submissions from researchers in the discrete mathematics and experimental and applied algorithms communities.
Submissions from the discrete mathematics community may address the design and analysis of algorithms for discrete structures or the development of algorithms as tools for investigating significant open questions in mathematics.

Experimental and applied submissions may deal, for example, with efficient implementation of fundamental algorithms or with heuristics for basic difficult problems. They should provide new and significant insights into algorithmic performance and/or design or discuss the methodology of doing experimental performance analysis. Applied papers should deal with algorithms applied in a specific practical setting and should include convincing evidence that the algorithms or data structures discussed are useful and efficient in the particular context.

Criteria:

When grading a paper, I ask you to address the following (interrelated) issues. A paper with high grade should score high on several of the parameters below.

Relevance

In what respect is the paper relevant to algorithms and discrete mathematics? Is it directly relevant for the design, use, or analysis of algorithms? Or may it have indirect implications for the development or the theory of algorithms?

Foundational/conceptual contribution

Note things like a new model, new notion, new definition, new approach, novel implementation. Note the significance and reasons for this novelty (and note the absence of such a novelty!).

Technical development

Does the paper make an

introduction of a new technique?
novel use of known technique?
talented use of known technique?
traditional use of known technique?
trivial use of technical knowledge?

Relation to open problems:

Does the paper solve completely/partially an open question? How important is this question? (central/important/interesting/legitimate/stupid). How much effort has been invested in solving it and by whom?

Social interest in paper:

Is it potentially interesting to the whole algorithmic community, to a major field (e.g. approximation algorithms), to everyone in a restricted area (e.g. motion planning, the knapsack problem), or interesting only to the authors?

How will it contribute?

fertilization, satisfy curiosity, who knows?

Paper type:

Is it a

first step (opens a new area)?
last step (closes an important area)?
giant step (makes essential progress)?
none of the above?

Is the paper

Is the paper a research contribution?
Is is an experience paper/experimental paper?
Or is it both?

Writing a review

I ask you to evaluate the paper and send me a short written report by e-mail, including a short summary. If you wish, you may also suggest a grade between 0.01 and 10.0 [10=excellent, 1=poor] and a confidence level of your judgement on an scale between 1 and 3, as indicated below. You may also write comments for the authors about mistakes or suggestions that would help them to improve their paper. Please indicate these comments with a line that starts with the words
Comments for author:
The text before this line will be made available only to the program committee, but not to the author. You may also make your evaluation available to the author(s), by placing the "Comments for author:"-line at the very beginning.

Score ratings

One general point. There is not enough room to accept all the papers we might want to. In this case, there will be publishable papers that won't be accepted. Therefore being a competent piece of work may not be sufficient for acceptance. The cut off for acceptance should be viewed as roughly 5.5.

9-10: An enthusiastic yes.  An excellent paper - advances the field in
  an important way - well written and makes it easy to understand what
  the significance of their result is - everyone should definitely
  attend the talk.  This should be among the top 10% of the papers 
  accepted to the conference.  I will fight strongly for this paper.

8-8.99 A strong vote for acceptance.  A solid contribution - I feel I
  learned something worthwhile from this paper - I want to attend
  the talk.  This paper should be in the top third of the papers in
  the conference.

7-7.99 A vote for acceptance.  This will be in the middle third of the
  papers at the conference.  Not a stellar result, but clearly worth
  accepting.

6-6.99: A weak vote for acceptance. A reasonable contribution to 
  an interesting problem - or maybe the contribution is good but 
  the authors don't seem to understand what it is and/or express it 
  well - or maybe it's a good paper, but the subject area is marginal
  for the conference.

5.0-5.99: Ambivalent. I might support accepting this paper,
   but I don't advocate accepting it.  
   Probably publishable as a journal paper in a medium
   journal, but a bit too specialized or too incremental
   for SODA '04.  Or perhaps it has nice ideas but
   is too preliminary, or too poorly written.

4.0-4.99 A competent paper, but not of sufficient interest/depth
   for SODA '04.  A weak to moderate vote for rejection, 
   but I might not object very strongly if others pushed for the paper.

2.5-3.99: A solid vote for rejection.  Too preliminary / badly-written /
   making-such-a-minor-improvement-on-such-an-esoteric-topic for SODA '04.
   It is very unlikely that I could be convinced to support this paper. 

1-2.49  A strong vote for rejection.  
   A poor paper, unsuitable for any journal.
   I will fight to have this paper rejected from the conference.

0.01-.99: Absolute reject.  Completely trivial and/or non-novel and/or 
   incorrect and/or completely out of scope.

Confidence ratings: Range 1-3

I am not an expert. My evaluation is that of an informed outsider. I have some idea of what this paper is about, but I'm not all that confident of my judgment on it.
I am fairly familiar with the area of this paper, and have read the paper closely enough to be reasonably confident of my judgment.
Consider me an "expert" on this paper. I understand it in detail.

Ethical Issues

Submitted papers are confidential. We are not supposed to distribute them, or use them for our research. Similarly, your grades and the deliberations of the committee will be kept confidential. Submissions should be judged solely on the basis of the submitted extended abstract. You may have a personal bias on some papers. The reasons are many - personal/professional ties to authors, your student is just working on the same problem, etc. only you can judge such a bias, and decide if you don't feel comfortable grading the paper.