Explanation of scoring terms

• Average score (of paper X). This is defined to be the weighted average of the scores given to X, where each score is weighted by its associated confidence.

• Normalized score (of member M for paper X ) This provides an alternative way to view the scores assigned by each member. It can be interpreted as a way to adjust scores to account for biases of individual committee members towards high or low scores. In this regard it is flawed since in the case of a particular member such a bias might also reflect that the member read a disproportionate number of good or bad papers, or assigned high confidence disproportionately to good or bad papers.

  For paper X and member M, the adjusted score of X for M is defined as follows. We partition the interval between 0 and 9 into intervals, where the length of the interval for paper X is proportional to C(M,X). The intervals are sorted by score S(M,X). Then the normalized score NS(M,X) is the midpoint of the interval containing all papers whose score equals S(M,X).

  In formulas: Let TC(M) denote the sum over Y of C(M,Y). Let CL(M,X) be the sum of C(M,Y) over all Y such that S(M,Y) is less than S(M,X) and let CE(M,X) be the sum of C(M,Y) over all Y such that S(M,Y)=S(M,X). Then the normalized score NS(M,X) is defined to be  9*[CL(M,X)+CE(M,X)/2]/TC(M).

  Note that the resulting scale makes the normalized score of the papers evenly spaced between 0 and 9, so that the score and the normalized score can't be directly compared (although the rankings with respect to these two scores can).

• Controversy level (of paper X). This is a measure of the level of disagreement in whether to accept or reject a paper. To compute this, we take any score above 5.9 to be a vote to accept the paper, and for score S, define Pos(S) to be S-5.9 if S>5.9 and 0 otherwise. Pos(S) represents the strength of the vote for acceptance. Similarly, take any score below 5.1 to be a vote to reject the paper and for score S, define Neg(S) to be 5.1-S if S<5.1 and 0 otherwise. Neg(S) represents the strength of the vote for rejection. The net positive score for a paper X, PS(X), is the sum over all members M of Pos(S(M,X))C(M,X). Similarly, the net negative vote for paper X, NS(X), is defined to be the sum over M of Neg(S(M,X))C(M,X). The controversy will be determined by the minimum of PS(X) and NS(X). So as not to increase controversy automatically when more people score the paper; this minimum is normalized by the total confidence TC(X) for the paper (i.e., the sum over M of C(M,X) for this paper). and then multiplied by 10. Thus the controversy level is defined to be the minimum of 10*PS(X)/TC(X) and 10*NS(X)/TC(X).