Overview

This document aims to help authors, reviewers, and Papers Chairs understand the double-blind review process that the SC Conference Series has adopted. Please contact us with any questions or comments.

 

Rationale

Papers submitted to the SC Conference will undergo a double-blind review process. In this process, authors do not know who reviews their papers and reviewers do not see author names.

The primary rationale for double-blind review is to mitigate implicit or explicit bias, as there is ample evidence that double-blind policies can reduce such bias. There is also evidence that papers subjected to double-blind review receive more citations than those that undergo single-blind review; thus double-blind review may also be associated with higher-quality papers.

Please see the references at the end of this document for additional discussion of double-blind review. Those of McKinley (2008; updated in 2015) and Snodgrass (2007) are particularly lucid; McKinley reminds us that “Double-blind is not perfect, just better,” while Snodgrass advises “… uncertainty as to the authors’ identity is often sufficient to realize most or all of the benefits of masking.” In other words, the goal is not “perfect” blinding, but rather enough uncertainty about authorship in the process to reduce actual or perceived bias.

 

Guidance for Authors

If you are an author, you should write your paper so as not to disclose your identity or the identities of your co-authors. The following guidelines are best practices for “blinding” a submission in a way that should not weaken it or the presentation of its ideas. These guidelines are broken up into the major submission and review phases: while writing (before submitting), at submission time, during the review period, and during the rebuttal process.

These practices were distilled from McKinley (2015) and Snodgrass (2007).

If you have additional questions or comments, please contact the Papers Chair.

 

While Writing

• Do not use your name or your co-authors’ names, affiliations, funding sources, or acknowledgments in the heading or body of the document.
• Do not eliminate self-references to your published work that are relevant and essential to a proper review of your paper solely in an attempt to anonymize your submission. Instead, write self-references in the third person. Recall that the goal and spirit of double-blind review is to create uncertainty about authorship, which is sufficient to realize most of its benefits.
• To reference your unpublished work, use anonymous citations. From Snodgrass (2007): “The authors developed … [1]” where the reference [1] appears as, “[1] Anonymous (omitted due to double-blind review).” You will have a chance to explain these references to the non-conflicted Papers Chair or their designee(s); see At Submission Time, below.

 

At Submission Time

• At submission time, you will be asked to declare conflicts of interest you may have with program committee members. You will also have the option to upload a list of conflicts. Reviewers will be asked separately to verify declared conflicts.
• Suppose you feel that there is supplemental material essential to reviewing your submission but which would also reveal your identity (e.g., an earlier technical report or published software). In this case, you will have the option to upload or link to those materials and include an explanation at the time of your submission. By default, reviewers will not see this material; instead, the non-conflicted Papers Chairs or their designee(s) will review and use their discretion to decide whether to include such materials during the review.
• Because of the double-blind process, there will no longer be a limit on the number of submissions by reviewers. (Area Chairs are subject to limits.) However, there will be a limit of four accepted papers for reviewers.

 

During the Review Period

• You are not forbidden from disseminating your work via talks or technical reports. However, you should not try to directly or otherwise unduly influence program committee members who may be reviewing your paper.

 

During the Rebuttal Period

• During the rebuttal period, authors should still assume double-blind review. Therefore, authors should not disclose their identities in their rebuttal to the reviewers. However, as with the original submission, authors will have the option of entering identity-revealing information in a separate part of the rebuttal form that will, by default, be visible only to a non-conflicted Chairs, or their designee(s) in the case of conflicts.

 

Guidance for Papers Chairs and Reviewers

The following is a set of guidelines for the Papers Chair or Area Chair (hereafter, “Chair”) and reviewers (i.e., Papers Committee members). Generally speaking, the procedures draw inspiration from the three principles suggested by Snodgrass:

“The first is that authors should not be required to go to great lengths to blind their submissions. The second is that comprehensiveness of the review trumps blinding efficacy. The final principle … is that [editors and chairs] retain flexibility and authority in managing the reviewing process.”

 

Before the Submissions Deadline

• Correctly identifying conflicts of interest (COIs) is one of the most important procedural aspects of double-blind review. Therefore, before the Papers submissions deadline, Chairs and reviewers should log into the review system to verify and upload their conflicts of interest. This process can be time-consuming, so please plan accordingly.
• During paper bidding, reviewers should let their Chair know if they suspect a conflict with a submission and what they believe is the nature of the conflict.

 

During the Review Period

• A reviewer may accidentally discover the identities of the authors during the review. (For instance, he or she might be checking references to determine the novelty of the submission and discover a technical report with the same content.) In this case, the reviewer should disclose this discovery to the Papers Chair. Such incidents do not necessarily “violate” the double-blind policy, and the reviewer may continue to review the paper. The spirit of double-blind reviewing is that reviewers should not actively try to discover who the authors of a submission are.
• A reviewer who thinks he or she knows an author’s identity should not reveal his or her suspicion in his or her review or during discussions with other reviewers (whether online or in-person).
• SC Papers follows a “double-blind until accept” procedure. That is, author identities remain hidden until the review committee has determined all of the accepted papers. For rejected papers, author identities remain hidden even after rejection.
• Reviewers who feel that knowing an author’s name or affiliations is necessary to review a submission can make their case to the Papers Chair at any time during the review process.
• Reviewers who wish to ask colleagues to help with reviews must clear these requests with the Chair first and take steps to ensure that the colleague understands the double-blind policy. In any case, a reviewer is responsible for representing his or her reviews fully.

 

During the Program Committee Meeting

• Chairs should still observe and manage conflicts as they would in a single-blind review. For instance, they should avoid discussing a paper until all of the paper’s conflicted reviewers have left the room.

 

Conflicts of Interest

A potential conflict of interest (COI) occurs when a person makes a decision that:

1. Could result in that person, a close associate of that person, or that person’s company or institution receiving significant financial gain, such as a contract or grant; or
2. Could result in that person, or a close associate of that person, receiving significant professional recognition, such as an award or the selection of a paper, work, exhibit, or another type of submitted presentation.

The Papers Committee will have a chance to disclose potential conflicts during the review process. Chairs will make every effort to avoid assignments that have a potential COI.

According to the SC Conference you have a conflict of interest with:

• Your PhD advisors, post-doctoral advisors, PhD students, and post-doctoral advisees forever;
• Family relations by blood or marriage, or equivalent (e.g., a partner);
• People with whom you collaborated in the past five years. Collaborators include: co-authors on an accepted/rejected/pending research paper; co-PIs on an accepted/pending grant; those who fund your research; researchers whom you fund; or researchers with whom you are actively collaborating;
• Close personal friends or others with whom you believe a conflict of interest exists;
• People who were employed by, or a student at, your primary institution(s) in the past five years, or people who are active candidates for employment at your primary institution(s).

Note that “service” collaborations, such as writing a DOE, NSF, or DARPA report, or serving on a program committee, or serving on the editorial board of a journal do not inherently create a COI..

Other situations can create COIs, and you should contact the Papers Chair for questions or clarification on any of these issues.

 

Frequently Asked Questions

 

Q: How strictly will the new double-blind policy be enforced?

A: Authors should consider the above as a set of guidelines, rather than as a set of strict rules the violation of which results in immediate rejection. The Papers Committee will have enough discretion to apply the double-blind policies in a reasonable way.

 

Q: My work builds on a system that I previously developed. How can I cite this work?

A: Follow the guidelines on the use of third-person self-references, as described above While Writing. Following the example of Snodgrass (2007), it would not be untrue even to write in the text of your paper that the developers of the system upon which you are building made this system available to you. Recall that the goal is to maintain uncertainty about your identity.

 

Q: I strongly object to double-blind review. What should I do?

A: Read the references below. If you remain unconvinced, you may vent to us with the subject line tag, “#ConscientiousObjector.”

 

Q: Doesn’t double-blind review hurt prolific or famous authors?

A: The empirical evidence that double-blind review affects the acceptances of such authors is inconclusive; see the references below for details.

 

References

Papers Frequently Asked Questions, 2016 http://sc16.supercomputing.org/submitters/technical-papers/papers-faq/.

McKinley, Kathryn S., “More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings,” 2015, http://www.cs.utexas.edu/users/mckinley/notes/blind-revised-2015.html.

Snodgrass, Richard T., “Editorial: Single-Versus Double-Blind Reviewing,” 2007, http://history.acm.org/pictures/tods/tods.acm.org/editorial.pdf.

Laband, D.N. and M. J. Piette, “A citation analysis of blinded peer review,” The Journal of the American Medical Association (JAMA): The Second International Congress on Peer Review in Biomedical Publication, 272(2) (1994):147–149, http://www.ncbi.nlm.nih.gov/pubmed/8015128.

 

Acknowledgements

Rich Vuduc and Lois Curfman McInnes prepared the initial version of this document at the request of the SC Steering Committee. They thank John West, Lori Diachin, Jeffrey Vetter, Mattan Erez, Michela Taufer, Rajeev Thakur, Wu Feng, and D.K. Panda, and the SC16 Area Chairs, among others, for their helpful comments, feedback, and pointers.