---------------------------------------------------------------------- How to write a review (for the COMSOC course) Ulle Endriss, 16/11/2011 ---------------------------------------------------------------------- For the course, you have been asked to review two short drafts of the papers your fellow students are working on. This task is similar to (but not quite the same as) reviewing a paper for a journal or a selective conference. Below I make a few suggestions on how to approach the task of writing a review. Most of them are not specifically tailored to the particular task of reviewing for the course. Why do people write reviews? The first reason is to help selecting good and formally correct work for publication, so that others in the field can focus their attention on these (hopefully) high-quality papers. The second reason is to help the authors of the paper under review to improve their work before it gets published. While you should be critical, the tone of your review must always remain polite and professional. Your goal should be to help the author to improve the paper as much as possible. ---------------------------------------------------------------------- HOW TO START ---------------------------------------------------------------------- It is usually a good idea to start your review with a paragraph summarising the main contributions of the paper under review. The main function of this is to show to the author (as well as to the person who has commissioned your review and possibly your fellow reviewers) that you have actually read the paper and made a sincere attempt at understanding it. You may not always agree with the author what the main contributions are and how they should be interpreted, and making this point clear is a second important role of this introductory paragraph. It is important to use your own words when giving this summary (otherwise the summary cannot fulfill either one of these two functions). You should definitely not just copy a few key sentences from the introduction written by the author (that's just a waste of virtual paper). ---------------------------------------------------------------------- REVIEW CRITERIA ---------------------------------------------------------------------- It is common practice to assess a paper under review according to the following criteria: * Relevance * Significance * Originality * Technical Soundness * Scholarship * Clarity * Presentation These criteria are explained in the "how to write a paper" pamphlet on the course website. For the specific task of reviewing the short drafts of your fellow students, I consider the criteria of clarity and presentation particularly important. Technical soundness may also play an important role: this is not just about the correctness of theorems and their proofs, but also, for instance, about the mathematical precision of definitions. When assessing the merits of the paper, you are not expected to do the work of the author for them. For example, if a proof appears unnecessarily difficult, then you should say so and clearly state that you have not been able to verify the result. More generally, if a particular result is claimed, but insufficient details are given for the reviewer to verify the claims, then this reflects poorly on the technical soundness of the paper. Similarly, if the author does not make it clear /why/ their result should be considered important, then you should feel free to evaluate the paper a scoring low with respect to significance. Alternatively, if you do happen to understand why the result is significant, but the author was not able to communicate this well in their paper, then you should help them an suggest appropriate revisions. Originality almost certainly won't play any role when you are reviewing the short drafts of your fellow students, but in general it is one of the central criteria. Relevance (in this case to the topics of the course) is also unlikely to be an issue here. Scholarship (i.e., whether the author clearly identifies the source of any of the material included) is important, even for the short drafts. In particular, as a reviewer who is unlikely to know the relevant literature well, it is the job of the author to be entirely unambiguous about all sources. If it is /unclear/ whether all the relevant sources have been properly cited, then this is just as bad as if some of them have actually been omitted. When reviewing a full paper and you believe that a result claimed to be new by the author actually is not new, then it is crucial that you give full references to that result yourself in your review. You could structure your review in terms of the review criteria, although it will often be a better approach to structure the comments you wish to make in terms of their importance: Begin with criticism aimed at the overall approach taken, then discuss specific technical points, and conclude with remarks on presentation quality and (possibly) a (short) list of typos and similar minor issues. ---------------------------------------------------------------------- FOR FUTURE REFERENCE ---------------------------------------------------------------------- While not relevant to this particular exercise, let me also add a few general remarks on reviewing for journals and conferences. If the paper ends up being rejected, then your comments should help the authors to be able to submit a significantly better paper next time; if the paper ends up being accepted, then your comments should help the authors to publish the best possible version of their originally submitted paper. An additional goal is to help the editor or programme committee chair to accept the best papers and/or to ensure that only papers that meet the standards of that publication venue are accepted. But keep in mind that you are not making these decisions yourself. Therefore, as a reviewer, you should never include an explicit sentence saying "this paper should be accepted" or "this paper should be rejected" (of course, you can, and usually are asked to make this kind of statement confidentially). Reviewing for conferences and journals is usually blind (and often double-blind). That means, an author does not know the identity of the reviewer (and for double-blind reviewing the reviewer does not know the identity of the author either). You should respect this convention and not reveal your identity to an author whose paper you have reviewed. These conventions are there to ensure that reviewers are free from social pressures when giving their assessment, which ultimately serves the academic integrity of the field and is vital for scientific progress. For this particular exercise things are not quite that serious (but I will still not reveal your identity as a reviewer to the students whose drafts you have reviewed). Finally: Every now and then you will hear people say that reviewing is a thankless task and that there are no incentives for people to invest time and effort into writing high-quality reviews. This is a severe misunderstanding of reality. Of course, high-quality reviews benefit the research area you work in and thereby indirectly yourself, but the way you write your reviews also has a much more immediate impact on your career. While your identity will not be known to the author, it usually is known to several (typically senior) colleagues: editors, PC chairs, senior PC members, and fellow reviewers. The views of all of these people regarding your professional qualities as a scientist will be influenced by your review. Concretely, if I ask a colleague for a review of a paper I am responsible for and I receive a substandard review, then I will be less likely to think of that person's name when I have a more prestigious task to give away in the future. I will also be somewhat annoyed with that person, as they will have caused me additional work (finding another reviewer, reviewing the paper myself, etc.) and as they have indirectly damaged my own reputation (as the person passing on the unprofessional review or as the person having to miss a deadline for delivering a decent review). Of course, somebody's reputation as a reviewer is just one of many aspects that others will use to judge them as professional scientists, but it is one of them. ----------------------------------------------------------------------