Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.

翻译：同行审议是学术界质量控制的主要手段;作为同行审议进程的一项成果,方案和地区主席根据审评报告和收到的评分对每份文件作出接受决定;科学工作的质量是多方面的;加上审查的主观性,最后决定难于和费时;为支持同行审议的最后一步,我们将其正式确定为纸质排名问题;我们为基于同行审议结果的提交材料的排序引入了一个新的、多层面的通用评价框架,其中考虑到有效性、效率和公平性;我们建议从学习角度出发考虑审查文本和评分以缓解审查中不可避免的偏差和噪音的任务;我们对2018年ACL会议同行审议数据的实验表明,我们基于优先学习的方法优于基线和先前的工作,同时强调使用审评文本和分数对提交材料进行排名的重要性。