Knowledge and expertise in the real-world can be disjointedly owned. To solve a complex question, collaboration among experts is often called for. In this paper, we propose CollabQA, a novel QA task in which several expert agents coordinated by a moderator work together to answer questions that cannot be answered with any single agent alone. We make a synthetic dataset of a large knowledge graph that can be distributed to experts. We define the process to form a complex question from ground truth reasoning path, neural network agent models that can learn to solve the task, and evaluation metrics to check the performance. We show that the problem can be challenging without introducing prior of the collaboration structure, unless experts are perfect and uniform. Based on this experience, we elaborate extensions needed to approach collaboration tasks in real-world settings.

翻译：现实世界中的知识和专长可以相互脱节地拥有。为了解决一个复杂的问题,常常需要专家之间的合作。在本文件中,我们建议科拉巴卡,这是一项全新的质量评估任务,由主持人协调的数名专家代理人共同工作,回答无法单独与任何单一代理人回答的问题。我们制作了一个大型知识图的合成数据集,可以分发给专家。我们界定了从地面真相推理路径、能够学会解决问题的神经网络代理模型以及用于检查业绩的评价标准中形成一个复杂的问题的过程。我们表明,除非专家是完美和统一的,否则这个问题在合作结构之前可能具有挑战性。我们根据这一经验,详细阐述了在现实世界环境中处理合作任务所需要的扩展。