Medical multi-document summarization (MDS) is a complex task that requires effectively managing cross-document relationships. This paper investigates whether incorporating hierarchical structures in the inputs of MDS can improve a model's ability to organize and contextualize information across documents compared to traditional flat summarization methods. We investigate two ways of incorporating hierarchical organization across three large language models (LLMs), and conduct comprehensive evaluations of the resulting summaries using automated metrics, model-based metrics, and domain expert evaluation of preference, understandability, clarity, complexity, relevance, coverage, factuality, and coherence. Our results show that human experts prefer model-generated summaries over human-written summaries. Hierarchical approaches generally preserve factuality, coverage, and coherence of information, while also increasing human preference for summaries. Additionally, we examine whether simulated judgments from GPT-4 align with human judgments, finding higher agreement along more objective evaluation facets. Our findings demonstrate that hierarchical structures can improve the clarity of medical summaries generated by models while maintaining content coverage, providing a practical way to improve human preference for generated summaries.

翻译：医学多文档摘要（MDS）是一项复杂的任务，需要有效管理跨文档关系。本文研究了与传统的扁平化摘要方法相比，在MDS输入中引入层次化结构是否能提升模型跨文档组织和情境化信息的能力。我们探索了三种大语言模型（LLMs）中融入层次化组织的两种方式，并采用自动化指标、基于模型的指标以及领域专家对偏好性、可理解性、清晰度、复杂性、相关性、覆盖度、事实性和连贯性的评估，对生成的摘要进行了全面评价。结果显示，人类专家更倾向于模型生成的摘要而非人工撰写的摘要。层次化方法通常能保持信息的事实性、覆盖度和连贯性，同时提高人类对摘要的偏好。此外，我们检验了GPT-4的模拟判断是否与人类判断一致，发现在更客观的评价维度上具有更高的一致性。我们的研究结果表明，层次化结构能提升模型生成的医学摘要的清晰度，同时保持内容覆盖度，为提高人类对生成摘要的偏好提供了实用途径。