Negative bias refers to the tendency of large language models (LLMs) to excessively generate negative responses in binary decision tasks (e.g., yes-no question answering). Previous research has focused on detecting and addressing negative attention heads that induce negative bias. However, the underlying detailed factors influencing negative bias remain underexplored. In this paper, we demonstrate that LLMs exhibit format-level negative bias, meaning the prompt format more influences their responses than the semantics of the negative response. For the fine-grained study of the negative bias, we introduce a pipeline for constructing the evaluation set, which systematically categorizes the dataset into three subsets based on the model's parametric knowledge: correct, incorrect, and insufficient relevant knowledge. Through analysis of this evaluation set, we identify a shortcut behavior in which models tend to generate negative responses when they lack sufficient knowledge to answer a yes-no question, leading to negative bias. We further examine how negative bias changes under various prompting scenarios related to parametric knowledge. We observe that providing relevant context and offering an "I don't know" option generally reduces negative bias, whereas chain-of-thought prompting tends to amplify the bias. Finally, we demonstrate that the degree of negative bias can vary depending on the type of prompt, which influences the direction of the response. Our work reveals the various factors that influence negative bias, providing critical insights for mitigating it in LLMs.

翻译：负面偏见指的是大型语言模型（LLMs）在二元决策任务（例如是非问答）中过度生成否定性回答的倾向。先前的研究主要集中于检测和解决引发负面偏见的负面注意力头。然而，影响负面偏见的内在详细因素仍未得到充分探索。本文中，我们证明了LLMs表现出格式层面的负面偏见，即提示格式对其回答的影响大于否定性回答的语义。为了对负面偏见进行细粒度研究，我们引入了一个构建评估集的流程，该流程基于模型的参数知识将数据集系统性地分为三个子集：正确、错误和知识不足。通过分析该评估集，我们发现了一种捷径行为：当模型缺乏足够知识来回答是非问题时，倾向于生成否定性回答，从而导致负面偏见。我们进一步考察了在不同与参数知识相关的提示场景下负面偏见的变化。我们观察到，提供相关上下文和提供“我不知道”选项通常能减少负面偏见，而思维链提示则倾向于放大该偏见。最后，我们证明了负面偏见的程度可能因提示类型而异，这会影响回答的方向。我们的工作揭示了影响负面偏见的多种因素，为减轻LLMs中的负面偏见提供了关键见解。