The emerging in-context learning (ICL) ability of large language models (LLMs) has prompted their use for predictive tasks in various domains with different data types, including tabular data, facilitated by serialization methods. However, with increasing applications in high-stakes domains, it has been shown that LLMs can inherit social bias and discrimination from their pre-training data. In this work, we investigate inherent bias in LLMs during in-context learning with tabular data. We focus on an optimal demonstration selection approach that utilizes latent concept variables for resource-efficient task adaptation. We design data augmentation strategies that reduce the correlation between predictive outcomes and sensitive variables, helping promote fairness during latent concept learning. We utilize the learned concept to select demonstrations and obtain fair predictions. The latent concept variables are learned using a smaller internal LLM and generalized to larger external LLMs. We empirically verify that the fair latent variable approach improves fairness results on tabular datasets compared to multiple heuristic demonstration selection methods.

翻译：大型语言模型（LLMs）新兴的上下文学习（ICL）能力，结合序列化方法，推动了其在包含表格数据在内的多种数据类型预测任务中的应用。然而，随着在高风险领域应用的增加，研究表明LLMs可能从其预训练数据中继承社会偏见与歧视。本文研究了LLMs在表格数据上下文学习中存在的固有偏见。我们聚焦于一种利用潜在概念变量实现资源高效任务适配的最优演示选择方法。通过设计数据增强策略，降低预测结果与敏感变量之间的相关性，以促进潜在概念学习过程中的公平性。我们利用习得的概念选择演示并获取公平的预测结果。潜在概念变量通过较小的内部LLM学习，并可泛化至更大的外部LLM。实验表明，与多种启发式演示选择方法相比，基于公平潜在变量的方法在表格数据集上显著提升了公平性结果。