Vision-language models achieve expert-level performance on medical imaging tasks but exhibit significant diagnostic accuracy disparities across demographic groups. We introduce fairness-aware Low-Rank Adaptation for medical VLMs, combining parameter efficiency with explicit fairness optimization. Our key algorithmic contribution is a differentiable MaxAccGap loss that enables end-to-end optimization of accuracy parity across demographic groups. We propose three methods: FR-LoRA integrates MaxAccGap regularization into the training objective, GR-LoRA applies inverse frequency weighting to balance gradient contributions, and Hybrid-LoRA combines both mechanisms. Evaluated on 10,000 glaucoma fundus images, GR-LoRA reduces diagnostic accuracy disparities by 69% while maintaining 53.15% overall accuracy. Ablation studies reveal that strong regularization strength achieves optimal fairness with minimal accuracy trade-off, and race-specific optimization yields 60% disparity reduction. Our approach requires only 0.24% trainable parameters, enabling practical deployment of fair medical AI in resource-constrained healthcare settings.

翻译：视觉语言模型在医学影像任务中已达到专家级性能，但在不同人口统计学群体间表现出显著的诊断准确率差异。我们提出了面向医学视觉语言模型的公平性感知低秩自适应方法，将参数效率与显式公平性优化相结合。我们的核心算法贡献是可微分的最大准确率差距损失函数，实现了跨人口统计学群体的端到端准确率均衡优化。我们提出了三种方法：FR-LoRA将最大准确率差距正则化整合至训练目标中，GR-LoRA采用逆频率加权以平衡梯度贡献，Hybrid-LoRA则结合了两种机制。在10,000张青光眼眼底图像上的评估显示，GR-LoRA将诊断准确率差异降低了69%，同时保持了53.15%的整体准确率。消融研究表明，强正则化强度能以最小准确率代价实现最优公平性，而针对种族的特定优化可带来60%的差异降低。我们的方法仅需0.24%的可训练参数，使得公平医疗人工智能在资源受限的医疗环境中具备实际部署可行性。