Despite emerging efforts to enhance the safety of Vision-Language Models (VLMs), current approaches face two main shortcomings. 1) Existing safety-tuning datasets and benchmarks only partially consider how image-text interactions can yield harmful content, often overlooking contextually unsafe outcomes from seemingly benign pairs. This narrow coverage leaves VLMs vulnerable to jailbreak attacks in unseen configurations. 2) Prior methods rely primarily on data-centric tuning, with limited architectural innovations to intrinsically strengthen safety. We address these gaps by introducing a holistic safety dataset and benchmark, \textbf{HoliSafe}, that spans all five safe/unsafe image-text combinations, providing a more robust basis for both training and evaluation (HoliSafe-Bench). We further propose a novel modular framework for enhancing VLM safety with a visual guard module (VGM) designed to assess the harmfulness of input images for VLMs. This module endows VLMs with a dual functionality: they not only learn to generate safer responses but can also provide an interpretable harmfulness classification to justify their refusal decisions. A significant advantage of this approach is its modularity; the VGM is designed as a plug-in component, allowing for seamless integration with diverse pre-trained VLMs across various scales. Experiments show that Safe-VLM with VGM, trained on our HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks. Additionally, the HoliSafe-Bench itself reveals critical vulnerabilities in existing VLM models. We hope that HoliSafe and VGM will spur further research into robust and interpretable VLM safety, expanding future avenues for multimodal alignment.

翻译：尽管已有研究致力于提升视觉语言模型（VLMs）的安全性，但现有方法仍存在两个主要不足。1）当前的安全调优数据集和基准仅部分考虑了图像-文本交互如何产生有害内容，往往忽视了看似良性的图文对在特定上下文中可能引发的不安全结果。这种有限的覆盖范围使得VLMs在面对未见过的配置时容易受到越狱攻击。2）先前的方法主要依赖以数据为中心的调优，在通过架构创新本质性增强安全性方面进展有限。为弥补这些不足，我们引入了一个全面的安全数据集与基准——\textbf{HoliSafe}，它涵盖了全部五种安全/不安全的图像-文本组合，为训练和评估（HoliSafe-Bench）提供了更稳健的基础。我们进一步提出了一种新颖的模块化框架，通过设计用于评估输入图像对VLMs危害性的视觉防护模块（VGM）来增强VLM的安全性。该模块赋予VLMs双重功能：它们不仅学习生成更安全的响应，还能提供可解释的危害性分类，以证明其拒绝决策的合理性。该方法的一个显著优势在于其模块化设计；VGM被设计为即插即用组件，能够无缝集成到不同规模、多样化的预训练VLMs中。实验表明，基于我们的HoliSafe训练、配备VGM的Safe-VLM在多个VLM基准测试中实现了最先进的安全性能。此外，HoliSafe-Bench本身也揭示了现有VLM模型的关键脆弱性。我们希望HoliSafe和VGM能够推动针对鲁棒且可解释的VLM安全性的进一步研究，拓展多模态对齐的未来路径。