The rapid progress of GANs and Diffusion Models poses new challenges for detecting AI-generated images. Although CLIP-based detectors exhibit promising generalization, they often rely on semantic cues rather than generator artifacts, leading to brittle performance under distribution shifts. In this work, we revisit the nature of semantic bias and uncover that Patch Shuffle provides an unusually strong benefit for CLIP, that disrupts global semantic continuity while preserving local artifact cues, which reduces semantic entropy and homogenizes feature distributions between natural and synthetic images. Through a detailed layer-wise analysis, we further show that CLIP's deep semantic structure functions as a regulator that stabilizes cross-domain representations once semantic bias is suppressed. Guided by these findings, we propose SemAnti, a semantic-antagonistic fine-tuning paradigm that freezes the semantic subspace and adapts only artifact-sensitive layers under shuffled semantics. Despite its simplicity, SemAnti achieves state-of-the-art cross-domain generalization on AIGCDetectBenchmark and GenImage, demonstrating that regulating semantics is key to unlocking CLIP's full potential for robust AI-generated image detection.

翻译：GAN和扩散模型的快速发展为检测AI生成图像带来了新的挑战。尽管基于CLIP的检测器展现出良好的泛化能力，但它们往往依赖语义线索而非生成器伪影，导致在分布偏移下性能脆弱。本研究重新审视了语义偏差的本质，发现Patch Shuffle为CLIP提供了异常强大的优势——它在破坏全局语义连续性的同时保留了局部伪影线索，从而降低了语义熵并使自然图像与合成图像的特征分布趋于同质化。通过逐层分析，我们进一步揭示了CLIP的深层语义结构在语义偏差被抑制后，可作为稳定跨域表征的调控器。基于这些发现，我们提出了SemAnti——一种语义对抗微调范式，该范式冻结语义子空间，仅在乱序语义下调整对伪影敏感的层。尽管方法简洁，SemAnti在AIGCDetectBenchmark和GenImage数据集上实现了最先进的跨域泛化性能，证明调控语义是释放CLIP在鲁棒AI生成图像检测中全部潜力的关键。