While the Contrastive Language-Image Pretraining(CLIP) model has achieved remarkable success in a variety of downstream vison language understanding tasks, enhancing its capability for fine-grained image-text alignment remains an active research focus. To this end, most existing works adopt the strategy of explicitly increasing the granularity of visual information processing, e.g., incorporating visual prompts to guide the model focus on specific local regions within the image. Meanwhile, researches on Multimodal Large Language Models(MLLMs) have demonstrated that training with long and detailed textual descriptions can effectively improve the model's fine-grained vision-language alignment. However, the inherent token length limitation of CLIP's text encoder fundamentally limits CLIP to process more granular textual information embedded in long text sequences. To synergistically leverage the advantages of enhancing both visual and textual content processing granularity, we propose PixCLIP, a novel framework designed to concurrently accommodate visual prompt inputs and process lengthy textual descriptions. Specifically, we first establish an automated annotation pipeline capable of generating pixel-level localized, long-form textual descriptions for images. Utilizing this pipeline, we construct LongGRIT, a high-quality dataset comprising nearly 1.5 million samples. Secondly, we replace CLIP's original text encoder with the LLM and propose a three-branch pixel-text alignment learning framework, facilitating fine-grained alignment between image regions and corresponding textual descriptions at arbitrary granularity. Experiments demonstrate that PixCLIP showcases breakthroughs in pixel-level interaction and handling long-form texts, achieving state-of-the-art performance.

翻译：尽管对比语言-图像预训练（CLIP）模型在多种下游视觉语言理解任务中取得了显著成功，提升其细粒度图像-文本对齐能力仍是当前研究的热点。为此，现有研究大多采用显式增加视觉信息处理粒度的策略，例如引入视觉提示引导模型关注图像中的特定局部区域。同时，多模态大语言模型（MLLMs）的研究表明，使用长而详细的文本描述进行训练能有效提升模型的细粒度视觉语言对齐能力。然而，CLIP文本编码器固有的令牌长度限制，从根本上制约了其处理长文本序列中嵌入的更细粒度文本信息的能力。为协同利用增强视觉与文本内容处理粒度的优势，我们提出了PixCLIP——一种新颖的框架，旨在同时支持视觉提示输入并处理长文本描述。具体而言，我们首先建立了一个自动化标注流程，能够为图像生成像素级定位的长文本描述。利用该流程，我们构建了LongGRIT数据集，这是一个包含近150万样本的高质量数据集。其次，我们将CLIP的原始文本编码器替换为大语言模型（LLM），并提出了一种三分支像素-文本对齐学习框架，促进图像区域与对应文本描述在任意粒度上的细粒度对齐。实验表明，PixCLIP在像素级交互与长文本处理方面展现出突破性进展，实现了最先进的性能。