Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal language models (MLLMs) for this purpose, but their substantial computational cost hinders practical application. This limitation motivates our development of a lightweight captioning model. Our investigation begins by replacing the large-scale language component in MLLMs with a compact 125M-parameter model. Surprisingly, this compact model, despite a 93x reduction in size, achieves comparable performance to MLLMs, suggesting that factual image captioning does not significantly require the complex reasoning abilities of LLMs. Despite this promising result, our lightweight model still lacks reliability. To address this, we draw inspiration from the human visual process: perceiving a global and coarse understanding of the scene before attending to finer details. Accordingly, we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions, identified by referencing the previous coarse caption, and to produce a refined description. Experimental results demonstrate the superiority of our model in both single-sentence and detailed captioning, extending even to long-range video QA tasks.

翻译：视频聊天机器人及导航机器人等系统常依赖流式图像描述任务以解析视觉输入。现有方法通常采用大型多模态语言模型（MLLMs）完成该任务，但其高昂的计算成本限制了实际应用。这一局限促使我们开发一种轻量化描述模型。我们首先将MLLMs中的大规模语言模块替换为仅含1.25亿参数的紧凑模型。令人惊讶的是，该紧凑模型在参数量减少93倍的情况下，仍能达到与MLLMs相当的性能，这表明事实性图像描述任务并不显著依赖大型语言模型的复杂推理能力。尽管结果令人鼓舞，我们的轻量化模型仍存在可靠性不足的问题。受人类视觉处理机制启发——即先获得场景的全局粗粒度理解，再关注局部细节——我们提出一种多模态自优化框架。该框架引导模型通过参考先前生成的粗粒度描述定位显著区域，并利用其特征生成精细化描述。实验结果表明，我们的模型在单句描述与细节描述任务中均表现优异，其优势甚至可延伸至长时序视频问答任务。