Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.

翻译：近期图像字幕生成的进展聚焦于数据和模型规模的扩展，使得预训练和微调的成本大幅增加。为解决这一问题，我们提出了 SmallCap，该模型会根据在数据存储库中检索的相关字幕生成一种有人类识别准确率的字幕。我们的模型轻巧，训练速度快，因为唯一的学习参数是在预训练的 CLIP 编码器和 GPT-2 解码器之间的新引入的跨注意力层中设定。SmallCap 可以在不进行额外微调的情况下转移到新的领域，并且可以利用训练无关的大规模数据，因为数据存储库的内容可以很容易地替换。我们的实验表明，SmallCap 只需在 COCO 数据上训练就能在该基准测试中获得具有竞争力的性能，并且仅通过从目标领域的数据中检索字幕就能转移到其他领域。通过训练无关的利用多样化的人类和网络数据，我们实现了进一步的改进，这些数据对于各种领域都非常有效，包括用于测试对未见过的视觉概念的泛化能力的 nocaps 基准测试。