Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images for "human-like" event understanding. Specifically, GSR task not only detects the salient activity verb (e.g. buying), but also predicts all corresponding semantic roles (e.g. agent and goods). Inspired by object detection and image captioning tasks, existing methods typically employ a two-stage framework: 1) detect the activity verb, and then 2) predict semantic roles based on the detected verb. Obviously, this illogical framework constitutes a huge obstacle to semantic understanding. First, pre-detecting verbs solely without semantic roles inevitably fails to distinguish many similar daily activities (e.g., offering and giving, buying and selling). Second, predicting semantic roles in a closed auto-regressive manner can hardly exploit the semantic relations among the verb and roles. To this end, in this paper we propose a novel two-stage framework that focuses on utilizing such bidirectional relations within verbs and roles. In the first stage, instead of pre-detecting the verb, we postpone the detection step and assume a pseudo label, where an intermediate representation for each corresponding semantic role is learned from images. In the second stage, we exploit transformer layers to unearth the potential semantic relations within both verbs and semantic roles. With the help of a set of support images, an alternate learning scheme is designed to simultaneously optimize the results: update the verb using nouns corresponding to the image, and update nouns using verbs from support images. Extensive experimental results on challenging SWiG benchmarks show that our renovated framework outperforms other state-of-the-art methods under various metrics.

翻译：地表状态识别 (GSR) 旨在生成结构化的图像语义摘要, 以便“ 人类” 事件理解。 具体地说, GSR 任务不仅检测突出的活动动词( 例如购买), 而且还预测所有相应的语义作用( 例如代理和货物 ) 。 受对象检测和图像字幕任务的启发, 现有方法通常使用一个两阶段框架:1 检测活动动词, 然后2) 根据所检测到的动词来预测语义作用。 显然, 这个不合逻辑的框架构成了对语义理解的巨大障碍。 首先, 仅仅在没有语义作用的情况下预检测动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动动词( 例如提供和提供、 购买和出售) 。 其次, 以封闭的自动反动动动动动动动动动动动动动动动动动动动动动动动动动动动动动的图像, 将演化的变动变动图显示每个变形变形变形变形的变形的变形, 变形的变形的演变形, 演变形的演变形的变形的变形, 变形的变形的演变形的演变形, 变形的变形, 变形的变形的变形的变形, 变形的变形的变形, 变形的变形的变形, 变形的变形的变形, 变形的变形的变形的变形的变形的变形的变形, 变形的变形的变形的变形的变形, 变形, 变形, 变形的变形的变形的变形的变形, 变形的变形, 变形的变形, 变形的变形的变形的变形的变形的变形, 变形的变形的变形的变形的变形的变形的变形的变形的变形, 变形的变形的变形的变形的变形的变形的变形的变形的变形的变形的变形,