Transformer-based methods have recently become the prevailing approach for Human-Object Interaction (HOI) detection. However, the Transformer architecture does not explicitly model the relational structures inherent in HOI detection, which impedes the recognition of interactions. In contrast, Graph Neural Networks (GNNs) are inherently better suited for this task, as they explicitly model the relationships between human-object pairs. Therefore, in this paper, we propose \textbf{M}ultimodal \textbf{G}raph \textbf{N}etwork \textbf{M}odeling (MGNM) that leverages GNN-based relational structures to enhance HOI detection. Specifically, we design a multimodal graph network framework that explicitly models the HOI task in a four-stage graph structure. Furthermore, we introduce a multi-level feature interaction mechanism within our graph network. This mechanism leverages multi-level visual and language features to enhance information propagation across human-object pairs. Consequently, our proposed MGNM achieves state-of-the-art (SOTA) performance on two widely used benchmarks: HICO-DET and V-COCO. Moreover, when integrated with a more advanced object detector, our method demonstrates a significant performance gain and maintains an effective balance between rare and non-rare classes.

翻译：基于Transformer的方法近期已成为人-物交互检测的主流方法。然而，Transformer架构并未显式建模人-物交互检测中固有的关系结构，这阻碍了交互识别的准确性。相比之下，图神经网络因其能显式建模人-物对之间的关系，本质上更适合此任务。为此，本文提出基于图神经网络关系结构的**多模态图网络建模**方法，以增强人-物交互检测性能。具体而言，我们设计了一个多模态图网络框架，通过四阶段图结构显式建模人-物交互任务。此外，我们在图网络中引入了多层次特征交互机制，该机制利用多层次视觉与语言特征来增强人-物对之间的信息传播。实验表明，我们提出的MGNM在两个广泛使用的基准数据集HICO-DET和V-COCO上取得了最先进的性能。当与更先进的目标检测器结合时，本方法展现出显著的性能提升，并在稀有类与非稀有类之间保持了有效的平衡。