Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framework that uses both audio and visual cues to track the target speaker, predict turn-taking, and generate coherent responses. By combining acoustic tokenization with multi-task, multi-stage training on monadic, synthetic, and real audio-visual dialogue datasets, AV-Dialog achieves robust streaming transcription, semantically grounded turn-boundary detection and accurate responses, resulting in a natural conversational flow. Experiments show that AV-Dialog outperforms audio-only models under interference, reducing transcription errors, improving turn-taking prediction, and enhancing human-rated dialogue quality. These results highlight the power of seeing as well as hearing for speaker-aware interaction, paving the way for {spoken} dialogue agents that perform {robustly} in real-world, noisy environments.

翻译：对话模型在嘈杂、多说话者环境中表现不佳，常产生无关响应和尴尬的轮替。我们提出AV-Dialog，首个利用音频与视觉线索追踪目标说话者、预测轮替并生成连贯响应的多模态对话框架。通过结合声学标记化技术，并在单说话者、合成及真实音频-视觉对话数据集上进行多任务、多阶段训练，AV-Dialog实现了鲁棒的流式转录、语义驱动的轮替边界检测与精准响应，从而形成自然的对话流。实验表明，在干扰条件下，AV-Dialog优于纯音频模型，减少了转录错误，提升了轮替预测准确性，并增强了人工评估的对话质量。这些结果凸显了视觉与听觉结合对于说话者感知交互的效能，为在现实世界嘈杂环境中{稳健}运行的{语音}对话智能体铺平了道路。