Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held-out conversations is less understood. We propose controllable counterfactuals (CoCo) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? CoCo leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turn-level by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with CoCo-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.

翻译：州级对话跟踪者在基准数据集方面取得了显著进展,但是他们对于超脱对话之外新颖和现实情景的概括化能力却不那么为人所理解。 我们提议了可控反事实(CoCo),以弥补这一差距并评价关于新情景的对话状态跟踪模型,即如果用户反应不同,但仍与对话流一致,系统能否成功应对请求? 州级对话跟踪者利用翻转信仰,将反事实性条件分为两个步骤,产生新的对话情景:(一) 在转弯层次,通过空档和添加后替换空档,反事实目标生成能力较低;(二) 反事实性对话生成,以(一)为条件,并与对话流保持一致。 评估多 WOZ 数据设置的最新DST 模型与CoCoCo生成的反现实相比,能否成功应对请求,导致绝对目标准确性下降30.8%(从49.4%到18.6%)。 相比之下,广泛使用的技术,例如语音生成技术,只能影响最高为2%的准确性。 人类评估显示CO-II的准确性,其原始对话比原始目标更准确地反映95的准确性,其原始的准确性更能,更反映其原始的准确性是加强原点,作为原始的用户对话作为基础的准确性,作为基础。