Effective evaluation methods remain a significant challenge for research on open-domain conversational dialogue systems. Explicit satisfaction ratings can be elicited from users, but users often do not provide ratings when asked, and those they give can be highly subjective. Post-hoc ratings by experts are an alternative, but these can be both expensive and complex to collect. Here, we explore the creation of automated methods for predicting both expert and user ratings of open-domain dialogues. We compare four different approaches. First, we train a baseline model using an end-to-end transformer to predict ratings directly from the raw dialogue text. The other three methods are variants of a two-stage approach in which we first extract interpretable features at the turn level that capture, among other aspects, user dialogue behaviors indicating contradiction, repetition, disinterest, compliments, or criticism. We project these features to the dialogue level and train a dialogue-level MLP regression model, a dialogue-level LSTM, and a novel causal inference model called counterfactual-LSTM (CF-LSTM) to predict ratings. The proposed CF-LSTM is a sequential model over turn-level features which predicts ratings using multiple regressors depending on hypotheses derived from the turn-level features. As a causal inference model, CF-LSTM aims to learn the underlying causes of a specific event, such as a low rating. We also bin the user ratings and perform classification experiments with all four models. In evaluation experiments on conversational data from the Alexa Prize SocialBot, we show that the CF-LSTM achieves the best performance for predicting dialogue ratings and classification.

翻译：有效的评价方法仍然是对开放域对话系统研究的重大挑战。 用户可以对公开域对话系统进行明确的满意度评级,但用户通常不会在被询问时提供评级,而他们提供的评级可能具有高度主观性。 专家的热后评级是一种替代办法,但也可能是昂贵和复杂的收集方法。 在这里, 我们探索创建自动方法, 用于预测开放域对话的专家和用户评级。 我们比较了四种不同的方法。 首先, 我们用端对端变压器来培训基线模型, 直接预测原始对话文本的评级。 其余三种方法是两阶段评级的变式, 我们首先在转档一级提取可解释的功能, 显示用户对话中显示矛盾、重复、不感兴趣、赞美或批评。 我们将这些功能投放到对话级别上, 培养对话级MLP回归模型, 对话级LSTM, 以及一种叫反事实- LSTM(C-LSTM)到预测评级。 提议的CLSTM(C- LSTM)是两个阶段的双级评级方法, 显示从B级的排序, 和结果级等级, 显示多级的排序。