Improving the quality of Natural Language Understanding (NLU) models, and more specifically, task-oriented semantic parsing models, in production is a cumbersome task. In this work, we present a system called AutoNLU, which we designed to scale the NLU quality improvement process. It adds automation to three key steps: detection, attribution, and correction of model errors, i.e., bugs. We detected four times more failed tasks than with random sampling, finding that even a simple active learning sampling method on an uncalibrated model is surprisingly effective for this purpose. The AutoNLU tool empowered linguists to fix ten times more semantic parsing bugs than with prior manual processes, auto-correcting 65% of all identified bugs.

翻译：提高自然语言理解模型的质量,更具体地说,在生产过程中,以任务为导向的语义分析模型的质量是一项繁琐的任务。在这项工作中,我们提出了一个名为AutoNLU的系统,我们设计这个系统是为了扩大自然语言理解模型质量的改进过程。它增加了自动化的三个关键步骤:发现、归属和纠正模型错误,即错误。我们检测到的失败任务比随机抽样要多四倍,发现即使是在未经校准的模型上采用简单的主动学习抽样方法,对于这个目的来说,也非常有效。AutoNLU工具授权语言学家将语义分析错误比以前的手动程序多十倍,自动纠正了所有被识别错误的65%。