AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial. Most prior methods build the routing framework by training a single model across all strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization. Prior models also typically use a single input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions. To address these gaps, we propose CONCUR, a continual routing framework that supports both constrained and unconstrained routing (i.e., routing with or without a budget). Our modular design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost. Our predictors also leverage multiple representations of both tasks and computation strategies to better capture overall problem complexity. Experiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting.

翻译：人工智能任务在复杂度上存在差异，最适合通过不同的计算策略（例如模型与解码方法的组合）来处理。因此，一个将任务映射到适当策略的有效路由系统至关重要。大多数现有方法通过在所有策略上训练单一模型来构建路由框架，这导致每当新策略出现时都需要完全重新训练，产生高昂开销。然而，此类持续路由的尝试常面临泛化困难。先前模型通常仅使用单一输入表示，限制了其捕捉路由问题完整复杂性的能力，导致次优的路由决策。为弥补这些不足，我们提出了CONCUR——一种支持约束与非约束路由（即在有预算或无预算条件下进行路由）的持续路由框架。我们的模块化设计为每个策略训练独立的预测器模型，能以较低的额外训练成本无缝整合新策略。我们的预测器还利用任务与计算策略的多重表示，以更好地捕捉整体问题复杂性。在分布内与分布外、知识密集与推理密集型任务上的实验表明：在持续与非持续设置中，我们的方法在端到端准确率上优于最佳单一策略及现有强路由技术，同时降低推理成本；在持续设置中还能减少训练成本。