Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to $\sim2\times$ at matched accuracy.

翻译：现代大型语言模型通过长链思维实现了令人瞩目的推理能力，但其在推理过程中会产生巨大的计算成本，这促使人们研究提升性能-成本比的技术。在这些技术中，推测解码通过使用快速但不准确的草稿模型自回归地生成候选词元，然后由能力更强的目标模型并行验证，从而加速推理。然而，由于在语义等价步骤中词元不匹配导致的不必要拒绝，传统的词元级推测解码在推理任务中表现不佳。尽管近期研究已转向步骤级语义验证——通过接受或拒绝整个推理步骤来提高效率，但现有的步骤级方法仍会重新生成许多被拒绝的步骤，改进有限，浪费了宝贵的目标模型计算资源。为应对这一挑战，我们提出套利，一种新颖的步骤级推测生成框架，该框架根据草稿模型与目标模型之间的相对优势动态路由生成过程。套利不采用固定的接受阈值，而是使用一个轻量级路由器来预测目标模型何时可能生成质量显著更优的步骤。这种路由方式近似于一个理想的套利预言机，该预言机总是选择质量更高的步骤，从而实现近乎最优的效率-准确性权衡。在多个数学推理基准测试中，套利持续超越先前的步骤级推测解码基线，在保持相同准确性的情况下，将推理延迟降低了高达约2倍。