Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capability of large language models (LLMs), enabling autonomous agents that can conduct effective multi-turn and tool-integrated reasoning. While instructions serve as the primary protocol for defining agents, RLVR typically relies on static and manually designed instructions. However, those instructions may be suboptimal for the base model, and the optimal instruction may change as the agent's policy improves and explores the interaction with the environment. To bridge the gap, we introduce INSPO, a novel Instruction-Policy co-evolution framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop. INSPO maintains a dynamic population of instruction candidates that are sampled with questions, where reward signals in RL loops are automatically attributed to each instruction, and low performers are periodically pruned. New instructions are generated and verified through an on-policy reflection mechanism, where an LLM-based optimizer analyzes past experience from a replay buffer and evolves more effective strategies given the current policy. We conduct extensive experiments on multi-turn retrieval and reasoning tasks, demonstrating that INSPO substantially outperforms strong baselines relying on static instructions. INSPO discovers innovative instructions that guide the agent toward more strategic reasoning paths, achieving substantial performance gains with only a marginal increase in computational overhead.

翻译：可验证奖励强化学习（RLVR）提升了大型语言模型（LLM）的推理能力，实现了能够进行有效多轮次及工具集成推理的自主智能体。虽然指令是定义智能体的主要协议，但RLVR通常依赖于静态且人工设计的指令。然而，这些指令对于基础模型可能并非最优，且最优指令可能随着智能体策略的改进及其与环境交互的探索而发生变化。为弥合此差距，我们提出了INSPO，一种新颖的指令-策略协同进化框架，将指令优化整合为强化学习（RL）循环的动态组成部分。INSPO维护一个动态的指令候选池，这些指令与问题一同采样，其中RL循环中的奖励信号自动归属于每条指令，并定期淘汰低性能指令。新指令通过一种基于策略的反思机制生成和验证，其中基于LLM的优化器分析回放缓冲区中的过往经验，并根据当前策略演化出更有效的策略。我们在多轮次检索与推理任务上进行了大量实验，结果表明INSPO显著优于依赖静态指令的强基线方法。INSPO发现了创新性指令，能够引导智能体走向更具战略性的推理路径，仅以边际计算开销的增加即实现了显著的性能提升。