Large language model based agents are increasingly deployed in complex, tool augmented environments. While reinforcement learning provides a principled mechanism for such agents to improve through interaction, its effectiveness critically depends on the availability of structured training tasks. In many realistic settings, however, no such tasks exist a challenge we term task scarcity, which has become a key bottleneck for scaling agentic RL. Existing approaches typically assume predefined task collections, an assumption that fails in novel environments where tool semantics and affordances are initially unknown. To address this limitation, we formalize the problem of Task Generation for Agentic RL, where an agent must learn within a given environment that lacks predefined tasks. We propose CuES, a Curiosity driven and Environment grounded Synthesis framework that autonomously generates diverse, executable, and meaningful tasks directly from the environment structure and affordances, without relying on handcrafted seeds or external corpora. CuES drives exploration through intrinsic curiosity, abstracts interaction patterns into reusable task schemas, and refines them through lightweight top down guidance and memory based quality control. Across three representative environments, AppWorld, BFCL, and WebShop, CuES produces task distributions that match or surpass manually curated datasets in both diversity and executability, yielding substantial downstream policy improvements. These results demonstrate that curiosity driven, environment grounded task generation provides a scalable foundation for agents that not only learn how to act, but also learn what to learn. The code is available at https://github.com/modelscope/AgentEvolver/tree/main/research/CuES.

翻译：基于大语言模型的智能体正日益部署于复杂且工具增强的环境中。尽管强化学习为这类智能体提供了通过交互进行改进的原则性机制，但其有效性关键依赖于结构化训练任务的可获得性。然而，在许多现实场景中，此类任务并不存在——我们称之为任务稀缺性挑战，这已成为扩展智能体强化学习的关键瓶颈。现有方法通常假设存在预定义的任务集合，这一假设在工具语义和可供性最初未知的新颖环境中并不成立。为应对这一局限，我们形式化了智能体强化学习的任务生成问题，即智能体必须在缺乏预定义任务的给定环境中进行学习。我们提出了CuES，一种好奇心驱动与环境接地的合成框架，能够直接从环境结构和可供性中自主生成多样化、可执行且有意义的学习任务，而无需依赖人工设计的种子任务或外部语料库。CuES通过内在好奇心驱动探索，将交互模式抽象为可重用的任务模式，并通过轻量级的自上而下引导和基于记忆的质量控制机制对其进行优化。在AppWorld、BFCL和WebShop三个代表性环境中的实验表明，CuES生成的任务分布在多样性和可执行性方面均达到或超越了人工标注数据集的水准，并带来了显著的下游策略性能提升。这些结果表明，好奇心驱动且环境接地的任务生成为智能体提供了可扩展的基础，使其不仅学会如何行动，更学会学习什么。代码发布于 https://github.com/modelscope/AgentEvolver/tree/main/research/CuES。