Low-altitude Unmanned Aerial Vehicle (UAV) networks rely on robust semantic segmentation as a foundational enabler for distributed sensing-communication-control co-design across heterogeneous agents within the network. However, segmentation foundation models deteriorate quickly under weather, lighting, and viewpoint drift. Resource-limited UAVs cannot run gradient-based test-time adaptation, while resource-massive UAVs adapt independently, wasting shared experience. To address these challenges, we propose AdaptFly, a prompt-guided test-time adaptation framework that adjusts segmentation models without weight updates. AdaptFly features two complementary adaptation modes. For resource-limited UAVs, it employs lightweight token-prompt retrieval from a shared global memory. For resource-massive UAVs, it uses gradient-free sparse visual prompt optimization via Covariance Matrix Adaptation Evolution Strategy. An activation-statistic detector triggers adaptation, while cross-UAV knowledge pool consolidates prompt knowledge and enables fleet-wide collaboration with negligible bandwidth overhead. Extensive experiments on UAVid and VDD benchmarks, along with real-world UAV deployments under diverse weather conditions, demonstrate that AdaptFly significantly improves segmentation accuracy and robustness over static models and state-of-the-art TTA baselines. The results highlight a practical path to resilient, communication-efficient perception in the emerging low-altitude economy.

翻译：低空无人机网络依赖鲁棒的语义分割作为网络内异构智能体间分布式感知-通信-控制协同设计的基础使能技术。然而，在天气变化、光照条件及视角偏移等因素影响下，分割基础模型的性能会迅速退化。资源受限的无人机无法运行基于梯度的测试时自适应方法，而资源充足的无人机则各自独立进行自适应，未能充分利用共享经验。为应对这些挑战，我们提出AdaptFly——一种无需权重更新的提示引导测试时自适应框架。AdaptFly包含两种互补的自适应模式：针对资源受限的无人机，采用基于共享全局记忆的轻量级令牌提示检索机制；针对资源充足的无人机，则通过协方差矩阵自适应进化策略实现无需梯度的稀疏视觉提示优化。系统通过激活统计检测器触发自适应过程，并借助跨无人机知识池整合提示知识，以可忽略的带宽开销实现全机队协同。在UAVid与VDD基准数据集上的大量实验，以及多种天气条件下的真实无人机部署验证表明：相较于静态模型与当前最先进的测试时自适应基线方法，AdaptFly显著提升了分割精度与鲁棒性。该研究为新兴低空经济领域实现具备弹性且通信高效的感知系统提供了切实可行的技术路径。