Bilevel optimization has emerged as a technique for addressing a wide range of machine learning problems that involve an outer objective implicitly determined by the minimizer of an inner problem. While prior works have primarily focused on the parametric setting, a learning-theoretic foundation for bilevel optimization in the nonparametric case remains relatively unexplored. In this paper, we take a first step toward bridging this gap by studying Kernel Bilevel Optimization (KBO), where the inner objective is optimized over a reproducing kernel Hilbert space. This setting enables rich function approximation while providing a foundation for rigorous theoretical analysis. In this context, we derive novel finite-sample generalization bounds for KBO, leveraging tools from empirical process theory. These bounds further allow us to assess the statistical accuracy of gradient-based methods applied to the empirical discretization of KBO. We numerically illustrate our theoretical findings on a synthetic instrumental variable regression task.

翻译：双层优化已成为解决一系列机器学习问题的技术，其中外层目标由内层问题最小化解隐式确定。尽管先前研究主要集中于参数化设定，但非参数情形下双层优化的学习理论基础仍相对缺乏探索。本文通过研究核双层优化（Kernel Bilevel Optimization, KBO）迈出了填补这一空白的第一步，其中内层目标在再生核希尔伯特空间上进行优化。该设定在提供严格理论分析基础的同时，实现了丰富的函数逼近能力。在此框架下，我们利用经验过程理论工具，推导了KBO的有限样本泛化界。这些界进一步使我们能够评估应用于KBO经验离散化的梯度方法的统计准确性。我们通过合成工具变量回归任务对理论结果进行了数值验证。