Stochastic bilevel optimization, which captures the inherent nested structure of machine learning problems, is gaining popularity in many recent applications. Existing works on bilevel optimization mostly consider either unconstrained problems or constrained upper-level problems. This paper considers the stochastic bilevel optimization problems with equality constraints both in the upper and lower levels. By leveraging the special structure of the equality constraints problem, the paper first presents an alternating implicit projected SGD approach and establishes the $\tilde{\cal O}(\epsilon^{-2})$ sample complexity that matches the state-of-the-art complexity of ALSET \citep{chen2021closing} for unconstrained bilevel problems. To further save the cost of projection, the paper presents two alternating implicit projection-efficient SGD approaches, where one algorithm enjoys the $\tilde{\cal O}(\epsilon^{-2}/T)$ upper-level and $\tilde{\cal O}(\epsilon^{-1.5}/T^{\frac{3}{4}})$ lower-level projection complexity with ${\cal O}(T)$ lower-level batch size, and the other one enjoys $\tilde{\cal O}(\epsilon^{-1.5})$ upper-level and lower-level projection complexity with ${\cal O}(1)$ batch size. Application to federated bilevel optimization has been presented to showcase the empirical performance of our algorithms. Our results demonstrate that equality-constrained bilevel optimization with strongly-convex lower-level problems can be solved as efficiently as stochastic single-level optimization problems.

翻译：包含机器学习问题固有嵌套结构的双层Stochatical优化, 在最近许多应用程序中越来越受欢迎。 双层优化的现有工作主要考虑非约束问题或受限制的上层问题。 本文考虑了上层和下层平等制约的双层优化问题。 通过利用平等制约问题的特殊结构, 本文首先呈现了一种交替的隐含预测SGD 方法, 并确立了美元( epsilon ⁇ -2} ( epsilon ⁇ -2} ) 样本复杂性, 与 ALSET 最新复杂程度( citep{ chen2021 clooming} 中未受限制的双层问题相比。 为了进一步节省预测成本, 文件提出了两种互换的隐含性预测效率的 SGD 方法, 其中一种算法使用 $tilde=cal O} ( eepsilon) / 2} / T), 美元( leveloplevelople) 和 美元( levelople) levelople le le) leal leveloplevelople 级别( lex) levelopleveloplex) 级别( le) 级别, levelop lex) 级别, levelop levelopleveloplevelop lex) 级别, leveloplevelop lex) lex) leveloplex levelopmental le 一级, 一级, levelopmental levelmental lex level level level level level level level level level level level level level level level level level level lex lex le le le le level level le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le le