Hebbian and anti-Hebbian plasticity are widely observed in the biological brain, yet their theoretical understanding remains limited. In this work, we find that when a learning method is regularized with L2 weight decay, its learning signal will gradually align with the direction of the Hebbian learning signal as it approaches stationarity. This Hebbian-like behavior is not unique to SGD: almost any learning rule, including random ones, can exhibit the same signature long before learning has ceased. We also provide a theoretical explanation for anti-Hebbian plasticity in regression tasks, demonstrating how it can arise naturally from gradient or input noise, and offering a potential reason for the observed anti-Hebbian effects in the brain. Certainly, our proposed mechanisms do not rule out any conventionally established forms of Hebbian plasticity and could coexist with them extensively in the brain. A key insight for neurophysiology is the need to develop ways to experimentally distinguish these two types of Hebbian observations.

翻译：Hebbian与反Hebbian可塑性在生物大脑中广泛存在，但其理论理解仍显不足。本研究发现，当学习方法采用L2权重衰减进行正则化时，其学习信号在接近稳态时会逐渐与Hebbian学习信号的方向对齐。这种类Hebbian行为并非SGD所独有：几乎所有学习规则（包括随机规则）在学习停止前很久即可表现出相同特征。我们同时为回归任务中的反Hebbian可塑性提供了理论解释，阐明其如何从梯度噪声或输入噪声中自然产生，并为大脑中观测到的反Hebbian效应提供了潜在成因。需要强调的是，本文提出的机制并不排除任何传统确立的Hebbian可塑性形式，且可能与它们在大脑中广泛共存。对神经生理学的关键启示在于：需要开发实验方法以区分这两类Hebbian观测现象。