The ability to dynamically adapt neural networks to newly-available data without performance deterioration would revolutionize deep learning applications. Streaming learning (i.e., learning from one data example at a time) has the potential to enable such real-time adaptation, but current approaches i) freeze a majority of network parameters during streaming and ii) are dependent upon offline, base initialization procedures over large subsets of data, which damages performance and limits applicability. To mitigate these shortcomings, we propose Cold Start Streaming Learning (CSSL), a simple, end-to-end approach for streaming learning with deep networks that uses a combination of replay and data augmentation to avoid catastrophic forgetting. Because CSSL updates all model parameters during streaming, the algorithm is capable of beginning streaming from a random initialization, making base initialization optional. Going further, the algorithm's simplicity allows theoretical convergence guarantees to be derived using analysis of the Neural Tangent Random Feature (NTRF). In experiments, we find that CSSL outperforms existing baselines for streaming learning in experiments on CIFAR100, ImageNet, and Core50 datasets. Additionally, we propose a novel multi-task streaming learning setting and show that CSSL performs favorably in this domain. Put simply, CSSL performs well and demonstrates that the complicated, multi-step training pipelines adopted by most streaming methodologies can be replaced with a simple, end-to-end learning approach without sacrificing performance.

翻译：将神经网络动态地适应新获得的数据而不造成性能下降的能力动态地调整神经网络以适应新获得的数据,将使深层学习应用发生革命性变革。将学习(即一次从一个数据实例中学习)流动起来(即一次从一个数据实例中学习)具有使这种实时适应的潜力,但当前的做法(i)在流流和(ii)期间冻结了大多数网络参数)取决于大量数据子集的离线、基础初始化程序,从而损害性能和限制适用性。为了减轻这些缺陷,我们提议冷启动流学习(CSSL),这是一种简单、端到端的方法,用来与使用重放和数据增强相结合的深层网络进行流学,以避免灾难性的遗忘。由于CSSL在流过程中更新所有模型参数,这种算法能够从随机初始初始化开始流,使基础初始初始初始化选项具有可选择性。此外,这种算法的简单化使得理论趋同保证能够利用对 Neal Tangent 随机性功能(NTRFRF) (NTRF) 的分析来。在实验中发现CSL,我们发现CSD-fread-formal-destrain-destrain-destrain-st 和C-drodudustruc-S) rodustrutal Stutection a laction a laction a laction astrual laction a laction acument the droduction arostration astration atoction laction rodudestrutection astration