Abstract
We address the challenge of representation learning from a continuous stream of video as input, in a self-supervised manner. This differs from the standard approaches to video learning where videos are chopped and shuffled during training in order to create a non-redundant batch that satisfies the independently and identically distributed (IID) sample assumption expected by conventional training paradigms. When videos are only available as a continuous stream of input, the IID assumption is evidently broken, leading to poor performance. We demonstrate the drop in performance when moving from shuffled to sequential learning on three tasks: the one-video representation learning method DoRA, standard VideoMAE on multi-video datasets, and the task of future video prediction.
Notes
Summary
2025 · Learning from Streaming Video with Orthogonal Gradients
Modify updates using problem structure. Subtract the component of the current gradient parallel to an EMA of past raw gradients, then feed the result into SGD/AdamW. Addresses correlated updates from sequential video.