Variance-Covariance Regularization Improves Representation Learning

Minimal Implimentation of VCRec (2024) for collapse prevention.

/layer.py Direct Gradient Manipulation implimentation of VCReg. Put this between your blocks. Assumes 0 mean.

/loss.py VICReg loss, for applying this regularization to an output representation.

Original Paper:

@misc{zhu2024variancecovariance,
title={Variance-Covariance Regularization Improves Representation Learning}, author={Jiachen Zhu and Katrina Evtimova and Yubei Chen and Ravid Shwartz-Ziv and Yann LeCun},
year={2024},
eprint={2306.13292},
archivePrefix={arXiv},
primaryClass={cs.LG}
}