fast-weights

Official repository for the paper "A Modern Self-Referential Weight Matrix That Learns to Modify Itself" (ICML 2022 & NeurIPS 2021 Deep RL Workshop) and "Accelerating Neural Self-Improvement via Bootstrapping" (ICLR 2023 Workshop)

Implementation of the paper [Using Fast Weights to Attend to the Recent Past](https://arxiv.org/abs/1610.06258)

Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers" (NeurIPS 2021)

Implementation of Fast Weight Attention

Official repository for the paper "Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules" (NeurIPS 2022)

PyTorch Language Modeling Toolkit for Fast Weight Programmers

Official repository for the paper "Automating Continual Learning"

Weights that never freeze. A continual-learning architecture, and the pre-registered experiment that falsified its central claim. https://arxiv.org/abs/2607.20792

Official repository for the paper "Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules" (ICLR 2023)

Your language model forgets everything. Sillage gives it a 4 MB memory that persists across sessions, no gradients, no fine-tuning, no growing index. Beats an unbounded kNN-LM at 1/13 the storage (perplexity 31 → 17). CPU-only, fully reproducible, four preprints with Zenodo DOIs.

Usage kit for the Fractale models — LMs whose long-term memory is a self-written fast-weight thought bank

PyTorch Implementation of the paper [Using Fast Weights to Attend to the Recent Past]

PyTorch implementation of DCT fast weight RNNs

Distributed Engrammics — gradient-free transfer of a persistent fast-weight associative state between model instances (DeltaNet, RWKV-7). Paper + reproducible code.

Inference-time drift experiment demonstrating reduced repetition collapse in frozen LLMs.

Matrix-valued fast-weight state as a representational medium for language models: measurable rank, super-linear capacity, exact in-context composition, and the measurement methodology behind the results (negative results included).

Naturally fractal, self-modifying language model research with persistent in-weights memory.

Two fast-weight experiments (FENRIR and FREKI) on whether a cheap recurrent layer can chain multi-hop lookups inside one forward pass, without attention. Honest results, builds on prior art.

Official implementation of Rethinking Expressivity and Efficiency in Test-Time Training