Working through how neural networks actually function — from a single multiply-and-add on paper to training and post-training a language model.
I already build with AI on the surface layer — Agentic Engineering. This repo is about the layer underneath that: not how to use the tools, but how they work and why. Everything here is built rather than read about, and every derivation gets checked numerically against code rather than taken on trust.
No deadline. Roughly 12–15 hours a week. Phases advance on a pass/fail exit criterion, never on elapsed time.
- Build first, pull the maths in when the implementation stalls. Never a maths block before the code.
- Pen and paper before an editor. Every operation gets hand-computed before it gets typed.
- Implement from scratch, then diff against a reference implementation.
- Roughly 80% hands-on, 20% reading.
- Where a derivation matters, hand-derive it and verify numerically — gradient checking, finite differences. The numerical check is the pass condition.
Handwritten notes are scanned in alongside the code, because the paper working is where the understanding actually happened.
Each advances only when its exit criterion is met.
| # | Phase | Exit criterion |
|---|---|---|
| 1 | Backprop | micrograd written blind, gradient checks matching finite differences to ~1e-6 |
| 2 | Training dynamics | diagnose a broken training run from the loss curve and gradient norms alone |
| 3 | Transformer | within 5% of nanoGPT val loss, written from a blank file |
| 4 | RL from scratch | PPO from scratch matching CleanRL on LunarLander-v2 |
| 5 | Scaling laws and data | justify parameter count, token count and data mixture from an own sweep |
| 6 | Distributed training | account for the gap between theoretical and achieved FLOPs |
| 7 | Post-training | base model to instruction-following, changes shown with evals |
| 8 | Inference | cached generation matching uncached output token-for-token at greedy decoding |
Phase 1 runs pen-and-paper for its first five lessons — the weighted sum, squashing, loss, sensitivity, and the chain rule — before any code gets written.
Each phase gets its own README recording what actually happened — including the things that didn't work, since those are usually where the learning was. Commit history is the progress log; there are no status tables to keep in sync.
- Karpathy, Neural Networks: Zero to Hero — phases 1–3
- Sutton & Barto, Reinforcement Learning: An Introduction — phase 4
- Spinning Up in Deep RL, CleanRL — phase 4 reference implementations
- Kaplan et al. 2020; Hoffmann et al. 2022 (Chinchilla) — phase 5
- ZeRO (Rajbhandari et al. 2019); Megatron-LM (Shoeybi et al. 2019) — phase 6
- InstructGPT (Ouyang et al. 2022); DPO (Rafailov et al. 2023) — phase 7
This is a learning log, not a library. The implementations here are deliberately naive — written to be understood rather than used. Anything useful for real work already exists elsewhere and is better.