tinystories

Codebase for Linguistic Collapse: Neural Collapse in (Large) Language Models [NeurIPS 2024] [arXiv:2405.17767]

Implementation of a small GPT-style transformer from scratch in PyTorch. Learn how Large Language Models work by building, training, generating text, and visualizing attention.

Train GPT style model on tinystories dataset

Code implementation for our paper "BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training" as part of the 2024 BabyLM Challenge

Small transformer trained from scratch

LLM Architecure, Open Source

Specialized agentic coding LLM — 3B, pre-trained from scratch, runs locally on Apple Silicon. No API key, no data leaving your machine, zero cost per call.

A 110M-parameter Llama-style transformer trained from scratch on the TinyStories dataset, optimized for high-throughput training on 4GB VRAM consumer GPUs. The project features a custom asynchronous CUDA-stream prefetcher and KV-cache inference, achieving 10k+ TPS on an RTX 3050.

How small can English get? 8 tiny language models (fp16 vs ternary) trained from scratch on TinyStories, judged by a frozen LLM judge, and running live in your browser.

StoryByte — a ~1M-parameter GPT trained from scratch on TinyStories. The model behind the Let's Data Science 'Build a Tiny LLM — From Tokens to Text' course; its whole forward pass runs in ~40 lines of NumPy in a browser.

Built a GPT-2 architecture from scratch in PyTorch and trained it on the TinyStories dataset. The result is a lightweight model that generates coherent, short form children's stories.

Byte Latent Transformer (BLT) LLM built from scratch in PyTorch — tokenizer-free, byte-level, trained end-to-end on TinyStories to 0.71 BPB. Clone & run.

A PyTorch implementation of a Bigram Language Model using Transformer architecture for character-level text generation.

Minimal GPT-style language model trained on Apple Silicon (MPS)

A 30M-parameter transformer written from scratch and trained overnight on a MacBook. TinyStories, 0.74B tokens, val loss 1.1527. No transformers library.

Developed and compared Logistic Regression and fine-tuned DistilBERT models for multilabel text classification.

Train a tiny GPT from scratch in plain PyTorch, with every transformer concept explained directly in the README.

A modern decoder-only Transformer (RoPE, RMSNorm, SwiGLU, KV-cache) implemented and trained from scratch in PyTorch on TinyStories, with a copy-task smoke test.

A small GPT-style language model trained from scratch on TinyStories.