Paper Replication: LoRA (Hu et al., 2022)

This repository reproduces the core setup of LoRA: Low-Rank Adaptation of Large Language Models on compact Transformer classifiers using a from-scratch LoRA implementation.

1. Paper Summary

• Problem: Full fine-tuning updates all model parameters, which is memory-heavy and expensive.
• Method: Freeze backbone weights and learn low-rank update matrices (A, B) injected into linear layers.
• Contribution: LoRA achieves near full fine-tuning performance with dramatically fewer trainable parameters.

2. Reproduction Setup

• Frameworks: PyTorch + Hugging Face Transformers + Datasets
• Logging: MLflow (local)
• Determinism: Fixed seeds, deterministic CUDA flags where possible
• Base model: distilbert-base-uncased
• Tasks:
  • SST-2 (binary sentiment)
  • AG News (4-way topic classification)
• Baselines:
  • full_finetune: train all parameters
  • frozen_backbone: train classification head only
  • lora: freeze backbone, train LoRA adapters (+ classification head).

3. Repository Structure

.
├── configs/ # YAML experiment configs
├── models/ # LoRA + baseline model setup
├── trainer/ # Training loop and optimization logic
├── data/ # Dataset loading & preprocessing
├── evaluation/ # Metrics
├── experiments/ # Run scripts
├── notebooks/ # Jupyter notebook implementation
├── README.md
└── reproducibility_report.md

4. One-Command Reproducibility

Install dependencies:

pip install -r requirements.txt

Run two-rank LoRA comparison (SST-2):

python experiments/run.py --configs configs/sst2_lora_r4.yaml configs/sst2_lora_r8.yaml

Run multi-baseline comparison (AG News):

python experiments/run.py --configs configs/agnews_frozen.yaml configs/agnews_full.yaml configs/agnews_lora_r4.yaml

Artifacts are written to outputs/ and MLflow logs to mlruns/.

Notebook workflow (for Jupyter users):

jupyter notebook notebooks/lora_replication.ipynb

5. Results Table (Fill After Running)

Task	Method	Main Metric	Paper	This Repo	Delta vs Paper
SST-2	LoRA (r=8)	Accuracy	95.1*	TBD	TBD
SST-2	Full FT	Accuracy	N/A	TBD	N/A
AG News	LoRA (r=4)	Accuracy	N/A	TBD	N/A
AG News	Frozen Backbone	Accuracy	N/A	TBD	N/A

* LoRA paper reports strong GLUE performance; exact numbers depend on backbone/task variant and training details.

6. Gap Analysis (Paper Ambiguities)

Missing Detail in Paper	Assumption in This Reproduction	Observed / Expected Impact
Exact preprocessing for non-GLUE text tasks	Standard HF tokenization, truncation to fixed max length	Can shift reported accuracy by ~0.2–1.0 points
Layer placement specifics for smaller encoder models	Inject LoRA into attention q_lin/v_lin for DistilBERT	Adapter placement materially affects efficiency/quality
Complete hyperparameter sweeps per task	Fixed learning rates and batch sizes per config	Can underperform best-case paper settings
Seed protocol across multiple trials	Single-seed deterministic runs by default	Variance may remain under-reported

7. Optional Reference Comparison

After from-scratch runs, compare parameter counts against PEFT:

python experiments/compare_reference_peft.py --config configs/sst2_lora_r8.yaml

This does not replace the manual LoRA implementation; it only validates design decisions.