[FEAT#82] neuron Phase 18 — large model + long training (Phase 17 plateau 미도달 한계 해소)

[juhy0987]

연관 이슈

• Closes [FEATURE] neuron Phase 18 — large model + long training (Phase 17 plateau 미도달 한계 해소) #82

구현 내용

Phase 17 (PR #81) 의 핵심 한계 (plateau 미도달 — 모든 mode drift -0.006) 해소를 위해 larger model (2x) + longer training (3x) 동시 적용. 사용자 명시 방향 반영.

코드 변경 없음

Phase 15/16/17 의 framework 이 모든 config 지원. 본 PR 의 검증 대상은 추가 scale + 시간의 영향 이므로 새 코드 없이 노트북만 추가.

Scale-up (Phase 17 대비)

항목	Phase 17	Phase 18	배수
hidden_dim	256	512	2x
n_heads	8	16 (head_dim=32 유지)	2x
n_layers	6	12	2x
ffn_dim (large)	512	1024	2x
max_steps	5000	15000	3x
approx params	~6.4M	~30-40M	~5-6x

신규 노트북

• notebooks/02-function-level/18-phase18-large-long.ipynb
• 5 mode × 2 seed = 10 run (Phase 17 동일 구성, scale 만 변경):
  • dense_large / static_prune_50 / DST_RigL_p16hp (control) / DST_RigL_p17hp / grown
• 자동 verdict 4건 (Phase 17 동일):
  • all-finite
  • DST_RigL_p16hp ≤ static + 0.02 (Phase 16a 가설 재검증)
  • grown ≤ dense_large + 0.02 (Phase 16b 가설 재검증)
  • plateau 도달 검증 (drift < -0.001 = true plateau) — Phase 18 의 핵심 verdict
• Phase 16/17/18 3단계 progression bar chart + dynamic advantage trend 정량화

Compute 추정

• ~30-40M params × 15K step × 5 mode × 2 seed = 10 run
• Phase 17 (6M × 5K × 10 run) 의 ~15x compute → 단일 GPU 4-6 시간 추정

CI / 머지 게이트 점검

변경 영향 범위

• 영향: notebooks/02-function-level/ (1 신규). 코드 / 테스트 변경 없음.
• 위험도: Low — 순수 노트북 추가

Required Status Checks

• Commit Lint / PR Title Lint / Linked Issue Check
• Format Check / Build / Test (221 tests, 변경 없음) / Lint (50 papers OK)

롤백 계획

• 노트북 추가만 — revert 영향 없음

Phase 18 후 시나리오 (사용자 결정 대기)

결과	Phase 19 후보
plateau 도달 + dynamic advantage 발현	결과 정리 + paper writing
plateau 도달 + dynamic 여전히 미실현	method 개선 (Net2WiderNet duplication, RigL scheduler) 또는 real dataset (OpenWebText)
plateau 여전히 미도달	learning rate scheduler 도입 / 더 큰 model 또는 더 긴 학습

GPU 사용자 측 실행 후 verdict 분석 → Phase 19 결정.

Summary by CodeRabbit

• Documentation
  • Added Phase 18 large-model long-training sweep notebook for benchmarking across multiple experiment modes and seeds.
  • Reports per-run metrics (loss, perplexity, parameter count, sparsity, DST cycles) and aggregates mean±std summaries.
  • Includes automated result validation (finiteness, tolerance checks, plateau detection) and loss-curve visualizations with annotated events and phase-to-phase comparisons.
  • Saves plots and provides conclusions and next steps for interpreting results.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fc527dd9-3820-4fc7-a6ae-189055097e69

📥 Commits

Reviewing files that changed from the base of the PR and between 0ea832b and c6dd0af.

⛔ Files ignored due to path filters (2)

• docs/figures/neuron-phase18/loss_curves.png is excluded by !**/*.png
• docs/figures/neuron-phase18/phase16_17_18_progression.png is excluded by !**/*.png

📒 Files selected for processing (1)

• notebooks/02-function-level/18-phase18-large-long.ipynb

🚧 Files skipped from review as they are similar to previous changes (1)

• notebooks/02-function-level/18-phase18-large-long.ipynb

---

📝 Walkthrough

Walkthrough

This PR adds a Phase 18 experimental notebook that scales model size and training length, runs a 5-mode × 2-seed sweep on TinyShakespeare, aggregates results with automated verdicts including plateau detection, visualizes loss curves with event markers, and compares Phase 16/17/18 final losses.

Changes

Phase 18 Large-Scale Training Sweep

Layer / File(s)	Summary
Introduction & environment setup notebooks/02-function-level/18-phase18-large-long.ipynb	Notebook overview and device setup; prints torch/version info and CPU warning.
Phase 18 hyperparams & mode definitions notebooks/02-function-level/18-phase18-large-long.ipynb	Loads TinyShakespeare dataset/tokenizer; defines Phase‑18 scaled architecture/training hyperparameters, DST/growth schedules, and enumerates 5 experiment modes.
Sweep execution (modes × seeds) notebooks/02-function-level/18-phase18-large-long.ipynb	Loops over modes and seeds, builds HybridTransformerTrainConfig per run (conditional DST/grow), invokes train_hybrid_transformer_lm, stores outputs, and prints per-run loss/perplexity/params/sparsity/DST cycles.
Result aggregation & verdict logic notebooks/02-function-level/18-phase18-large-long.ipynb	Prints per-run table; computes per-mode mean±std across seeds; runs automated checks: finiteness, DST_RigL ≤ static_prune tolerance, grown ≤ dense tolerance, and plateau detection using drift on last 500 losses.
Loss curves & Phase comparison notebooks/02-function-level/18-phase18-large-long.ipynb	Computes per-step rolling-mean losses per seed and plots mean±std trajectories per mode with vertical dynamic/DST markers; saves loss_curves.png. Builds Phase 16/17 baselines, plots grouped Phase 16/17/18 bar chart, computes dynamic advantage deltas, and saves comparison plot.
Conclusions and metadata notebooks/02-function-level/18-phase18-large-long.ipynb	Concluding/next-steps markdown and notebook metadata/end-of-file structure.

Sequence Diagram(s)

sequenceDiagram
  participant Notebook
  participant HybridTransformerTrainConfig
  participant train_hybrid_transformer_lm
  participant RunsStorage
  Notebook->>HybridTransformerTrainConfig: build config(mode, seed, DST/grow params)
  Notebook->>train_hybrid_transformer_lm: call with config
  train_hybrid_transformer_lm->>RunsStorage: save run outputs (losses, metrics, model)
  train_hybrid_transformer_lm-->>Notebook: return summary (final loss, params, sparsity)

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related issues

• #82: [FEATURE] neuron Phase 18 — large model + long training — This PR implements the Phase 18 experimental notebook described in the issue (larger model + longer training sweep, plateau checks, and dynamic advantage quantification).
• #80: Similar notebook-level use of HybridTransformerTrainConfig / train_hybrid_transformer_lm and DST/grow vs static_prune comparisons; Phase 18 extends the same sweep pattern at larger scale.

Possibly related PRs

• EinSofINTEREST/GraphLM#81: Phase 17 notebook that established the sweep framework, config structure, verdict logic, and mode enumeration reused and scaled in Phase 18.

Suggested labels

enhancement

Poem

🐰 In Phase eighteen I hop and count the runs,
More layers, longer steps, and broader suns,
I chart the losses, mark the DST’s end,
Compare the phases — numbers to befriend,
Hopping toward plateau, notebooks full of puns.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding Phase 18 notebook with larger model and longer training to address Phase 17 plateau limitations.
Linked Issues check	✅ Passed	The PR fully implements the primary objective from issue #82: a Phase 18 notebook with 5 modes × 2 seeds sweep using 2× scale-up (hidden_dim, heads, layers, ffn) and 3× longer training (15K steps), automated verdicts including plateau verification and dynamic-advantage quantification.
Out of Scope Changes check	✅ Passed	All changes are scoped to the new Phase 18 notebook and directly support the Phase 17 plateau resolution objective; no extraneous modifications to framework code or unrelated files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/#82/neuron-phase18-large-long

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request adds a new Jupyter notebook for Phase 18 of the neuron experiment, scaling up the model size and training steps to reach a true plateau. The review feedback highlights a logical issue in the plateau verification code, where a positive drift (representing diverging or increasing loss) would incorrectly be marked as a plateau. A bounded check is suggested to resolve this.

[Copilot]

Pull request overview

Phase 17 의 plateau 미도달 한계를 검증하기 위해 더 큰 Transformer 설정과 더 긴 학습 budget 을 적용한 Phase 18 실험 노트북을 추가합니다.

Changes:

• hidden/layer/FFN/step 수를 확대한 5 mode × 2 seed sweep 구성 추가
• all-finite, DST/static, grown/dense, plateau verdict 자동 산출
• Phase 16/17/18 progression 및 loss curve 시각화 추가

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@notebooks/02-function-level/18-phase18-large-long.ipynb`:
- Line 20: The plateau criterion text says "drift < -0.001" but the
implementation checks "mean_drift > -0.001", causing opposite behavior; update
the implementation so the check uses the same relation as the spec (replace
mean_drift > -0.001 with mean_drift < -0.001) or alternatively change the
descriptive text to match the implemented logic—ensure all occurrences of the
plateau test (variables drift, mean_drift and the threshold -0.001) use the same
sign and wording so the verdict is consistent.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9d09d259-7384-4188-9885-c377ab28bfb9

📥 Commits

Reviewing files that changed from the base of the PR and between 62035da and 0ea832b.

📒 Files selected for processing (1)

• notebooks/02-function-level/18-phase18-large-long.ipynb