[FEAT#80] neuron Phase 17 — scale-up validation (DST + grow advantage at larger scale)

[juhy0987]

연관 이슈

• Closes [FEATURE] neuron Phase 17 — scale-up validation (DST + grow advantage at larger scale) #80

구현 내용

Phase 16 (16a + 16b) 의 가설 검증 — paradigm 의 dynamic method (DST, Net2Net grow) 의 advantage 가 scale-up 으로 발현되는지 검증. Phase 16 의 negative result 가 scale 한계 로 결론났으니, 실제로 scale 을 키워 확인.

코드 변경 없음

Phase 15/16 의 framework 이 이미 모든 config 지원 (TrainConfig 의 hidden_dim/n_layers/ffn_dim/max_steps 등). Phase 17 의 검증 대상은 scale 자체의 영향 이므로 새 코드 없이 노트북만 추가.

Scale-up 변화 (Phase 16 대비)

항목	Phase 16	Phase 17	배수
hidden_dim	128	256	2x
n_layers	4	6	1.5x
ffn_dim (large)	256	512	2x
max_steps	1500	5000	3.3x
≈ params	~1M	~5M	~5x

신규 노트북

• notebooks/02-function-level/17-phase17-scale-up.ipynb
• 4 mode × 2 seed = 8 run:
  • dense_large (큰 모델 baseline, target reference)
  • static_prune_50 (Phase 15 at scale)
  • DST_RigL_50 (Phase 16a at scale, period=200/swap=5% — 16a 의 보수적 변형)
  • grown (Phase 16b at scale, ffn 256 → 512 at step 2500)
• 자동 verdict 4건:
  • all-finite
  • DST_RigL ≤ static + 0.02 (Phase 16a 가설 재검증)
  • grown ≤ dense_large + 0.02 (Phase 16b 가설 재검증)
  • plateau 도달 분석 (마지막 500 step drift)
• Phase 16 vs Phase 17 막대그래프 비교 (scale 효과 시각화)

DST hyperparameter 조정 의도

• Phase 16a: period=50, swap=10% (aggressive, swap overhead 큼)
• Phase 17: period=200, swap=5% (4x 보수적) — 16a 의 "swap 비용 > 이득" 진단에 따라 swap 빈도 감소, 학습 안정성 우선

Compute 추정

• 5000 step × 8 run ≈ 40K step-equivalents
• 5M params @ batch_size 32, block_size 64 → 단일 GPU 1-2시간 가정

CI / 머지 게이트 점검

변경 영향 범위

• 영향: notebooks/02-function-level/ (1 신규). 코드 / 테스트 변경 없음.
• 위험도: Low — 순수 노트북 추가, 기존 framework 사용

Required Status Checks

• Commit Lint / PR Title Lint / Linked Issue Check
• Format Check / Build / Test (221 tests, 변경 없음) / Lint (50 papers OK)

롤백 계획

• 노트북 추가만 — revert 영향 없음

Phase 17 후 다음 단계 (사용자 결정)

결과	후보
DST/Net2Net advantage 발현	Phase 18: 16a + 16b 결합 (grow + DST 동시)
여전히 미실현	Phase 18: dataset 확대 (OpenWebText subset) 또는 method 개선
명백한 우수성	scaling law 분석 + paper writing

GPU 사용자 측 실행 후 verdict 분석 + 다음 단계 결정.

Summary by CodeRabbit

• New Features
  • Added a Phase 17 scale-up experiment notebook to run and compare multiple training modes and seeds.
  • Automatically computes final loss, perplexity, parameter count, sparsity per run and aggregated mean±std.
  • Performs automated validation checks for stability, convergence, and mode comparisons.
  • Generates smoothed loss curves with uncertainty bands and vertical markers for key events.
  • Produces Phase 16 vs Phase 17 comparison charts and a brief conclusions/next-steps section.

[coderabbitai]

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (2)

• docs/figures/neuron/phase17/loss_curves.png is excluded by !**/*.png
• docs/figures/neuron/phase17/phase16_vs_phase17.png is excluded by !**/*.png

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2fcc7ba8-f000-47d1-b6e0-9a303f9175cb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a new Phase 17 Jupyter notebook that runs a 10-run scale-up sweep (5 modes × 2 seeds) on a larger Hybrid Transformer config using TinyShakespeare, constructs per-run HybridTransformerTrainConfig entries, runs training, computes per-mode statistics and automated verdicts, and produces loss-curve and Phase16-vs-Phase17 comparison plots.

Changes

Phase 17 Scale-up Validation Experiment

Layer / File(s)	Summary
Introduction and Environment Setup notebooks/02-function-level/17-phase17-scale-up.ipynb	Notebook adds Phase 17 description/hypotheses, environment imports (math/statistics/pathlib/matplotlib/torch), TinyShakespeare tokenizer/dataset loader, training runner/config, and device selection.
Scale-up Configuration and Mode Definitions notebooks/02-function-level/17-phase17-scale-up.ipynb	Defines hidden=256, n_layers=6, ffn=512, max_steps=5000, DYNAMIC_AT_STEP/DST_END_STEP, DST p16/p17 period/swap values, seeds, and MODES including dense_large, static_prune_50, DST_RigL_p16hp, DST_RigL_p17hp, and grown.
Experiment Sweep Execution notebooks/02-function-level/17-phase17-scale-up.ipynb	Loops over MODES×seeds, builds HybridTransformerTrainConfig per run (conditionally sets dst_end_step when regrow enabled), calls train_hybrid_transformer_lm, and records final metrics (loss, perplexity, param count, sparsity, DST cycles).
Result Analysis and Automated Verdicts notebooks/02-function-level/17-phase17-scale-up.ipynb	Prints per-run table, computes per-mode mean±std (loss, perplexity, params, sparsity), asserts all-final-loss finiteness, compares DST_RigL_p16hp vs static_prune_50 and grown vs dense_large using fixed loss-difference thresholds, reports DST_RigL_p17hp vs DST_RigL_p16hp delta, and computes per-run plateau-drift from the last 500 losses.
Loss Curves Visualization notebooks/02-function-level/17-phase17-scale-up.ipynb	Smooths per-seed loss traces with a rolling mean, aggregates mean±std per mode, plots mean±std bands with vertical markers at DYNAMIC_AT_STEP and DST_END_STEP, and saves loss_curves.png.
Phase 16 vs Phase 17 Comparison Visualization notebooks/02-function-level/17-phase17-scale-up.ipynb	Uses hardcoded Phase 16 baseline losses and computed Phase 17 final losses to create a paired bar chart, saves phase16_vs_phase17.png, and prints the DST_RigL_p17hp final loss.
Conclusion and Notebook Metadata notebooks/02-function-level/17-phase17-scale-up.ipynb	Adds a "결론 / 다음 단계" markdown section with evaluation questions and Phase 18 candidate directions, and finalizes notebook kernelspec/language metadata.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• EinSofINTEREST/GraphLM#78: Phase 16a RigL/DST regrow integration — Phase 17 notebook runs DST_RigL_p16hp/DST_RigL_p17hp using the same DST/regrow config fields.
• [FEAT#77] neuron Phase 16b — Net2Net function-preserving FFN grow (cross-shape expansion) #79: Phase 16b "grown" FFN growth mechanism — Phase 17 notebook exercises the grow path (FFN 256→512 at DYNAMIC_AT_STEP).

"I nibble data, token by token bright,
Five modes, two seeds, I raced each night,
Loss curves smoothed like moonlit fur,
Phase seventeen — results that stir,
A hop toward next experiments' light."

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title clearly summarizes the main change: adding Phase 17 scale-up validation experiments for dynamic methods (DST and grow) to test if advantages appear at larger model/training scale.
Linked Issues check	✅ Passed	PR adds the required Phase 17 notebook with 5 modes × 2 seeds (including Phase 16 hyperparameter control), scale-up configuration (hidden 256, 6 layers, FFN 512, 5000 steps), automated verdicts (finiteness, DST vs static, grown vs dense, plateau drift), and comparison plots, fully addressing issue #80 objectives.
Out of Scope Changes check	✅ Passed	PR contains only the single notebook addition as planned; no framework code changes, test modifications, or unrelated alterations outside the scope of Phase 17 scale-up validation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/#80/neuron-phase17-scale-up

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces a new Jupyter notebook (17-phase17-scale-up.ipynb) to evaluate the scale-up performance of dynamic model methods (DST, Net2Net grow) across four training modes and two seeds. The review feedback highlights several potential runtime and visualization issues when running with non-default configurations, such as low step counts or a single seed. Specifically, suggestions were made to safely calculate DST_END_STEP to avoid non-positive values, dynamically split the loss history for plateau analysis when steps are fewer than 500, and handle standard deviation calculations gracefully when only a single seed is provided.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@notebooks/02-function-level/17-phase17-scale-up.ipynb`:
- Around line 375-377: The current out_dir =
Path("../../runs/notebook-neuron-phase17") and subsequent fig.savefig calls use
a CWD-dependent path; change them to compute a repository-root–stable path
(e.g., determine the repo root via git repo discovery or a well-known repo root
variable and then set out_dir = repo_root / "runs" / "notebook-neuron-phase17")
and use that out_dir for fig.savefig of "loss_curves.png" and the later
"phase16_vs_phase17.png" so artifacts always save inside the repository
regardless of working directory; update the code that defines out_dir and the
two fig.savefig calls to reference the new repo-root–based out_dir.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: da0267b2-6ba2-44af-8da4-6838ffe22bdd

📥 Commits

Reviewing files that changed from the base of the PR and between 0879b14 and c9d2805.

📒 Files selected for processing (1)

• notebooks/02-function-level/17-phase17-scale-up.ipynb

[Copilot]

Pull request overview

Phase 17 scale-up 검증을 위해 기존 graphlm.neuron 학습 framework를 사용하는 신규 실험 노트북을 추가합니다. 코드 변경 없이 Phase 16의 DST/RigL 및 Net2Net grow 가설을 더 큰 모델/긴 학습 설정에서 재검증하는 목적입니다.

Changes:

• 4개 mode(dense_large, static_prune_50, DST_RigL_50, grown) × 2개 seed sweep 구성 추가
• scale-up 설정 및 자동 verdict/all-finite/plateau 분석 셀 추가
• loss curve 및 Phase 16 vs Phase 17 비교 시각화 추가