[FEAT#77] neuron Phase 16b — Net2Net function-preserving FFN grow (cross-shape expansion) by juhy0987 · Pull Request #79 · EinSofINTEREST/GraphLM

juhy0987 · 2026-05-27T03:44:57Z

연관 이슈

Closes [FEATURE] neuron Phase 16b — Net2Net-style function-preserving grow (FFN expansion) #77 (Phase 16 메인 [FEATURE] neuron Phase 16 — dynamic topology evolution (RigL regrow + Net2Net grow) [메인] #75 의 sub 2/2)

구현 내용

Phase 16a (PR #78) 가 within-shape DST (sparsity 재할당, parameter 수 동일) 였다면, Phase 16b 는 cross-shape expansion — 학습 중 ffn_dim 을 확장하여 parameter 수 자체 증가. function preservation 보장 (Net2Net-style: 새 input weight = 0).

신규 모듈

src/graphlm/neuron/graph_hybrid.py 확장 (2 신규 메서드):
- grow_out(n_new_groups): G_out 차원 확장
  - 새 weight rows: 같은 fan_in 기반 작은 random init
  - 새 adj/mask = 1, 새 bias = 0
- grow_in(n_new_groups): G_in 차원 확장
  - 새 weight columns = 0 (function preservation 의 핵심)
  - 새 adj/mask = 1
src/graphlm/neuron/hybrid_transformer_demo.py 확장:
- TrainConfig 신규 2 인자: grow_at_step / grow_ffn_target + __post_init__ 검증
- _grow_ffn_in_model: 모델의 모든 HybridGraphFFN 확장 (fc1 grow_out + fc2 grow_in 동시)
- train loop: grow_at_step 에서 grow + optimizer 재생성 (parameter 객체 replace)
- result 에 grow_event + final_param_count 추가
notebooks/02-function-level/16-phase16b-net2net-grow.ipynb 신규
- 3 mode (small_baseline / grown / large_baseline) × 2 seed = 6 run
- small: ffn=128, grown: 128→256 at step 750, large: ffn=256
- 자동 verdict 4: all-finite / grown < small / grown ≤ large + 0.05 / function preservation spike 측정

Function preservation 메커니즘

FFN-style chain (fc1 → ... → fc2):

fc1.grow_out(n): ffn_dim 차원 추가 — 새 output 채널의 weight 는 작은 random
fc2.grow_in(n): 같은 차원 확장 — 새 input 채널의 weight = 0
결과: fc1 의 새 채널이 어떤 값을 출력하든, fc2 가 0 weight 로 받아 forward 영향 = 0
→ grow 직전과 직후의 forward 정확히 동일 (atol=1e-5, unit test 로 검증)

테스트 (213 → 221, +8 신규)

test_grow_out_shape_increased / test_grow_in_shape_increased: shape 정확성
test_grow_in_function_preservation: 새 input 위치에 임의 값을 채워도 기존 output 정확히 동일
test_grow_out_then_grow_in_downstream_preserves_forward: FFN-style chain (fc1 grow_out + fc2 grow_in) 의 forward 정확 보존 (핵심 verification)
test_grow_out_then_train_step_runs / test_grow_in_then_train_step_runs: grow 후 학습 가능
test_grow_negative_rejected: 입력 검증
test_grow_preserves_existing_prune_mask: Phase 15/16a 와 호환 (기존 pruned 위치 보존)
221 tests local all green

CPU smoke 결과

baseline (ffn=64): 37,856 params, final_loss 4.1249
grown (64 → 128 at step 10): 54,368 params (+44%), final_loss 4.1187 (function preservation 후 학습 정상 지속)

CI / 머지 게이트 점검

변경 영향 범위

src/graphlm/neuron/graph_hybrid.py (확장), hybrid_transformer_demo.py (확장), tests/neuron/test_graph_hybrid.py (확장), notebooks/02-function-level/ (1 신규)
위험도: Low — grow_at_step=None (default) 시 Phase 15/16a 와 동일 동작. opt-in.

Required Status Checks

Commit Lint / PR Title Lint / Linked Issue Check
Format Check / Build / Test (221 passed) / Lint

롤백 계획

grow_at_step 미사용 시 backwards-compat. revert 영향 없음.
단 parameter replace 시 optimizer state 손실은 정상 동작 (의도된 design).

Phase 16 의 메인 #75 완료 직전 단계

본 PR 머지 시 메인 #75 의 두 sub 모두 close:

✅ Sub [FEATURE] neuron Phase 16a — RigL/SET-style edge regrow (constant sparsity DST) #76 (16a RigL/SET DST): PR [FEAT#76] neuron Phase 16a — RigL/SET edge regrow (constant sparsity DST) #78 머지 완료
⏳ Sub [FEATURE] neuron Phase 16b — Net2Net-style function-preserving grow (FFN expansion) #77 (16b Net2Net grow): 본 PR

머지 후 paradigm 의 두 dynamic 방향 모두 검증 완료:

16a: within-shape (constant params, topology 재할당) — framework OK, perf advantage 미실현 (scale 한계)
16b: cross-shape (capacity 증가) — 본 scale 에서도 명확한 effect 기대 (capacity 부족 모델을 확장)

다음 단계 (Phase 17 후보)

16a + 16b 결합 — grow + shrink 동시 dynamic
layer-wise 차등 (attention vs FFN 별 다른 grow / prune 정책)
hidden_dim 까지 grow (downstream 영향 — 더 invasive)

Summary by CodeRabbit

New Features
- Dynamic Net2Net-style FFN dimension expansion during training, with configurable grow step and target size and automatic growth handling.
- Experiment notebook with automated stability checks, comparative summaries, training visualizations, and saved figures.
- Training run outputs now include growth event metadata and final parameter counts.
Tests
- Added tests covering expansion behavior, function preservation, post-growth training, input validation, and mask preservation.

…et2Net-style grow) - src/graphlm/neuron/graph_hybrid.py 확장 (2 신규 메서드): - grow_out(n_new_groups): G_out 차원 확장 (out_features 증가) - 새 weight rows: 같은 fan_in 기반 작은 random init - adj_outer/inner/edge_mask 새 rows = 1 - bias 새 entries = 0 - grow_in(n_new_groups): G_in 차원 확장 (in_features 증가) - **새 weight columns = 0 (function preservation 의 핵심)** - adj_outer/inner/edge_mask 새 columns = 1 - Parameter replace 방식 — caller (train loop) 가 optimizer 재생성 필요 - 8 신규 unit tests: - shape 정확성 (out / in 각각) - **function preservation 핵심 2건**: - grow_in: 새 input position 에 random 채워도 기존 output 정확히 동일 - grow_out + downstream grow_in: FFN-style chain (fc1 grow_out + fc2 grow_in) 의 forward 정확 보존 - grow 후 학습 가능성 (gradient 흐름) - 입력 검증 (negative / 0 거부) - prune mask 보존 (기존 pruned 위치 손실 없이 확장) - 213 → 221 tests, all green

- HybridTransformerTrainConfig 확장: - grow_at_step: int | None - grow_ffn_target: int | None (target ffn_dim) - __post_init__ 에 grow 인자 검증 (target > current, divisible by group_size, 일관성 둘 다 set/None) - _grow_ffn_in_model: 모델의 모든 HybridGraphFFN 의 ffn_dim 확장 - fc1.grow_out(n) + fc2.grow_in(n) 동시 (function preservation 보장) - train loop: grow_at_step 에서 _grow_ffn_in_model 실행 + optimizer 재생성 (parameter replace 됐으므로) - result 에 grow_event + final_param_count 추가 - smoke test 검증: ffn 64→128 grow 직후 학습 정상 지속, params 37856→54368 (+44%), loss 거의 유지 (function preservation 확인)

- notebooks/02-function-level/16-phase16b-net2net-grow.ipynb 신규 - 3 mode × 2 seed = 6 run: - small_baseline: ffn=128 끝까지 (capacity 부족) - grown: ffn=128 시작 → step 750 에서 256 으로 grow (function-preserving) - large_baseline: ffn=256 끝까지 (large capacity reference) - arch 고정: hybrid_around_one_around_one + use_full_graph=True - 자동 verdict 4가지: - all-finite (grow stability) - grown < small_baseline (grow advantage) - grown ≤ large_baseline + 0.05 (capacity recovery) - function preservation 정성 검증 (grow 직후 spike 측정) - loss curve (grow step 수직선) + params vs loss trade-off plot

coderabbitai · 2026-05-27T03:45:08Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 025f119a-4fe7-46f1-889b-f528a3f7a6bb

📥 Commits

Reviewing files that changed from the base of the PR and between be3cd24 and 171b462.

⛔ Files ignored due to path filters (2)

docs/figures/neuron/phase16b/loss_curves.png is excluded by !**/*.png
docs/figures/neuron/phase16b/params_vs_loss.png is excluded by !**/*.png

📒 Files selected for processing (2)

src/graphlm/neuron/graph_hybrid.py
src/graphlm/neuron/hybrid_transformer_demo.py

🚧 Files skipped from review as they are similar to previous changes (1)

src/graphlm/neuron/hybrid_transformer_demo.py

📝 Walkthrough

Walkthrough

Adds Phase 16b Net2Net-style FFN expansion: HybridGraphLinear gains grow_out/grow_in, training config and orchestration perform coordinated FFN growth at a specified step (recreating the optimizer and recording grow_event), unit tests validate behavior, and a notebook runs a sweep evaluating function-preservation and training dynamics.

Changes

Net2Net-style FFN expansion with function preservation

Layer / File(s)	Summary
HybridGraphLinear grow_out and grow_in methods and unit tests `src/graphlm/neuron/graph_hybrid.py`, `tests/neuron/test_graph_hybrid.py`	Adds `_grow_param`, `grow_out(n_new_groups)` and `grow_in(n_new_groups)` to expand output/input group dimensions by concatenating new tensor regions (weights, routing `adj_*`, `edge_mask`, optional bias). Unit tests confirm shape updates, forward preservation (single & chained layers), gradient/backprop compatibility, input validation, and preservation of existing pruned mask entries.
Training config, validation, FFN growth helper, and training loop integration `src/graphlm/neuron/hybrid_transformer_demo.py`	`HybridTransformerTrainConfig` adds `grow_at_step` and `grow_ffn_target` with validation. `_grow_ffn_in_model` applies coordinated `fc1.grow_out` + `fc2.grow_in` to each `HybridGraphFFN`, returns per-layer growth metadata. Training triggers growth at `grow_at_step`, recreates AdamW optimizer if layers grew, and returns `grow_event` and `final_param_count`.
Phase 16b experimental validation notebook `notebooks/02-function-level/16-phase16b-net2net-grow.ipynb`	New notebook runs a three-mode sweep (small_baseline, grown, large_baseline) across seeds using `train_hybrid_transformer_lm`, records per-run metrics and grow_event, evaluates automated verdicts for stability and function-preservation, and writes loss-curve and params-vs-loss plots.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

[FEATURE] neuron Phase 16b — Net2Net-style function-preserving grow (FFN expansion) #77: Directly implements Phase 16b Net2Net-style FFN expansion (grow_out/grow_in, train config, and experimental sweep).

Possibly related PRs

EinSofINTEREST/GraphLM#72: Related work on edge_mask-based pruning that Phase 16b's grow methods expand and initialize during growth.
EinSofINTEREST/GraphLM#66: Prior changes extending HybridGraphLinear that Phase 16b builds upon for parameter-expansion logic.
EinSofINTEREST/GraphLM#68: Related integration of hybrid transformer training stack that Phase 16b extends with grow-at-step orchestration.

Suggested labels

enhancement

Poem

A rabbit pads through tensors late at night,
Appending rows and columns out of sight,
It keeps the function steady, neat and bright,
Watching losses settle, growth taking flight—
Hoppity hops, the FFN grows just right. 🐇✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title directly and specifically describes the main change: implementing Phase 16b Net2Net-style function-preserving FFN growth with cross-shape expansion. It accurately captures the core feature from the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feature/#77/neuron-phase16b-net2net-grow

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

This pull request implements Phase 16b cross-shape expansion (Net2Net-style FFN growth) to dynamically increase model capacity during training while ensuring function preservation. It introduces grow_out and grow_in methods in HybridGraphLinear (where new input weights are initialized to zero to preserve output), integrates this growth mechanism into the training loop within hybrid_transformer_demo.py (including optimizer re-initialization), adds a demonstration notebook, and includes comprehensive unit tests to verify correctness, shape changes, and gradient flow. No review comments were provided, so there is no feedback to address.

Copilot

Pull request overview

이 PR은 neuron Phase 16b로, 학습 중 FFN 차원을 Net2Net-style (function-preserving) 로 확장할 수 있도록 HybridGraphLinear에 grow_out/grow_in을 추가하고, 데모 학습 루프에서 지정 step에 확장을 트리거해 cross-shape expansion 실험을 가능하게 합니다.

Changes:

HybridGraphLinear.grow_out/grow_in 추가로 G_out/G_in 차원 확장 지원(특히 grow_in은 새 input weight=0으로 function preservation 핵심 구현)
HybridTransformerTrainConfig에 grow 관련 옵션 추가 및 train loop에서 grow step 처리(+ optimizer 재생성, 결과에 grow_event/param_count 기록)
shape/보존성/학습 가능성에 대한 테스트 및 Phase 16b 실험 노트북 추가

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
`src/graphlm/neuron/graph_hybrid.py`	`HybridGraphLinear`에 `grow_out/grow_in` 구현 추가(파라미터/버퍼 확장 로직)
`src/graphlm/neuron/hybrid_transformer_demo.py`	grow 설정 검증 + 모델 내 FFN 확장 헬퍼/학습 루프 grow 트리거 및 로깅 추가
`tests/neuron/test_graph_hybrid.py`	grow의 shape 변화, function preservation, grow 후 backward 동작 등을 검증하는 테스트 추가
`notebooks/02-function-level/16-phase16b-net2net-grow.ipynb`	small/grown/large 3모드×2시드 sweep 및 자동 verdict/시각화 추가

…grow no-op - Copilot #3308323611 + #3308323627 (grow_out + grow_in 동일 이슈): - .data 사용 비권장 + requires_grad 상태 (freeze 한 layer) 가 Parameter replace 시 unfreeze 되는 버그 → _grow_param helper 추가 (detach().clone() + 기존 requires_grad 복원), grow_out / grow_in 둘 다 사용 - Copilot #3308323642: - plain arch / HybridGraphFFN 없는 경우 _grow_ffn_in_model 의 n_layers_grown=0 인데도 optimizer 재생성 + grow_event 기록 (무용 + 주석 불일치) → n_layers_grown > 0 일 때만 optimizer 재생성 + grow_event. plain arch 시 진정한 no-op. - 18 grow tests still green, lint OK

juhy0987 added 3 commits May 27, 2026 12:38

Copilot AI review requested due to automatic review settings May 27, 2026 03:44

Copilot started reviewing on behalf of juhy0987 May 27, 2026 03:45 View session

juhy0987 added the enhancement New feature or request label May 27, 2026

gemini-code-assist Bot reviewed May 27, 2026

View reviewed changes

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread src/graphlm/neuron/graph_hybrid.py

Comment thread src/graphlm/neuron/graph_hybrid.py

Comment thread src/graphlm/neuron/hybrid_transformer_demo.py Outdated

Comment thread notebooks/02-function-level/16-phase16b-net2net-grow.ipynb

juhy0987 self-assigned this May 27, 2026

[CHORE]: Phase 16b figure 자산 추가 (Notion 임베드용)

171b462

juhy0987 merged commit 0879b14 into main May 28, 2026
10 checks passed

juhy0987 mentioned this pull request May 28, 2026

[FEATURE] neuron Phase 17 — scale-up validation (DST + grow advantage at larger scale) #80

Closed

4 tasks

coderabbitai Bot mentioned this pull request May 28, 2026

[FEAT#80] neuron Phase 17 — scale-up validation (DST + grow advantage at larger scale) #81

Merged

2 tasks

Navigation Menu

Uh oh!

[FEAT#77] neuron Phase 16b — Net2Net function-preserving FFN grow (cross-shape expansion) - #79

[FEAT#77] neuron Phase 16b — Net2Net function-preserving FFN grow (cross-shape expansion)#79
juhy0987 merged 5 commits into
mainfrom
feature/#77/neuron-phase16b-net2net-grow

juhy0987 commented May 27, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 27, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Possibly related PRs

Suggested labels

Poem

❌ Failed checks (1 warning)

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

juhy0987 commented May 27, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

연관 이슈

구현 내용

신규 모듈

Function preservation 메커니즘

테스트 (213 → 221, +8 신규)

CPU smoke 결과

CI / 머지 게이트 점검

변경 영향 범위

Required Status Checks

롤백 계획

Phase 16 의 메인 #75 완료 직전 단계

다음 단계 (Phase 17 후보)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related issues

Possibly related PRs

Suggested labels

Poem

❌ Failed checks (1 warning)

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

juhy0987 commented May 27, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 27, 2026 •

edited

Loading