[FEAT#71] neuron Phase 15 — sparsity-driven edge prune (정적 → 동적 위상 진입) by juhy0987 · Pull Request #72 · EinSofINTEREST/GraphLM

juhy0987 · 2026-05-26T23:46:34Z

연관 이슈

Closes [FEATURE] neuron Phase 15 — sparsity-driven edge prune (training-time dynamic param count 의 첫 실질 단계) #71

구현 내용

Phase 14 (PR #70) 까지는 dense topology + learned magnitude 의 정적 위상이었음. Phase 15 는 edge 자체를 영구 제거 하는 첫 동적 위상 단계 — paradigm 의 ultimate goal (training-time dynamic parameter count) 의 실질적 시작.

신규 모듈

src/graphlm/neuron/graph_hybrid.py 확장
- edge_mask: register_buffer (학습 X, state_dict 포함), shape (G_out, G_in, k, k), 초기값 모두 1
- forward 에 edge_mask 곱셈 추가 → pruned edge 의 기여도 0 + gradient chain 차단 (resurrection 방지)
- effective_edge_magnitude(): 현재 forward 적용 magnitude
- prune_by_magnitude(threshold) / prune_bottom_fraction(fraction): 영구 0 처리
- effective_sparsity() / n_alive_edges(): 측정 helper
src/graphlm/neuron/hybrid_transformer_demo.py 확장
- HybridTransformerTrainConfig.prune_at_step / prune_fraction
- train loop 에 prune step 추가, result['final_sparsity'] + result['prune_event'] 반환
notebooks/02-function-level/14-phase15-sparsity-prune.ipynb 신규
- 4 prune fraction (dense, 30%, 50%, 70%) × 2 seed = 8 run
- arch: hybrid_around_one_around_one + use_full_graph=True (Phase 14 최저 loss 구조 고정)
- prune 시점: max_steps/2 (학습 중간)
- 자동 verdict 4가지: all-finite / soft degradation / monotonic / moderate prune ≈ dense
- loss curve (prune step 수직선) + sparsity vs final_loss trade-off plot

gradient resurrection 방지 메커니즘

eff_w = adj_outer · adj_inner · W · edge_mask

mask=0 위치에서 forward 기여 0 + chain rule 로 adj_outer, adj_inner, weight 모두 gradient 0 → optimizer 가 살릴 수 없음. unit test test_pruned_edges_do_not_resurrect_via_gradient 로 명시적 검증.

테스트 (182 → 202, +10 신규 prune 테스트)

test_edge_mask_initial_all_ones — 초기 상태
test_forward_with_initial_mask_unchanged — function preservation
test_prune_by_magnitude_basic / test_prune_idempotent_below_threshold — threshold prune
test_pruned_edges_do_not_resurrect_via_gradient — gradient 차단 (핵심)
test_prune_bottom_fraction / test_prune_bottom_fraction_zero_fraction_noop — top-k 류
test_prune_negative_threshold_rejected / test_prune_invalid_fraction_rejected — 입력 검증
test_edge_mask_in_state_dict — save/load 보존
192 tests local all green (CPU smoke: 30% prune → 30% sparsity, loss 차 0.008)

0-init 금지 + magnitude rule 일관 적용 (변경 없음)

Phase 12 부터의 0-init 금지 규칙 그대로 유지. prune 은 학습 후 edge 가 자연히 작은 것들을 제거 — 처음부터 0 으로 시작하는 것과 다름.

CI / 머지 게이트 점검

변경 영향 범위

영향 패키지/모듈: src/graphlm/neuron/graph_hybrid.py (확장), src/graphlm/neuron/hybrid_transformer_demo.py (확장), tests/neuron/test_graph_hybrid.py (확장), notebooks/02-function-level/ (1 신규)
위험도: Low — edge_mask 는 초기값 1 이라 prune 호출 없으면 Phase 14 forward 와 정확히 동일 (function preservation). prune 은 opt-in (config 의 prune_at_step is None 시 noop).

Required Status Checks

롤백 계획

edge_mask 미사용 시 Phase 14 와 backwards-compat (default no-op). revert 시 영향 없음.
단 state_dict 에 edge_mask buffer 추가됨 — 기존 checkpoint 와 strict load 시 key mismatch 가능. strict=False 또는 마이그레이션 필요 (현재 본 프로젝트는 체크포인트 저장 안 함).

Phase 15 의 paradigm 의미 — 정적 → 동적 위상 진입

단계	위상	채널 경로
8-11	MLP-LM 위 graph 표현	dense, 동일
12	HybridGraphLinear 통합 module	dense, 동일
13	Transformer FFN graph	dense, 동일
14	block 전체 graph	dense, 동일
15	edge prune	dynamic — 채널마다 effective 경로 길이 다름

이제 채널마다 다른 edge fan-out / fan-in 발생 — paradigm 의 진짜 dynamic phase 시작.

다음 단계 (Phase 16 후보)

Net2Net / LiGO 식 grow — pruned slot 에 신규 edge 추가, dynamic grow + shrink 동시
RigL / SET dynamic sparse training — prune 후 같은 수의 edge 를 다른 위치 재할당 (constant sparsity 유지)
layer-wise 차등 prune — attention vs FFN 별 다른 sparsity target

Summary by CodeRabbit

New Features
- Added configurable edge pruning for graph transformers (prune-at-step and prune-fraction) and runtime sparsity reporting.
- Added a Phase‑15 sparsity-pruning notebook with training sweeps, loss-curve and sparsity-tradeoff visualizations and automated summaries.
Tests
- Added extensive tests for edge-mask behavior, magnitude- and fraction-based pruning, gradient masking, validation of pruning parameters, and state persistence.

- src/graphlm/neuron/graph_hybrid.py 확장 - edge_mask: register_buffer (학습 X, state_dict 포함), 초기값 모두 1 (no prune) - forward 에 edge_mask 곱셈 추가 → pruned edge 의 기여도 0 + gradient chain 차단 - effective_edge_magnitude(): 현재 forward 적용 magnitude (mask 곱한 상태) - prune_by_magnitude(threshold): |adj·W| < threshold 영구 0 - prune_bottom_fraction(fraction): 살아있는 edge 중 하위 fraction 영구 0 - effective_sparsity(): 영구 prune 비율 - n_alive_edges(): 살아있는 edge 수 - 10 신규 unit tests - edge_mask 초기 / forward 기능 보존 / prune 정확성 - **gradient resurrection 방지 검증** (pruned 위치의 weight/adj_inner grad == 0) - state_dict 보존 (save/load 후 sparsity 유지) - 입력 검증 (negative threshold, invalid fraction) - 182 → 192 tests, all green

- HybridTransformerTrainConfig 에 prune_at_step / prune_fraction 필드 추가 - _prune_model: 모든 HybridGraphLinear 에 prune_bottom_fraction 일괄 적용 - _model_sparsity: 전체 HybridGraphLinear edge 평균 sparsity - train loop: prune_at_step 도달 시 1회 prune 실행, prune_event 기록 - result 에 final_sparsity / prune_event 추가 - smoke test 검증: baseline 0% sparsity vs prune 30% sparsity, loss 차 작음 (0.008)

- notebooks/02-function-level/14-phase15-sparsity-prune.ipynb 신규 - 4 prune fraction (dense, 30%, 50%, 70%) × 2 seed = 8 run - prune 시점: max_steps/2 = 750 (학습 중간) - arch 고정: hybrid_around_one_around_one + use_full_graph=True (Phase 14 최저 loss 구조) - 자동 verdict 4가지: all-finite / soft degradation (70% ≤ dense+1.0) / monotonic loss / moderate (30%) ≈ dense - loss curve (prune step 수직선 표시) + sparsity vs final_loss trade-off plot - ruff format 적용 (graph_hybrid / test_graph_hybrid)

coderabbitai · 2026-05-26T23:46:45Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 811769cf-6f98-49eb-af95-4a587a0dcab5

📥 Commits

Reviewing files that changed from the base of the PR and between 1802964 and c549022.

⛔ Files ignored due to path filters (2)

docs/figures/neuron/phase15/loss_curves.png is excluded by !**/*.png
docs/figures/neuron/phase15/sparsity_tradeoff.png is excluded by !**/*.png

📒 Files selected for processing (4)

src/graphlm/neuron/graph_hybrid.py
src/graphlm/neuron/hybrid_transformer_demo.py
tests/neuron/test_graph_hybrid.py
tests/neuron/test_hybrid_transformer_demo.py

🚧 Files skipped from review as they are similar to previous changes (2)

src/graphlm/neuron/graph_hybrid.py
src/graphlm/neuron/hybrid_transformer_demo.py

📝 Walkthrough

Walkthrough

Adds persistent per-edge masks and magnitude/fraction pruning APIs to HybridGraphLinear, integrates a one-shot prune trigger into the training helper (recording a prune_event), validates behavior with unit tests, and supplies a notebook that sweeps prune fractions and seeds while plotting and auto-evaluating results.

Changes

Phase 15 Sparsity-Driven Edge Pruning

Layer / File(s)	Summary
Edge Pruning Core API & Validation `src/graphlm/neuron/graph_hybrid.py`, `tests/neuron/test_graph_hybrid.py`	`HybridGraphLinear` registers persistent `edge_mask` (all ones initially) and applies it in forward. Adds `effective_edge_magnitude()`, `prune_by_magnitude(threshold)`, `prune_bottom_fraction(fraction)`, `effective_sparsity()`, and `n_alive_edges()`. Tests cover initialization, forward equivalence, zero-mask output, magnitude/fraction pruning semantics, idempotency, gradient blocking at pruned positions, deterministic tie-breaking, parameter validation, and `state_dict` persistence.
Training Integration & Orchestration `src/graphlm/neuron/hybrid_transformer_demo.py`	`HybridTransformerTrainConfig` adds `prune_at_step` and `prune_fraction` with validation. Training helper adds `_prune_model` and `_model_sparsity`, advances a step counter, triggers a one-shot prune when configured, and returns a `prune_event` dict containing step, edges pruned, and post-prune sparsity.
Phase 15 Experimental Validation `notebooks/02-function-level/14-phase15-sparsity-prune.ipynb`	New notebook runs a 4×2 sweep across prune fractions (0.0, 0.3, 0.5, 0.7) and seeds, aggregating mean ± std final loss and sparsity, applying automated verdict checks (finite losses and fixed numeric thresholds vs dense baseline), and saving loss-curve and sparsity-vs-final-loss plots.
Training Config Tests `tests/neuron/test_hybrid_transformer_demo.py`	Unit tests validate default pruning settings, rejection of invalid `prune_fraction` and `prune_at_step` values, and acceptance/storage of valid pruning config.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

EinSofINTEREST/GraphLM#66: Earlier work on HybridGraphLinear that this PR extends with edge-mask and pruning APIs.
EinSofINTEREST/GraphLM#68: Prior changes to hybrid_transformer_demo.py; Phase 15 builds on the same training helper and config surface.

🐰 In quiet code gardens edges fall away,

Masks hold the memory of paths gone gray,
Prune the small whispers, keep the strong tone,
A smaller map, yet the model still hums on.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately references Phase 15 sparsity-driven edge pruning as the main feature and includes issue reference [FEAT#71].
Linked Issues check	✅ Passed	The PR implements all key coding objectives: edge_mask buffer and pruning APIs [`#71`], gradient-blocking prevention [`#71`], sparsity tracking helpers [`#71`], one-shot pruning in training loop [`#71`], validation/tests [`#71`], and notebook sweep [`#71`].
Out of Scope Changes check	✅ Passed	All changes are directly aligned with Phase 15 objectives—edge pruning mechanism, training integration, test coverage, and experimental notebook.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feature/#71/neuron-phase15-sparsity-prune

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

This pull request introduces dynamic edge pruning (Phase 15) to the hybrid graph transformer model. It registers an edge_mask buffer to permanently prune edges by magnitude or bottom fraction, preventing gradient resurrection. The review comments correctly identify a critical tie-breaking issue in prune_bottom_fraction where identical magnitude values could lead to over-pruning, and suggest using torch.topk for deterministic pruning. A corresponding update to the test suite is also suggested to enforce exact pruned count matching.

Copilot

Pull request overview

이 PR은 Phase 14의 “dense topology + learned magnitude” 상태에서 한 단계 나아가, 학습 중 edge 자체를 영구적으로 제거(prune) 하는 Phase 15 동적 위상 진입을 구현합니다. 핵심은 HybridGraphLinear에 edge_mask 버퍼를 도입해 forward 기여도를 0으로 만들고, 동일 곱셈 경로를 통해 pruned edge의 gradient resurrection을 차단하는 것입니다.

Changes:

HybridGraphLinear에 edge_mask(buffer)와 magnitude 기반 prune API 및 sparsity 측정 헬퍼 추가
데모 학습 루프에 one-shot prune 훅 추가 및 결과에 sparsity/prune 이벤트 리포팅 추가
prune 동작/상태 저장/gradient 차단을 검증하는 테스트 추가 + Phase 15 실험 노트북 추가

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
`src/graphlm/neuron/graph_hybrid.py`	`edge_mask` 적용 및 prune/sparsity 관련 메서드 추가
`src/graphlm/neuron/hybrid_transformer_demo.py`	학습 중 prune 실행 및 `final_sparsity`/`prune_event` 반환 추가
`tests/neuron/test_graph_hybrid.py`	edge_mask 초기값, prune 동작, gradient 차단, state_dict 보존 테스트 추가
`notebooks/02-function-level/14-phase15-sparsity-prune.ipynb`	prune fraction sweep 실험/시각화 노트북 추가

- gemini #3307531740 (HIGH): prune_bottom_fraction 의 kthvalue+mag<=kth 가 동률 시 의도보다 많이 prune 위험 (극단적으로 100%) → torch.topk(largest=False) 로 하위 n 개 정확 인덱스 추출 후 mask=0. deterministic. - gemini #3307531745: 테스트의 ±10 tolerance → topk 정확성으로 `==` 강화. 추가로 모든 magnitude 동률 시나리오 신규 테스트 (tie-breaking deterministic 검증). - Copilot #3307536521: test_forward_with_initial_mask_unchanged 가 이름과 검증 불일치 → mask=1 forward 를 mask 없는 직접 계산과 정확 비교 (function preservation 명시). - Copilot #3307536553: HybridTransformerTrainConfig 의 prune 인자 silent no-op 위험 → __post_init__ 추가, prune_fraction ∈ [0,1] / prune_at_step ∈ [1, max_steps] 강제. test_hybrid_transformer_demo.py 신규 (9 validation tests). - 192 → 202 tests, all green

juhy0987 added 3 commits May 27, 2026 00:22

Copilot AI review requested due to automatic review settings May 26, 2026 23:46

juhy0987 added the enhancement New feature or request label May 26, 2026

Copilot started reviewing on behalf of juhy0987 May 26, 2026 23:46 View session

gemini-code-assist Bot reviewed May 26, 2026

View reviewed changes

Comment thread src/graphlm/neuron/graph_hybrid.py Outdated

Comment thread tests/neuron/test_graph_hybrid.py

Copilot AI reviewed May 26, 2026

View reviewed changes

Comment thread tests/neuron/test_graph_hybrid.py Outdated

Comment thread src/graphlm/neuron/hybrid_transformer_demo.py

juhy0987 added 2 commits May 27, 2026 09:00

[CHORE]: Phase 15 figure 자산 추가 (Notion 임베드용)

c549022

juhy0987 self-assigned this May 27, 2026

juhy0987 merged commit 400e2d8 into main May 27, 2026
10 checks passed

coderabbitai Bot mentioned this pull request May 27, 2026

[FEAT#77] neuron Phase 16b — Net2Net function-preserving FFN grow (cross-shape expansion) #79

Merged

2 tasks

juhy0987 mentioned this pull request May 29, 2026

[FEATURE] neuron Phase 18 — large model + long training (Phase 17 plateau 미도달 한계 해소) #82

Open

3 tasks

Navigation Menu

Uh oh!

[FEAT#71] neuron Phase 15 — sparsity-driven edge prune (정적 → 동적 위상 진입) - #72

[FEAT#71] neuron Phase 15 — sparsity-driven edge prune (정적 → 동적 위상 진입)#72
juhy0987 merged 5 commits into
mainfrom
feature/#71/neuron-phase15-sparsity-prune

juhy0987 commented May 26, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented May 26, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

juhy0987 commented May 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

연관 이슈

구현 내용

신규 모듈

gradient resurrection 방지 메커니즘

테스트 (182 → 202, +10 신규 prune 테스트)

0-init 금지 + magnitude rule 일관 적용 (변경 없음)

CI / 머지 게이트 점검

변경 영향 범위

Required Status Checks

롤백 계획

Phase 15 의 paradigm 의미 — 정적 → 동적 위상 진입

다음 단계 (Phase 16 후보)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

❌ Failed checks (1 warning)

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

juhy0987 commented May 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 26, 2026 •

edited

Loading