[FEAT#65] neuron Phase 12 — hierarchical hybrid graph hidden layer (outer group + inner channel)

[juhy0987]

연관 이슈

Closes #65

배경 — paradigm 의 ultimate 단계

Phase 11 (PR #64) 의 magnitude rule 입증으로 paradigm 안정성 확보. 이제 사용자 vision (히든 레이어 = graph) 의 ultimate 구조 = 계층적 hybrid 진입.

hidden_dim H = G groups × k channels per group
weight W: (G_out, G_in, k, k)
adj_outer: (G_out, G_in)         # Phase 9 group-level routing
adj_inner: (G_out, G_in, k, k)   # Phase 10/11 channel-level fine-grained gate

forward:
  contrib[go, gi] = (adj_inner[go, gi] * W[go, gi]) @ x[gi]
  y[go] = Σ_gi adj_outer[go, gi] · contrib[go, gi]

effective edge weight = adj_outer · adj_inner · W (계층적 dual routing)

구현 내용

1. src/graphlm/neuron/graph_hybrid.py (신규)

• HybridGraphLinear(in_features, out_features, group_size, adj_outer_init, adj_inner_init)
• adj_outer_init: "full" / "identity" (square only) / "uniform_around_one"
• adj_inner_init: "full" / "uniform_around_one"
• ❌ 둘 다 "zero" 거부 (0-init 금지 규칙 자동 적용)
• adj_outer_sparsity / adj_inner_sparsity / freeze helpers

2. src/graphlm/neuron/graph_hybrid_demo.py (신규)

• HybridGraphMLPLM — Phase 9/10/11 통합 표현 데모 (4 arch)
• train_hybrid_graph_mlp — sweep 학습 unit

3. tests/neuron/test_graph_hybrid.py — 18 신규

• shape, adj_init 조합 valid 매트릭스
• function preservation 수학적 검증 (atol=1e-5)
• 양쪽 adj 의 0-init 거부 (parametrized 4건)
• identity outer 의 square requirement
• weight + adj_outer + adj_inner 3 param 모두 gradient 흐름
• sparsity / freeze metrics

4. notebooks/02-function-level/11-phase12-hybrid-graph-foundations.ipynb

• 4 × 2 sweep: arch × seed
• Phase 9/10/11 baseline 직접 비교 (통합 표현 검증 — init 조합으로 이전 결과 재현 가능성)
• 자동 verdict 코드 (3개 expected pattern 자동 판정)
• §6 adj_outer + adj_inner 분리 시각화 (group-level + block-aggregated channel)
• §7 4 arch loss curve mean ± σ

검증할 가설

1. function preservation — hybrid_full_full ≈ plain?
2. Phase 통합 표현 — identity_full ≈ Phase 9 group_identity, full_around_one ≈ Phase 11 channel_around_one?
3. 계층적 routing 의 dual learning — outer + inner 학습 패턴 분리?
4. paradigm 안정성 — 모든 init 조합 안전?

CI / 머지 게이트 점검

• make fmt 통과
• make lint 통과 (nbqa --fix 1건 자동)
• make test 통과 — 122 passed (기존 104 + 신규 18)

변경 영향 범위 + 위험도

• 신규 파일 3개 (graph_hybrid.py, graph_hybrid_demo.py, test_graph_hybrid.py) + 노트북 1개
• 기존 코드 무영향 (parallel architecture line)
• 위험도: 매우 낮음 (additive)

Phase 13+ 계획

• Phase 13: Transformer 통합 (Q/K/V/O 모두 GraphLinear / HybridGraphLinear)
• Phase 14: scale-up 정량 실험
• Phase 15+: real LM benchmark (cloud GPU)

Summary by CodeRabbit

• New Features

  • Added hybrid graph linear layer with learnable two-level routing gates for hierarchical connectivity control.

• Documentation

  • Added phase 12 analysis notebook with hybrid graph foundation experiments, baseline comparisons, loss curves visualization, and phase 13 recommendations.

• Tests

  • Added comprehensive test coverage for the hybrid graph layer, validating initialization modes, gradient flow, parameter freezing, and equivalence checks.

• Chores

  • Updated test coverage configuration to exclude demo-only modules.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces Phase 12 of the neuron architecture project: a hierarchical hybrid graph linear layer combining group-level (adj_outer) and channel-level (adj_inner) learnable routing gates with 4D block weights. It includes a complete test suite, demo language-modeling infrastructure, and an experimental notebook validating the design across four architectures and two random seeds.

Changes

Phase 12 Hybrid Graph Implementation & Validation

Layer / File(s)	Summary
HybridGraphLinear core implementation src/graphlm/neuron/graph_hybrid.py, src/graphlm/neuron/__init__.py	Implements HybridGraphLinear with two-level hierarchical routing: adj_outer (group-level, shape (G_out, G_in)) gates group-to-group flow, and adj_inner (channel-level, shape (G_out, G_in, k, k)) gates fine-grained channel pairs. Class validates divisibility, supports multiple init modes ("full", "identity", "uniform_around_one"), rejects zero-init, and provides forward via grouped einsum, sparsity metrics, and gradient-freeze utilities.
HybridGraphLinear test suite tests/neuron/test_graph_hybrid.py	Fourteen tests covering parameter shapes, forward output shape, init combinations and constraints (e.g., identity requires square groups), function equivalence to nn.Linear for full/full config, zero-init rejection with "vanishing" error message, gradient flow to all three parameters (weight, adj_outer, adj_inner), feature validation (divisibility and positivity), initial sparsity metrics, and freeze-adj helper behavior.
Demo model & training infrastructure src/graphlm/neuron/graph_hybrid_demo.py	Defines HybridGraphMLPLM for character-level language modeling with configurable fc1 (plain or hybrid) and conditional fc2 projection. _make_linear factory creates layers per architecture (plain, hybrid_full_full, hybrid_identity_full, hybrid_full_around_one) with proper divisibility validation. make_ngram_iter produces fixed-length n-gram batches. train_hybrid_graph_mlp runs AdamW optimization for max_steps, returns final loss and conditional adjacency snapshots.
Phase 12 experimental notebook & analysis notebooks/02-function-level/11-phase12-hybrid-graph-foundations.ipynb	Runs a 4 × 2 (architecture × seed) training sweep on TinyShakespeare, aggregates final losses per architecture with mean and range, compares against Phase 9/10/11 baselines, applies automatic verdict logic, visualizes learned adj_outer and adj_inner as heatmaps, plots smoothed mean ± std loss curves across all four architectures, and concludes with Phase 13 decision checklist and scenario-based criteria.
Test coverage configuration pyproject.toml	Adds */neuron/*_demo.py and */neuron/positional_analysis.py to coverage omit list.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• #61 — Phase 10 ChannelGraphLinear with learnable per-edge adjacency and "no zero-init, function-preserving full init" design that directly precedes the hierarchical hybrid approach in this PR.
• #59 — Phase 9 GroupGraphLinear with block weights and group-as-node routing that this PR extends into a two-level hierarchy with added channel-level gating.

Poem

🐰 Two gates dance in harmony,
Group and channel routing free,
Hybrid paths learn to align,
Zero-init ne'er shall shine,
Phase 12 blooms—Phase 13 nigh! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main feature being added: Phase 12 hierarchical hybrid graph linear layer with outer (group) and inner (channel) routing adjacencies.
Linked Issues check	✅ Passed	All primary coding objectives from issue #65 are met: HybridGraphLinear module with 4D weights and dual adjacency matrices, zero-init rejection, function preservation test, gradient flow validation, comprehensive test suite, and demo notebook with 4-architecture sweep.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to Phase 12 objectives. The only ancillary change is pyproject.toml coverage configuration to exclude demo modules, which is a necessary maintenance adjustment.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/#65/neuron-phase12-hybrid-graph

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces Phase 12 of the neuron paradigm, implementing a hierarchical hybrid graph hidden layer (HybridGraphLinear) that combines outer group-level routing with inner channel-level fine-grained gating. It includes a demo MLP-LM, a training helper, unit tests, and a Jupyter notebook demonstrating the foundations of this hybrid architecture. The review feedback highlights a critical memory optimization in the forward pass of HybridGraphLinear to prevent potential GPU OOM issues by combining adj_outer with the effective weights before performing the block matrix multiplication, thereby avoiding a large intermediate tensor.

[Copilot]

Pull request overview

Phase 12로 계층적 hybrid graph hidden layer(outer group + inner channel) 를 도입해, Phase 9/10/11의 그래프 기반 hidden layer 표현을 하나의 모듈로 통합 가능하게 만드는 PR입니다. 연구/실험 흐름은 노트북에서, 핵심 로직은 src/graphlm/ 모듈로 분리하는 기존 구조에 맞춰 구현/데모/테스트/노트북이 함께 추가되었습니다.

Changes:

• HybridGraphLinear 신규 구현: group-level adj_outer + channel-level adj_inner의 dual routing을 지원
• 데모 모델/학습 헬퍼(HybridGraphMLPLM, train_hybrid_graph_mlp) 추가로 노트북에서 sweep 실험 가능
• 기능 보존(function preservation), 0-init 거부, gradient 흐름, sparsity/freeze 헬퍼 등을 포함한 단위 테스트 18개 추가

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/graphlm/neuron/graph_hybrid.py	Phase 12 핵심 레이어 HybridGraphLinear 및 adj 초기화/메트릭/프리징 헬퍼 구현
src/graphlm/neuron/graph_hybrid_demo.py	노트북에서 import할 데모 MLP-LM 및 sweep 학습 유틸 추가
tests/neuron/test_graph_hybrid.py	shape/초기화 조합/동치성/예외/gradient/sparsity/freeze 테스트 추가
src/graphlm/neuron/__init__.py	HybridGraphLinear 공개 export 추가
notebooks/02-function-level/11-phase12-hybrid-graph-foundations.ipynb	4 arch × 2 seed sweep 및 baseline 비교/시각화 노트북 추가

[coderabbitai]

🧹 Nitpick comments (4)

src/graphlm/neuron/graph_hybrid_demo.py (2)

107-163: 🏗️ Heavy lift

Refactor train_hybrid_graph_mlp to reduce API and function size complexity.

This function currently exceeds the 50-line limit and takes more than 5 parameters. Please move run settings into a dataclass and split step/snapshot logic into helpers.

♻️ Suggested direction

+from dataclasses import dataclass
+
+@dataclass(frozen=True)
+class HybridTrainConfig:
+    vocab_size: int
+    seed: int
+    arch: Arch
+    emb_dim: int
+    hidden_dim: int
+    group_size: int
+    n_gram: int
+    batch_size: int
+    lr: float
+    max_steps: int
+    device: str = "cpu"
+
-def train_hybrid_graph_mlp(*, dataset: TinyShakespeareDataset, vocab_size: int, seed: int, arch: Arch, emb_dim: int, hidden_dim: int, group_size: int, n_gram: int, batch_size: int, lr: float, max_steps: int, device: str = "cpu") -> dict:
+def train_hybrid_graph_mlp(*, dataset: TinyShakespeareDataset, cfg: HybridTrainConfig) -> dict:
     ...

As per coding guidelines: "Functions must not exceed 50 lines" and "Use dataclass to group function arguments when there are 5 or more parameters".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/graphlm/neuron/graph_hybrid_demo.py` around lines 107 - 163, Refactor
train_hybrid_graph_mlp by extracting the many run parameters into a dataclass
(e.g., TrainingConfig or HybridRunConfig) and moving loop/snapshot logic into
small helpers: implement run_training_loop(config, model, data_iter, optimizer,
device, max_steps) to perform the step loop and return losses,
compute_final_loss(losses, n_last=100) to compute final_loss, and
extract_final_adj(model, arch) to build the final_adj dict (returning None for
"plain"); update train_hybrid_graph_mlp to accept the dataclass plus minimal
args (dataset, vocab_size, seed, arch) and call these helpers so the top-level
function stays under 50 lines and fewer than 5 direct parameters.

---

99-105: ⚡ Quick win

Add a docstring to the public helper make_ngram_iter.

Line 99 introduces a public function without a docstring. Please document its contract (input/output shapes and next-token target behavior).

As per coding guidelines: "All public functions must have type hints and docstrings".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/graphlm/neuron/graph_hybrid_demo.py` around lines 99 - 105, Add a
docstring to the public helper function make_ngram_iter describing its
parameters and return contract: document that dataset is a
TinyShakespeareDataset, batch_size and n_gram are ints, seed is the RNG seed,
and that it internally uses iter_random_batches with block_size=n_gram+1;
specify it yields an Iterator[tuple[Tensor, Tensor]] where the first Tensor has
shape (batch_size, n_gram) representing the input n-gram context and the second
Tensor has shape (batch_size,) representing the next-token target (the token at
position n_gram from the generated block). Also note any assumptions (e.g.,
tokens are integer-encoded) and that targets are aligned with inputs as
next-token prediction.

src/graphlm/neuron/graph_hybrid.py (2)

198-199: ⚡ Quick win

Add docstrings to public helper methods.

freeze_adj_outer, freeze_adj_inner, and extra_repr are public methods and should include docstrings for API consistency.

♻️ Proposed patch

@@
     def freeze_adj_outer(self) -> None:
+        """Freeze gradient updates for outer group-level adjacency."""
         self.adj_outer.requires_grad_(False)
@@
     def freeze_adj_inner(self) -> None:
+        """Freeze gradient updates for inner channel-level adjacency."""
         self.adj_inner.requires_grad_(False)
@@
     def extra_repr(self) -> str:
+        """Return compact module configuration for debugging/printing."""
         return (
             f"in_features={self.in_features}, out_features={self.out_features}, "
             f"group_size={self.group_size}, n_groups_in={self.n_groups_in}, "
             f"n_groups_out={self.n_groups_out}"
         )

As per coding guidelines, "All public functions must have type hints and docstrings."

Also applies to: 201-202, 204-209

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/graphlm/neuron/graph_hybrid.py` around lines 198 - 199, Public helper
methods freeze_adj_outer, freeze_adj_inner, and extra_repr lack docstrings; add
concise docstrings to each method in graph_hybrid.py describing purpose,
parameters (none) and return type (None or str) following the project's
docstring style (one-line summary plus optional short description), e.g., for
freeze_adj_outer/freeze_adj_inner state that they disable gradient updates on
self.adj_outer/self.adj_inner and return None, and for extra_repr describe the
string representation returned; keep type hints intact.

---

73-73: ⚡ Quick win

Extract repeated numeric literals into named constants.

0.95, 1.05, and 0.05 are repeated policy values; promote them to module-level constants for clarity and safer tuning.

♻️ Proposed patch

@@
 AdjOuterInit = Literal["full", "identity", "uniform_around_one"]
 AdjInnerInit = Literal["full", "uniform_around_one"]
+
+UNIFORM_AROUND_ONE_LOW = 0.95
+UNIFORM_AROUND_ONE_HIGH = 1.05
+DEFAULT_SPARSITY_THRESHOLD = 0.05
@@
-        return torch.empty(n_groups_out, n_groups_in).uniform_(0.95, 1.05)
+        return torch.empty(n_groups_out, n_groups_in).uniform_(
+            UNIFORM_AROUND_ONE_LOW, UNIFORM_AROUND_ONE_HIGH
+        )
@@
-        return torch.empty(shape).uniform_(0.95, 1.05)
+        return torch.empty(shape).uniform_(UNIFORM_AROUND_ONE_LOW, UNIFORM_AROUND_ONE_HIGH)
@@
-    def adj_outer_sparsity(self, threshold: float = 0.05) -> float:
+    def adj_outer_sparsity(self, threshold: float = DEFAULT_SPARSITY_THRESHOLD) -> float:
@@
-    def adj_inner_sparsity(self, threshold: float = 0.05) -> float:
+    def adj_inner_sparsity(self, threshold: float = DEFAULT_SPARSITY_THRESHOLD) -> float:

As per coding guidelines, "Replace magic numbers with named constants" and "Use UPPER_SNAKE_CASE for constants."

Also applies to: 93-93, 184-184, 191-191

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/graphlm/neuron/graph_hybrid.py` at line 73, Extract the magic numbers
into module-level UPPER_SNAKE_CASE constants (e.g., GROUP_UNIFORM_LOW = 0.95,
GROUP_UNIFORM_HIGH = 1.05, GROUP_MARGIN = 0.05) and replace all direct uses of
0.95, 1.05, and 0.05 with those constants; specifically update the expression
torch.empty(n_groups_out, n_groups_in).uniform_(0.95, 1.05) and the other
occurrences referenced in the review so they read .uniform_(GROUP_UNIFORM_LOW,
GROUP_UNIFORM_HIGH) or use GROUP_MARGIN where appropriate, keeping the constants
near the top of the module for clear tuning and reuse.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/graphlm/neuron/graph_hybrid_demo.py`:
- Around line 107-163: Refactor train_hybrid_graph_mlp by extracting the many
run parameters into a dataclass (e.g., TrainingConfig or HybridRunConfig) and
moving loop/snapshot logic into small helpers: implement
run_training_loop(config, model, data_iter, optimizer, device, max_steps) to
perform the step loop and return losses, compute_final_loss(losses, n_last=100)
to compute final_loss, and extract_final_adj(model, arch) to build the final_adj
dict (returning None for "plain"); update train_hybrid_graph_mlp to accept the
dataclass plus minimal args (dataset, vocab_size, seed, arch) and call these
helpers so the top-level function stays under 50 lines and fewer than 5 direct
parameters.
- Around line 99-105: Add a docstring to the public helper function
make_ngram_iter describing its parameters and return contract: document that
dataset is a TinyShakespeareDataset, batch_size and n_gram are ints, seed is the
RNG seed, and that it internally uses iter_random_batches with
block_size=n_gram+1; specify it yields an Iterator[tuple[Tensor, Tensor]] where
the first Tensor has shape (batch_size, n_gram) representing the input n-gram
context and the second Tensor has shape (batch_size,) representing the
next-token target (the token at position n_gram from the generated block). Also
note any assumptions (e.g., tokens are integer-encoded) and that targets are
aligned with inputs as next-token prediction.

In `@src/graphlm/neuron/graph_hybrid.py`:
- Around line 198-199: Public helper methods freeze_adj_outer, freeze_adj_inner,
and extra_repr lack docstrings; add concise docstrings to each method in
graph_hybrid.py describing purpose, parameters (none) and return type (None or
str) following the project's docstring style (one-line summary plus optional
short description), e.g., for freeze_adj_outer/freeze_adj_inner state that they
disable gradient updates on self.adj_outer/self.adj_inner and return None, and
for extra_repr describe the string representation returned; keep type hints
intact.
- Line 73: Extract the magic numbers into module-level UPPER_SNAKE_CASE
constants (e.g., GROUP_UNIFORM_LOW = 0.95, GROUP_UNIFORM_HIGH = 1.05,
GROUP_MARGIN = 0.05) and replace all direct uses of 0.95, 1.05, and 0.05 with
those constants; specifically update the expression torch.empty(n_groups_out,
n_groups_in).uniform_(0.95, 1.05) and the other occurrences referenced in the
review so they read .uniform_(GROUP_UNIFORM_LOW, GROUP_UNIFORM_HIGH) or use
GROUP_MARGIN where appropriate, keeping the constants near the top of the module
for clear tuning and reuse.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e455d075-9d24-4242-84e9-9f8b1a3a2767

📥 Commits

Reviewing files that changed from the base of the PR and between baaeaa4 and 3f3d616.

⛔ Files ignored due to path filters (2)

• docs/figures/neuron/phase12/hybrid_adj.png is excluded by !**/*.png
• docs/figures/neuron/phase12/loss_curves.png is excluded by !**/*.png

📒 Files selected for processing (6)

• notebooks/02-function-level/11-phase12-hybrid-graph-foundations.ipynb
• pyproject.toml
• src/graphlm/neuron/__init__.py
• src/graphlm/neuron/graph_hybrid.py
• src/graphlm/neuron/graph_hybrid_demo.py
• tests/neuron/test_graph_hybrid.py