[FEAT#59] neuron Phase 9 — group-as-node graph hidden layer foundations

[juhy0987]

연관 이슈

Closes #59

배경

Phase 8 (PR #56) 의 structural axis foundations 검증 완료 후, GraphLM 원래 vision (히든 레이어 자체를 graph 구조로) 에 본격 진입. 노션 아키텍처 구성 계획 의 B (group-as-node) 먼저 권장 따름.

구현 내용

1. src/graphlm/neuron/graph_group.py (신규)

• GroupGraphLinear(in_features, out_features, group_size, adj_init)
  • 채널을 group_size 단위로 묶어 graph node 로 다루는 linear
  • weight: 4D block tensor (n_groups_out, n_groups_in, group_size, group_size)
  • adj: 학습 가능 routing scalar (n_groups_out, n_groups_in)
  • forward: y[go] = Σ_gi adj[go, gi] · (x[gi] @ W[go, gi]) (einsum block matmul)
  • adj_init="full" → 표준 Linear 와 forward 동치 (function preservation)
  • adj_init="identity" → block-diagonal (가장 sparse 시작)
• freeze_adjacency / sparsify_adjacency — Phase 10+ sparsification 학습 stub

2. src/graphlm/neuron/graph_group_demo.py (신규)

• GroupGraphMLPLM — 3 architecture (plain Linear / group_full / group_identity) 비교용 MLP-LM
• train_group_graph_mlp — sweep 학습 unit (노트북 분리 규약 준수)

3. tests/neuron/test_graph_group.py — 13 신규 테스트

• shape / validation (group_size 배수 / invalid adj_init)
• function preservation 수학적 검증 — adj=full + 같은 W 로 init 한 standard Linear 와 forward 동일 (atol=1e-5)
• adj=identity → block-diagonal forward (다른 group 의 입력 변경이 영향 없음) 정량 입증
• weight + adj 양쪽 gradient 흐름
• freeze_adjacency / sparsify_adjacency 동작 + edge case (negative threshold)

4. notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb (신규)

• 3 × 2 sweep: arch ∈ {plain, group_full, group_identity} × seed ∈ {42, 123} = 6 run
• HIDDEN_DIM=256, GROUP_SIZE=16 (n_groups=16, heatmap 적합)
• §5 final_loss 비교 + arch 별 mean
• §6 학습된 adjacency heatmap (group_full + group_identity 의 fc1/fc2)
• §7 loss curve 비교 (mean ± σ across seeds)
• vocab_size 65 → 80 padding (group_size 배수)

검증할 가설

1. function preservation — group_full 이 plain 과 forward 동치 + final_loss 비슷?
2. adjacency 학습 — adj.grad 흐름 + heatmap 에서 학습된 패턴 emerge?
3. inductive bias — group_identity (block-diagonal) 가 plain 과 비교해 학습 능력?

CI / 머지 게이트 점검

• make fmt 통과
• make lint 통과 (nbqa --fix 1건 자동 해소)
• make test 통과 — 85 passed (기존 72 + 신규 13)

변경 영향 범위 + 위험도

• 신규 파일 3개 (graph_group.py, graph_group_demo.py, test_graph_group.py) + 노트북 1개
• 기존 코드 무영향 (parallel architecture line, isolated)
• 위험도: 매우 낮음

Phase 10+ 계획

• Phase 10: A (channel-as-node) foundations — fine-grained sparse linear
• Phase 11: 계층적 hybrid (그룹 graph 안에 채널 graph nest)
• Phase 12: Transformer 통합 (Q/K/V/O 모두 GroupGraphLinear, RMSNorm)
• Phase 13: 학습된 routing (DARTS / L0 / Gumbel) 로 sparsification

Summary by CodeRabbit

• New Features

  • Added group-based linear layer supporting alternative architectural patterns with adjustable connectivity.
  • Introduced GroupGraphMLPLM language model with three configurable architecture variants (plain, group_full, group_identity).
  • Added Phase 9 experimental notebook comparing architectures, including loss curves and connectivity pattern visualizations.

• Tests

  • Added comprehensive test suite for the group-based linear layer.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6f9eaa6e-77e4-4678-a5a9-b5152c19b5b3

📥 Commits

Reviewing files that changed from the base of the PR and between a9df057 and 60b1990.

📒 Files selected for processing (5)

• notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb
• src/graphlm/neuron/__init__.py
• src/graphlm/neuron/graph_group.py
• src/graphlm/neuron/graph_group_demo.py
• tests/neuron/test_graph_group.py

---

📝 Walkthrough

Walkthrough

This PR introduces Phase 9 group-as-node graph linear layers, implementing GroupGraphLinear with learnable adjacency-based routing between grouped channels. The implementation includes comprehensive tests validating function preservation, gradient flow, and sparsification. A demo module trains three architectures on TinyShakespeare, and a Jupyter notebook runs a full experimental sweep comparing convergence and learned adjacency patterns.

Changes

Phase 9 Group-as-Node Graph Linear Implementation

Layer / File(s)	Summary
GroupGraphLinear Core Module and Export src/graphlm/neuron/graph_group.py, src/graphlm/neuron/__init__.py	Implements GroupGraphLinear with 4D block weights (n_groups_out, n_groups_in, group_size, group_size) and 2D learnable adjacency (n_groups_out, n_groups_in). Supports adj_init="full" (all-ones routing) and "identity" (block-diagonal). Forward reshapes inputs to grouped channels, applies grouped block contributions via einsum, and aggregates via adjacency-weighted summation. Includes freeze_adjacency() and sparsify_adjacency(threshold) utility methods. Exported publicly via __init__.py.
GroupGraphLinear Test Suite tests/neuron/test_graph_group.py	Validates initialization shapes, feature divisibility by group_size, adjacency initialization modes, functional equivalence to nn.Linear with full adjacency, block-diagonal routing under identity mode, gradient flow to both weight and adjacency, freeze_adjacency() behavior, and sparsify_adjacency() threshold logic with error handling.
Phase 9 Demo Architecture and Training src/graphlm/neuron/graph_group_demo.py	Defines GroupGraphMLPLM combining embedding, selectable first linear layer (plain or grouped with full/identity routing), GELU+LayerNorm, and final layer. Provides _make_linear routing helper, make_ngram_iter for n-gram batch conversion, and train_group_graph_mlp training loop that collects per-step losses, computes averages, and snapshots learned adjacency matrices for grouped variants.
Phase 9 Experiment Notebook notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb	Runs comprehensive Phase 9 experiments: configures sweep over three architectures (plain, group_full, group_identity) and two random seeds, loads TinyShakespeare with GROUP_SIZE-aligned vocabulary padding, executes training sweep, reports final losses by architecture/seed with mean and range statistics, generates adjacency heatmaps for grouped variants, plots smoothed loss curves with mean ± std across seeds, and provides decision checklist for Phase 10 direction selection.

Sequence Diagram(s)

sequenceDiagram
    participant Notebook
    participant TrainFunc as train_group_graph_mlp
    participant Model as GroupGraphMLPLM
    participant Embedding
    participant FC1 as fc1 (GroupGraphLinear)
    participant Act as GELU+LayerNorm
    participant FC2 as fc2 (Linear/GroupGraphLinear)
    Notebook->>TrainFunc: Call with arch, hyperparams
    TrainFunc->>Model: Initialize model (arch-dependent)
    loop max_steps
        TrainFunc->>Model: Forward on batch
        Model->>Embedding: Token IDs
        Embedding->>FC1: Embeddings (flattened)
        FC1->>Act: Hidden (plain or grouped)
        Act->>FC2: Activated features
        FC2->>Model: Logits
        Model->>TrainFunc: Loss
        TrainFunc->>Model: Backward + optimizer step
    end
    alt arch != "plain"
        TrainFunc->>FC1: Snapshot adj
        TrainFunc->>FC2: Snapshot adj
    end
    TrainFunc->>Notebook: Return dict{losses, final_loss, final_adj}

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

A rabbit hops through grouped channels bright,
Where adjacency learns which paths are right,
Block-sparse routing, learnable and free—
Phase 9's foundations, let's review and see! 🐇✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/#59/neuron-phase9-group-graph-foundations

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces Phase 9 group-as-node graph hidden layer foundations, implementing the GroupGraphLinear layer, a demo MLP-LM, and associated tests. The reviewer feedback points out three valuable improvement opportunities: correcting a Kaiming initialization bug where nn.init.kaiming_uniform_ miscalculates fan_in on a 4D tensor, simplifying the identity adjacency matrix creation using torch.eye directly, and optimizing the sparsification logic using masked_fill_ instead of multiplying by a casted mask.

[Copilot]

Pull request overview

Phase 9(neuron)에서 hidden layer를 group-as-node 그래프 구조로 표현하기 위한 기반을 추가하는 PR입니다. GroupGraphLinear를 도입해 채널을 group_size 단위로 묶어 block weight + 학습 가능한 adjacency로 라우팅을 학습할 수 있게 하고, 이를 검증/실험하기 위한 demo와 테스트/노트북을 함께 제공합니다.

Changes:

• GroupGraphLinear (block weight + adjacency) 신규 구현 및 public export 추가
• 비교 실험용 GroupGraphMLPLM/학습 루프 헬퍼 신규 추가
• shape/검증/function-preservation/gradient/sparsify 등 테스트 13개 및 Phase 9 노트북 추가

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/graphlm/neuron/graph_group.py	Group-as-node 그래프 선형 레이어(블록 weight + adjacency) 핵심 구현 추가
src/graphlm/neuron/graph_group_demo.py	plain vs group_full vs group_identity 비교용 MLP-LM 및 학습 헬퍼 추가
tests/neuron/test_graph_group.py	초기화/shape/동치성/gradient/freeze/sparsify 검증 테스트 추가
src/graphlm/neuron/__init__.py	GroupGraphLinear를 neuron 패키지 public API로 export
notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb	Phase 9 sweep 실험 및 adjacency/loss 시각화 노트북 추가