[FEAT#69] neuron Phase 14 — attention 도 HybridGraphLinear 로 (full graph block) - #70
Conversation
- src/graphlm/neuron/graph_attention.py 신규 - qkv (hidden_dim → 3·hidden_dim) + out (hidden_dim → hidden_dim) 모두 HybridGraphLinear - sdpa 는 standard 그대로 - identity outer 미지원 (qkv rectangular) - 0-init 거부 + magnitude rule 은 HybridGraphLinear 상속 - 12 unit tests — function preservation (standard CausalSelfAttention 와 atol=1e-5 동치) / arch dispatch / gradient flow / 0-init 거부
…_block dispatch - src/graphlm/neuron/hybrid_transformer.py 확장: - FullGraphTransformerBlock: attention + FFN 모두 HybridGraphLinear (Phase 13 의 HybridGraphTransformerBlock 는 FFN-only graph 그대로 유지) - make_full_block: 4 arch dispatch (plain / hybrid_full_full / hybrid_full_around_one / hybrid_around_one_around_one) - __init__.py exports 갱신 - 7 신규 unit tests — full block function preservation (standard 와 atol=1e-5) / gradient flow / make_full_block arch dispatch
- HybridTransformerTrainConfig: use_full_graph: bool 필드 추가 (default False = Phase 13)
- HybridGraphTransformerLM: use_full_graph=True 시 make_full_block 사용 → attention 도 graph
- _snapshot_adj: full graph 일 때 attention qkv/out 의 adj 도 포함 (Phase 13 FFN-only 호환 유지)
- 4 arch × {Phase 13, Phase 14} 동시 sweep 가능 (use_full_graph 토글)
- notebooks/02-function-level/13-phase14-graph-attention.ipynb 신규
- 4 arch × 2 seed × {Phase 13 FFN-only, Phase 14 full graph} = 14 unique run
- 자동 verdict 4가지: function preservation / attention graph not hurting / all-finite / dual scale-corrected 우위
- loss curve (Phase 13 실선 vs Phase 14 점선) + attention adj heatmap (qkv / out / fc1 / fc2)
- ruff format / unused import 정리 (graph_attention.py, neuron/__init__.py, test_graph_attention.py)
|
Warning Review limit reached
More reviews will be available in 49 minutes and 36 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR realizes Phase 14 of the GraphLM paradigm by extending hybrid graph routing from FFN-only (Phase 13) to full transformer blocks. It introduces ChangesPhase 14 Full Graph Attention
Sequence Diagram(s)sequenceDiagram
participant Client as Training Loop
participant Config as HybridTransformerTrainConfig
participant Model as HybridGraphTransformerLM
participant Block as FullGraphTransformerBlock
participant Attn as HybridGraphCausalSelfAttention
Client->>Config: use_full_graph=True
Config->>Model: construct with use_full_graph flag
Model->>Model: select make_full_block factory
Model->>Block: create block for each layer
Block->>Attn: initialize HybridGraphCausalSelfAttention
Client->>Model: forward pass (x)
Model->>Block: residual + attn + ffn
Block->>Attn: apply hybrid-graph attention
Attn->>Attn: qkv via HybridGraphLinear
Attn->>Attn: scaled_dot_product_attention (causal)
Attn->>Attn: out via HybridGraphLinear
Block->>Model: attention output
Model->>Client: final logits
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsStopped waiting for pipeline failures after 30000ms. One of your pipelines takes longer than our 30000ms fetch window to run, so review may not consider pipeline-failure results for inline comments if any failures occurred after the fetch window. Increase the timeout if you want to wait longer or run a Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
이 PR은 Phase 13에서 FFN에만 적용되던 HybridGraphLinear를 attention(qkv/out)까지 확장하여, Transformer block 내 모든 Linear가 graph 표현이 되는 Phase 14 “full graph block”을 도입합니다(기본 동작은 use_full_graph=False로 유지).
Changes:
HybridGraphCausalSelfAttention(qkv/out =HybridGraphLinear) 신규 추가FullGraphTransformerBlock및make_full_block추가로 Phase 14 full-graph block 생성 경로 제공- 데모/노트북/테스트 확장:
use_full_graph옵션 및 attention adj 스냅샷/시각화 추가, 관련 테스트 추가
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/graphlm/neuron/graph_attention.py |
HybridGraphCausalSelfAttention 신규 구현 (qkv/out graph화) |
src/graphlm/neuron/hybrid_transformer.py |
FullGraphTransformerBlock + make_full_block 추가 |
src/graphlm/neuron/hybrid_transformer_demo.py |
use_full_graph 옵션 도입 및 full graph 시 attention adj 스냅샷 확장 |
src/graphlm/neuron/__init__.py |
Phase 14 신규 심볼 export 추가 |
tests/neuron/test_graph_attention.py |
graph attention 단위 테스트 신규 추가 |
tests/neuron/test_hybrid_transformer.py |
full graph block 및 make_full_block 관련 테스트 추가 |
notebooks/02-function-level/13-phase14-graph-attention.ipynb |
Phase 13 vs 14 sweep/시각화 노트북 신규 추가 |
There was a problem hiding this comment.
Code Review
This pull request implements Phase 14 of the project, introducing full graph blocks where both the attention mechanism (qkv and out projections) and the FFN layers are replaced with HybridGraphLinear modules. It adds the HybridGraphCausalSelfAttention and FullGraphTransformerBlock classes, updates the transformer demo and training configuration to support the new full graph mode, includes a demonstration notebook, and adds comprehensive unit tests to verify shape, gradient flow, and function preservation. There are no review comments to address, so I have no feedback to provide.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
notebooks/02-function-level/13-phase14-graph-attention.ipynb (1)
76-176: ⚡ Quick winSplit cells to match required notebook section order
Please split the combined setup flow into explicit sections (
config,data,model,training,evaluation) so the notebook structure follows the repository rule verbatim.As per coding guidelines, "Notebook cells should be organized: imports / config / data / model / training / evaluation".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@notebooks/02-function-level/13-phase14-graph-attention.ipynb` around lines 76 - 176, The current combined cell mixes config, data loading, model instantiation and the training sweep; split it into distinct notebook cells titled/configured as: (1) Config — define hyperparameters and constants (HIDDEN_DIM, N_HEADS, FFN_DIM, N_LAYERS, GROUP_SIZE, BLOCK_SIZE, BATCH_SIZE, LR, MAX_STEPS, SEEDS, ARCHS, MODES), (2) Data — call load_tinyshakespeare_text(), create tokenizer = CharTokenizer(text) and dataset = TinyShakespeareDataset(text, tokenizer) and compute vocab_size, (3) Model / Params Check — instantiate HybridGraphTransformerLM for the parameter count loop and call count_parameters(m) (keeping the plain/use_full guard), (4) Training — build HybridTransformerTrainConfig and run the sweep loop that calls train_hybrid_transformer_lm(cfg) and stores results in results dict, and (5) Evaluation — print final_loss / safe_perplexity from results; ensure each cell only contains its respective symbols (the variables and function calls listed) and move print statements to the appropriate cells.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@notebooks/02-function-level/13-phase14-graph-attention.ipynb`:
- Around line 279-280: The rolling-average comprehension that builds the
smoothed losses uses sum(losses[max(0, i - window) : i + 1]) but divides by
min(i + 1, window), causing an off-by-one when the slice includes window+1
items; update the divisor to use min(i + 1, window + 1) (or otherwise ensure the
slice length and divisor match) so the average is computed over the actual
number of elements (refer to the list comprehension that iterates "for i in
range(len(losses))" and the slice expression "losses[max(0, i - window) : i +
1]").
---
Nitpick comments:
In `@notebooks/02-function-level/13-phase14-graph-attention.ipynb`:
- Around line 76-176: The current combined cell mixes config, data loading,
model instantiation and the training sweep; split it into distinct notebook
cells titled/configured as: (1) Config — define hyperparameters and constants
(HIDDEN_DIM, N_HEADS, FFN_DIM, N_LAYERS, GROUP_SIZE, BLOCK_SIZE, BATCH_SIZE, LR,
MAX_STEPS, SEEDS, ARCHS, MODES), (2) Data — call load_tinyshakespeare_text(),
create tokenizer = CharTokenizer(text) and dataset =
TinyShakespeareDataset(text, tokenizer) and compute vocab_size, (3) Model /
Params Check — instantiate HybridGraphTransformerLM for the parameter count loop
and call count_parameters(m) (keeping the plain/use_full guard), (4) Training —
build HybridTransformerTrainConfig and run the sweep loop that calls
train_hybrid_transformer_lm(cfg) and stores results in results dict, and (5)
Evaluation — print final_loss / safe_perplexity from results; ensure each cell
only contains its respective symbols (the variables and function calls listed)
and move print statements to the appropriate cells.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 73be3226-7307-4b26-ac34-27085f91f331
⛔ Files ignored due to path filters (2)
docs/figures/neuron/phase14/attention_adj.pngis excluded by!**/*.pngdocs/figures/neuron/phase14/loss_curves.pngis excluded by!**/*.png
📒 Files selected for processing (7)
notebooks/02-function-level/13-phase14-graph-attention.ipynbsrc/graphlm/neuron/__init__.pysrc/graphlm/neuron/graph_attention.pysrc/graphlm/neuron/hybrid_transformer.pysrc/graphlm/neuron/hybrid_transformer_demo.pytests/neuron/test_graph_attention.pytests/neuron/test_hybrid_transformer.py
- CodeRabbit #3304780219: slice 가 window+1 elements 인데 divisor 는 window 라 bias - 수정: slice 시작점을 `i - window + 1` 로 시프트 → window 정확히 일치 - Phase 13 / Phase 14 두 노트북에 동일 패턴 → 둘 다 일관 수정 - verdict 영향 없음 (Phase 13/14 결과는 이미 검증 완료) — figure 의 rolling mean 정확도만 개선
연관 이슈
구현 내용
Phase 13 (PR #68) 에서 FFN 만 graph 였던
HybridGraphTransformerBlock위에, attention 의 qkv / out 도HybridGraphLinear로 교체한FullGraphTransformerBlock추가. block 전체가 graph 가 되는 paradigm 의 다음 단계.신규 모듈
src/graphlm/neuron/graph_attention.py—HybridGraphCausalSelfAttentionsrc/graphlm/neuron/hybrid_transformer.py확장FullGraphTransformerBlock: attention + FFN 둘 다 HybridGraphLinearmake_full_block: Phase 13make_block과 동일 arch literal, hybrid_* 만 full graphsrc/graphlm/neuron/hybrid_transformer_demo.py확장HybridTransformerTrainConfig.use_full_graph: bool = False(default Phase 13)HybridGraphTransformerLM가use_full_graph=True시make_full_block사용_snapshot_adj가 full graph 시 attention qkv/out 의 adj 도 capturenotebooks/02-function-level/13-phase14-graph-attention.ipynb신규4 arch (Phase 13 와 동일)
plain— PlainTransformerBlock (RMSNorm + 표준 attention + 표준 FFN)hybrid_full_full— function preserving baseline (full graph 시 attention 도 full)hybrid_full_around_one— Phase 11 channel scale-corrected (full graph 시 attention 도 동일 init)hybrid_around_one_around_one— outer / inner 둘 다 scale-corrected0-init 금지 + magnitude rule 일관 적용
HybridGraphCausalSelfAttention의 qkv/out 도 underlyingHybridGraphLinear의 ValueError 상속테스트 (160 → 182, +22 신규)
test_graph_attention.py— 12 teststest_hybrid_transformer.py— 7 tests (full block / make_full_block)CI / 머지 게이트 점검
변경 영향 범위
src/graphlm/neuron/(1 신규 + 2 확장),notebooks/02-function-level/(1 신규),tests/neuron/(1 신규 + 1 확장)Low— Phase 13 의HybridGraphTransformerBlock/make_block그대로 유지 (default behavior 변경 없음).use_full_graph는 opt-in.Required Status Checks
Commit Lint(로컬 통과)PR Title LintLinked Issue CheckFormat CheckBuildTest(로컬 182 passed)Lint롤백 계획
use_full_graph=False= Phase 13). revert 시 영향 없음.다음 단계 (Phase 15 후보)
본 단계로 block 전체가 graph 확립. 다음은:
Summary by CodeRabbit
New Features
Tests