[FEAT#69] neuron Phase 14 — attention 도 HybridGraphLinear 로 (full graph block)

notebooks/02-function-level/12-phase13-hybrid-transformer.ipynb

@@ -144,49 +144,7 @@
    "id": "10",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\n",
-    "colors = {\n",
-    "    \"plain\": \"tab:gray\",\n",
-    "    \"hybrid_full_full\": \"tab:blue\",\n",
-    "    \"hybrid_full_around_one\": \"tab:orange\",\n",
-    "    \"hybrid_around_one_around_one\": \"tab:green\",\n",
-    "}\n",
-    "window = 50  # rolling mean for smoothing\n",
-    "\n",
-    "for arch in ARCHS:\n",
-    "    losses_per_seed = [results[(arch, s)][\"losses\"] for s in SEEDS]\n",
-    "    # rolling mean\n",
-    "    smoothed = []\n",
-    "    for losses in losses_per_seed:\n",
-    "        smoothed.append(\n",
-    "            [\n",
-    "                sum(losses[max(0, i - window) : i + 1]) / min(i + 1, window)\n",
-    "                for i in range(len(losses))\n",
-    "            ]\n",
-    "        )\n",
-    "    # mean ± σ across seeds\n",
-    "    arr = torch.tensor(smoothed)\n",
-    "    mean = arr.mean(dim=0)\n",
-    "    std = arr.std(dim=0)\n",
-    "    steps = list(range(len(mean)))\n",
-    "    color = colors[arch]\n",
-    "    ax.plot(steps, mean, label=arch, color=color, linewidth=1.5)\n",
-    "    ax.fill_between(steps, mean - std, mean + std, color=color, alpha=0.15)\n",
-    "\n",
-    "ax.set_xlabel(\"step\")\n",
-    "ax.set_ylabel(f\"loss (rolling mean w={window})\")\n",
-    "ax.set_title(\"Phase 13 — Transformer FFN: 4 arch loss curves (mean ± σ over 2 seeds)\")\n",
-    "ax.legend(loc=\"upper right\")\n",
-    "ax.grid(alpha=0.3)\n",
-    "plt.tight_layout()\n",
-    "\n",
-    "out_dir = Path(\"../../runs/notebook-neuron-phase13\")\n",
-    "out_dir.mkdir(parents=True, exist_ok=True)\n",
-    "fig.savefig(out_dir / \"loss_curves.png\", dpi=150, bbox_inches=\"tight\")\n",
-    "plt.show()\n",
-    "print(f\"saved: {out_dir / 'loss_curves.png'}\")"
-   ]
+   "source": "fig, ax = plt.subplots(1, 1, figsize=(10, 5))\ncolors = {\n    \"plain\": \"tab:gray\",\n    \"hybrid_full_full\": \"tab:blue\",\n    \"hybrid_full_around_one\": \"tab:orange\",\n    \"hybrid_around_one_around_one\": \"tab:green\",\n}\nwindow = 50  # rolling mean for smoothing\n\nfor arch in ARCHS:\n    losses_per_seed = [results[(arch, s)][\"losses\"] for s in SEEDS]\n    # rolling mean — slice 시작점 +1 시프트로 window 와 divisor 일치 (CodeRabbit #3304780219)\n    smoothed = []\n    for losses in losses_per_seed:\n        smoothed.append(\n            [\n                sum(losses[max(0, i - window + 1) : i + 1]) / min(i + 1, window)\n                for i in range(len(losses))\n            ]\n        )\n    # mean ± σ across seeds\n    arr = torch.tensor(smoothed)\n    mean = arr.mean(dim=0)\n    std = arr.std(dim=0)\n    steps = list(range(len(mean)))\n    color = colors[arch]\n    ax.plot(steps, mean, label=arch, color=color, linewidth=1.5)\n    ax.fill_between(steps, mean - std, mean + std, color=color, alpha=0.15)\n\nax.set_xlabel(\"step\")\nax.set_ylabel(f\"loss (rolling mean w={window})\")\nax.set_title(\"Phase 13 — Transformer FFN: 4 arch loss curves (mean ± σ over 2 seeds)\")\nax.legend(loc=\"upper right\")\nax.grid(alpha=0.3)\nplt.tight_layout()\n\nout_dir = Path(\"../../runs/notebook-neuron-phase13\")\nout_dir.mkdir(parents=True, exist_ok=True)\nfig.savefig(out_dir / \"loss_curves.png\", dpi=150, bbox_inches=\"tight\")\nplt.show()\nprint(f\"saved: {out_dir / 'loss_curves.png'}\")"
   },
   {
    "cell_type": "markdown",

notebooks/02-function-level/13-phase14-graph-attention.ipynb

@@ -0,0 +1,336 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# 13-phase14-graph-attention\n",
+    "\n",
+    "**neuron Phase 14** — attention 의 qkv / out 까지 `HybridGraphLinear` 로 교체 (**block 전체가 graph**). Phase 13 은 FFN 만 graph 였고, Phase 14 에서 paradigm 의 *full graph block* 단계 진입.\n",
+    "\n",
+    "핵심 가설:\n",
+    "1. **function preservation (attention 포함)** — full graph block (adj=full/full) ≈ plain block?\n",
+    "2. **attention graph 가 학습에 도움?** — Phase 13 (FFN-only) vs Phase 14 (full graph) 동일 arch 비교에서 final_loss 개선?\n",
+    "3. **dual scale-corrected 우위 재현** — Phase 12/13 의 `hybrid_around_one_around_one` 우위가 full graph 에서도 유지?\n",
+    "4. **파라미터 증가 대비 효과** — full graph 는 attention adj 도 추가되어 파라미터 ↑. ROI?\n",
+    "\n",
+    "설계: 4 arch × 2 seed × {Phase 13 (FFN-only), Phase 14 (full graph)} = 16 run.\n",
+    "- arch: plain / hybrid_full_full / hybrid_full_around_one / hybrid_around_one_around_one\n",
+    "- plain 은 use_full_graph 무관 (둘 다 PlainTransformerBlock)\n",
+    "- → 실제 비교: 4 arch × 2 seed × 2 mode - 2 (plain 중복) = 14 unique run\n",
+    "\n",
+    "데이터: TinyShakespeare (char-LM, block_size=64)\n",
+    "시드: [42, 123]\n",
+    "작성일: 2026-05-26\n",
+    "연관: Issue [#69](https://github.com/EinSofINTEREST/GraphLM/issues/69) / Phase 13 baseline PR [#68](https://github.com/EinSofINTEREST/GraphLM/pull/68)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## 0. 환경 / 의존성"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import annotations\n",
+    "\n",
+    "import math\n",
+    "import statistics\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import matplotlib.pyplot as plt\n",
+    "import torch\n",
+    "\n",
+    "from graphlm.data.tinyshakespeare import (\n",
+    "    CharTokenizer,\n",
+    "    TinyShakespeareDataset,\n",
+    "    load_tinyshakespeare_text,\n",
+    ")\n",
+    "from graphlm.neuron.hybrid_transformer_demo import (\n",
+    "    HybridGraphTransformerLM,\n",
+    "    HybridTransformerTrainConfig,\n",
+    "    count_parameters,\n",
+    "    train_hybrid_transformer_lm,\n",
+    ")\n",
+    "from graphlm.utils import safe_perplexity\n",
+    "\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"device: {device}\")\n",
+    "print(f\"torch: {torch.__version__}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## 1. Config + 데이터"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = load_tinyshakespeare_text()\n",
+    "tokenizer = CharTokenizer(text)\n",
+    "dataset = TinyShakespeareDataset(text, tokenizer)\n",
+    "vocab_size = tokenizer.vocab_size\n",
+    "print(f\"vocab_size = {vocab_size}, dataset size = {len(dataset)}\")\n",
+    "\n",
+    "# Phase 13 과 동일 hyperparameter (공정 비교)\n",
+    "HIDDEN_DIM = 128\n",
+    "N_HEADS = 4\n",
+    "FFN_DIM = 256\n",
+    "N_LAYERS = 4\n",
+    "GROUP_SIZE = 16\n",
+    "BLOCK_SIZE = 64\n",
+    "BATCH_SIZE = 32\n",
+    "LR = 3e-4\n",
+    "MAX_STEPS = 1500\n",
+    "SEEDS = [42, 123]\n",
+    "ARCHS = [\n",
+    "    \"plain\",\n",
+    "    \"hybrid_full_full\",\n",
+    "    \"hybrid_full_around_one\",\n",
+    "    \"hybrid_around_one_around_one\",\n",
+    "]\n",
+    "MODES = [(\"phase13_ffn_only\", False), (\"phase14_full_graph\", True)]\n",
+    "\n",
+    "# arch x mode 별 파라미터 수 비교\n",
+    "print(\"\\n== Parameter count by arch × mode ==\")\n",
+    "for arch in ARCHS:\n",
+    "    for mode_name, use_full in MODES:\n",
+    "        if arch == \"plain\" and use_full:\n",
+    "            continue  # plain 은 use_full 무관 (PlainTransformerBlock 동일)\n",
+    "        m = HybridGraphTransformerLM(\n",
+    "            vocab_size=vocab_size,\n",
+    "            hidden_dim=HIDDEN_DIM,\n",
+    "            n_heads=N_HEADS,\n",
+    "            ffn_dim=FFN_DIM,\n",
+    "            n_layers=N_LAYERS,\n",
+    "            max_seq_len=BLOCK_SIZE,\n",
+    "            arch=arch,\n",
+    "            group_size=GROUP_SIZE,\n",
+    "            use_full_graph=use_full,\n",
+    "        )\n",
+    "        print(f\"  {arch:32s} {mode_name:25s} params = {count_parameters(m):,}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## 2. Sweep 실행 (4 arch × 2 seed × 2 mode = 14 unique run)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results = {}\n",
+    "for arch in ARCHS:\n",
+    "    for mode_name, use_full in MODES:\n",
+    "        if arch == \"plain\" and use_full:\n",
+    "            continue\n",
+    "        for seed in SEEDS:\n",
+    "            key = (arch, mode_name, seed)\n",
+    "            print(f\"\\n== arch={arch} mode={mode_name} seed={seed} ==\")\n",
+    "            cfg = HybridTransformerTrainConfig(\n",
+    "                dataset=dataset,\n",
+    "                vocab_size=vocab_size,\n",
+    "                hidden_dim=HIDDEN_DIM,\n",
+    "                n_heads=N_HEADS,\n",
+    "                ffn_dim=FFN_DIM,\n",
+    "                n_layers=N_LAYERS,\n",
+    "                group_size=GROUP_SIZE,\n",
+    "                arch=arch,\n",
+    "                use_full_graph=use_full,\n",
+    "                block_size=BLOCK_SIZE,\n",
+    "                batch_size=BATCH_SIZE,\n",
+    "                lr=LR,\n",
+    "                max_steps=MAX_STEPS,\n",
+    "                seed=seed,\n",
+    "                device=device,\n",
+    "            )\n",
+    "            out = train_hybrid_transformer_lm(cfg)\n",
+    "            results[key] = out\n",
+    "            print(\n",
+    "                f\"  final_loss = {out['final_loss']:.4f}  (perplexity = {safe_perplexity(out['final_loss']):.2f})\"\n",
+    "            )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## 3. 결과 표 + 자동 verdict"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"{'arch':32s} {'mode':25s} {'seed':>6s} {'final_loss':>12s} {'perplexity':>12s}\")\n",
+    "print(\"-\" * 95)\n",
+    "for (arch, mode_name, seed), out in results.items():\n",
+    "    fl = out[\"final_loss\"]\n",
+    "    print(f\"{arch:32s} {mode_name:25s} {seed:>6d} {fl:>12.4f} {safe_perplexity(fl):>12.2f}\")\n",
+    "\n",
+    "# arch × mode 별 평균\n",
+    "print(\"\\n== Arch × mode summary (mean ± σ across seeds) ==\")\n",
+    "summary = {}\n",
+    "for arch in ARCHS:\n",
+    "    for mode_name, use_full in MODES:\n",
+    "        if arch == \"plain\" and use_full:\n",
+    "            continue\n",
+    "        vals = [results[(arch, mode_name, s)][\"final_loss\"] for s in SEEDS]\n",
+    "        mean = statistics.mean(vals)\n",
+    "        std = statistics.stdev(vals) if len(vals) > 1 else 0.0\n",
+    "        summary[(arch, mode_name)] = (mean, std)\n",
+    "        print(\n",
+    "            f\"  {arch:32s} {mode_name:25s} {mean:.4f} ± {std:.4f}  (perplexity ≈ {safe_perplexity(mean):.2f})\"\n",
+    "        )\n",
+    "\n",
+    "# 자동 verdict\n",
+    "print(\"\\n== Verdict ==\")\n",
+    "plain_loss = summary[(\"plain\", \"phase13_ffn_only\")][0]\n",
+    "ff_p14 = summary[(\"hybrid_full_full\", \"phase14_full_graph\")][0]\n",
+    "aa_p13 = summary[(\"hybrid_around_one_around_one\", \"phase13_ffn_only\")][0]\n",
+    "aa_p14 = summary[(\"hybrid_around_one_around_one\", \"phase14_full_graph\")][0]\n",
+    "\n",
+    "# 1. function preservation 확장 — full graph 도 plain 근방?\n",
+    "diff_ff_p14 = abs(ff_p14 - plain_loss)\n",
+    "verdict_1 = \"PASS\" if diff_ff_p14 < 0.15 else \"FAIL\"\n",
+    "print(\n",
+    "    f\"1. full graph function preservation: |hybrid_full_full(p14) - plain| = {diff_ff_p14:.4f}  [{verdict_1}]\"\n",
+    ")\n",
+    "\n",
+    "# 2. attention graph 효과 — 동일 arch 에서 Phase 14 ≤ Phase 13 + 0.05?\n",
+    "diff_aa = aa_p14 - aa_p13\n",
+    "verdict_2 = \"PASS\" if diff_aa <= 0.05 else \"FAIL\"\n",
+    "print(f\"2. attention graph not hurting (aa): Phase14 - Phase13 = {diff_aa:+.4f}  [{verdict_2}]\")\n",
+    "\n",
+    "# 3. 모두 finite\n",
+    "all_finite = all(math.isfinite(out[\"final_loss\"]) for out in results.values())\n",
+    "verdict_3 = \"PASS\" if all_finite else \"FAIL\"\n",
+    "print(f\"3. all-finite stability (RMSNorm + graph attention): {all_finite}  [{verdict_3}]\")\n",
+    "\n",
+    "# 4. dual scale-corrected 우위 in full graph — aa_p14 가 ff_p14 보다 ≤?\n",
+    "diff_aa_vs_ff = aa_p14 - ff_p14\n",
+    "verdict_4 = \"PASS\" if diff_aa_vs_ff <= 0.05 else \"FAIL\"\n",
+    "print(\n",
+    "    f\"4. around_one×around_one ≤ full_full + 0.05 (p14): diff = {diff_aa_vs_ff:+.4f}  [{verdict_4}]\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## 4. Loss curve 시각화 — Phase 13 (실선) vs Phase 14 (점선)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": "fig, ax = plt.subplots(1, 1, figsize=(12, 6))\ncolors = {\n    \"plain\": \"tab:gray\",\n    \"hybrid_full_full\": \"tab:blue\",\n    \"hybrid_full_around_one\": \"tab:orange\",\n    \"hybrid_around_one_around_one\": \"tab:green\",\n}\nwindow = 50\n\nfor arch in ARCHS:\n    for mode_name, use_full in MODES:\n        if arch == \"plain\" and use_full:\n            continue\n        losses_per_seed = [results[(arch, mode_name, s)][\"losses\"] for s in SEEDS]\n        # rolling mean — slice 시작점 +1 시프트로 window 와 divisor 일치 (CodeRabbit #3304780219)\n        smoothed = []\n        for losses in losses_per_seed:\n            smoothed.append(\n                [\n                    sum(losses[max(0, i - window + 1) : i + 1]) / min(i + 1, window)\n                    for i in range(len(losses))\n                ]\n            )\n        arr = torch.tensor(smoothed)\n        mean = arr.mean(dim=0)\n        std = arr.std(dim=0)\n        steps = list(range(len(mean)))\n        color = colors[arch]\n        # Phase 13 = solid, Phase 14 = dashed\n        linestyle = \"-\" if mode_name == \"phase13_ffn_only\" else \"--\"\n        label = f\"{arch} ({mode_name.replace('_', ' ')})\"\n        ax.plot(steps, mean, label=label, color=color, linewidth=1.5, linestyle=linestyle)\n        ax.fill_between(steps, mean - std, mean + std, color=color, alpha=0.10)\n\nax.set_xlabel(\"step\")\nax.set_ylabel(f\"loss (rolling mean w={window})\")\nax.set_title(\n    \"Phase 14 — full graph block: 4 arch × {Phase 13 FFN-only, Phase 14 full graph} (mean ± σ over 2 seeds)\"\n)\nax.legend(loc=\"upper right\", fontsize=8)\nax.grid(alpha=0.3)\nplt.tight_layout()\n\nout_dir = Path(\"../../runs/notebook-neuron-phase14\")\nout_dir.mkdir(parents=True, exist_ok=True)\nfig.savefig(out_dir / \"loss_curves.png\", dpi=150, bbox_inches=\"tight\")\nplt.show()\nprint(f\"saved: {out_dir / 'loss_curves.png'}\")"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11",
+   "metadata": {},
+   "source": [
+    "## 5. attention adj 시각화 (Phase 14, hybrid_around_one_around_one 만)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Phase 14 의 attention qkv / out 의 학습된 adj_outer 모양 확인 (block 0, seed 42)\n",
+    "snap = results[(\"hybrid_around_one_around_one\", \"phase14_full_graph\", 42)][\"final_adj\"][0]\n",
+    "print(f\"snap keys (Phase 14 full graph): {sorted(snap.keys())}\")\n",
+    "\n",
+    "fig, axes = plt.subplots(2, 2, figsize=(10, 8))\n",
+    "for ax, layer_name in zip(axes.flat, [\"qkv\", \"out\", \"fc1\", \"fc2\"], strict=True):\n",
+    "    outer = snap[layer_name][\"outer\"].numpy()\n",
+    "    im = ax.imshow(outer, cmap=\"RdBu_r\", vmin=-2, vmax=2)\n",
+    "    ax.set_title(f\"{layer_name} — adj_outer (block 0)\")\n",
+    "    ax.set_xlabel(\"G_in\")\n",
+    "    ax.set_ylabel(\"G_out\")\n",
+    "    plt.colorbar(im, ax=ax, fraction=0.046)\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "fig.savefig(out_dir / \"attention_adj.png\", dpi=150, bbox_inches=\"tight\")\n",
+    "plt.show()\n",
+    "print(f\"saved: {out_dir / 'attention_adj.png'}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## 6. 결론 / 다음 단계\n",
+    "\n",
+    "(셀 출력 보고 사용자가 채울 영역)\n",
+    "\n",
+    "- full graph block 도 function preservation 성립?\n",
+    "- attention graph 가 학습에 유의미한 효과?\n",
+    "- qkv vs out 의 adj 학습 패턴 차이?\n",
+    "\n",
+    "**Phase 15 후보**:\n",
+    "- sparsity-driven prune — adj magnitude < threshold edge 영구 제거 → dead channel 발생 (DST 계열, training-time dynamic parameter count 의 진정한 첫 단계)\n",
+    "- Net2Net / LiGO 식 grow — 학습 중 channel / group 추가 (function preservation 유지)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "GraphLM (uv .venv)",
+   "language": "python",
+   "name": "graphlm-uv-venv"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}