EinSofINTEREST · juhy0987 · May 26, 2026 · May 26, 2026 · May 26, 2026 · May 26, 2026
diff --git a/notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb b/notebooks/02-function-level/08-phase9-group-graph-foundations.ipynb
@@ -0,0 +1,333 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# 08-phase9-group-graph-foundations\n",
+    "\n",
+    "**neuron Phase 9** — Hidden layer 자체를 graph 로 (group-as-node, 사용자 vision 의 architecture\n",
+    "구성 계획 B 진입).\n",
+    "\n",
+    "핵심 가설:\n",
+    "1. **function preservation** — GroupGraphLinear(adj=full) 가 standard Linear 와 forward 동치?\n",
+    "2. **adjacency 학습** — adj 파라미터에 gradient 가 흘러 routing 이 학습됨?\n",
+    "3. **identity init (block-diagonal)** 의 inductive bias — 시작부터 group-wise 독립 학습 vs full 보다 빠르거나 느림?\n",
+    "4. **adjacency 시각화** — 학습 후 어떤 group 들이 서로 강하게 연결되는가?\n",
+    "\n",
+    "설계: 3-way sweep × 2 seed = 6 run, max_steps=1500.\n",
+    "- arch ∈ {plain, group_full, group_identity}\n",
+    "- seed ∈ {42, 123}\n",
+    "\n",
+    "데이터: TinyShakespeare (char-LM)\n",
+    "시드: [42, 123]\n",
+    "작성일: 2026-05-26\n",
+    "연관: Issue [#59](https://github.com/EinSofINTEREST/GraphLM/issues/59) / Phase 8 baseline PR [#56](https://github.com/EinSofINTEREST/GraphLM/pull/56) / [아키텍처 구성 계획 (Notion)](https://www.notion.so/36ce8b70b7aa818cbf1fe71687b449b8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## 1. 환경"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "import sys\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "import graphlm\n",
+    "from graphlm.data.tinyshakespeare import (\n",
+    "    CharTokenizer,\n",
+    "    TinyShakespeareDataset,\n",
+    "    load_tinyshakespeare_text,\n",
+    ")\n",
+    "from graphlm.neuron.graph_group_demo import train_group_graph_mlp\n",
+    "from graphlm.utils import repo_root\n",
+    "\n",
+    "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "logging.basicConfig(\n",
+    "    level=logging.WARNING, format=\"%(asctime)s %(levelname)s %(message)s\", datefmt=\"%H:%M:%S\"\n",
+    ")\n",
+    "print(\"python  :\", sys.version.split()[0])\n",
+    "print(\"graphlm :\", graphlm.__version__)\n",
+    "print(\"torch   :\", torch.__version__)\n",
+    "print(\"device  :\", DEVICE)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## 2. 실험 설정"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ROOT = repo_root()\n",
+    "DATA_PATH = ROOT / \"data\" / \"tinyshakespeare.txt\"\n",
+    "OUT_DIR = ROOT / \"runs\" / \"notebook-neuron-phase9\"\n",
+    "OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "SEEDS = [42, 123]\n",
+    "ARCHS = [\"plain\", \"group_full\", \"group_identity\"]\n",
+    "EMB_DIM = 64\n",
+    "HIDDEN_DIM = 256  # group_size 16 의 배수\n",
+    "GROUP_SIZE = 16  # → n_groups_in/out = 16 (256/16) — 시각화 적합\n",
+    "N_GRAM = 4\n",
+    "BATCH_SIZE = 32\n",
+    "LR = 3e-4\n",
+    "MAX_STEPS = 1500\n",
+    "\n",
+    "# vocab*N_GRAM*EMB_DIM = N_GRAM * EMB_DIM = 256 = HIDDEN_DIM = vocab_logits 입력\n",
+    "# vocab_size 는 65 라 GROUP_SIZE 의 배수가 아님 — group_*는 vocab 도 padding 또는 unequal split\n",
+    "# 단순화: vocab_size 도 GROUP_SIZE 배수가 되도록 pad — 노트북에서 wrapper 처리\n",
+    "\n",
+    "print(f\"SEEDS      = {SEEDS}\")\n",
+    "print(f\"ARCHS      = {ARCHS}\")\n",
+    "print(\n",
+    "    f\"HIDDEN_DIM = {HIDDEN_DIM}, GROUP_SIZE = {GROUP_SIZE} → n_groups = {HIDDEN_DIM // GROUP_SIZE}\"\n",
+    ")\n",
+    "print(f\"MAX_STEPS  = {MAX_STEPS}\")\n",
+    "print(f\"전체 run   = {len(SEEDS) * len(ARCHS)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## 3. 데이터 로드"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = load_tinyshakespeare_text(DATA_PATH)\n",
+    "tokenizer = CharTokenizer(text)\n",
+    "dataset = TinyShakespeareDataset(text, tokenizer)\n",
+    "V = tokenizer.vocab_size\n",
+    "\n",
+    "# vocab_size 가 GROUP_SIZE 의 배수가 아니면 group_*는 fc2 의 out_features 가 안 맞음.\n",
+    "# 노트북 단순화: vocab_size 를 GROUP_SIZE 배수로 padding (예: 65 → 80) 한 가상 vocab 사용\n",
+    "import math\n",
+    "\n",
+    "V_PADDED = math.ceil(V / GROUP_SIZE) * GROUP_SIZE\n",
+    "print(f\"vocab_size : {V}, padded to {V_PADDED} (GROUP_SIZE {GROUP_SIZE} 배수)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## 4. sweep 학습\n",
+    "\n",
+    "각 (seed, arch) 에 대해 1 run. plain 은 baseline, group_full 은 function-preserving 시작,\n",
+    "group_identity 는 block-diagonal sparse 시작."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "runs = {}\n",
+    "for seed in SEEDS:\n",
+    "    for arch in ARCHS:\n",
+    "        key = (seed, arch)\n",
+    "        print(f\"--- seed={seed}, arch={arch} ---\")\n",
+    "        runs[key] = train_group_graph_mlp(\n",
+    "            dataset=dataset,\n",
+    "            vocab_size=V_PADDED,\n",
+    "            seed=seed,\n",
+    "            arch=arch,\n",
+    "            emb_dim=EMB_DIM,\n",
+    "            hidden_dim=HIDDEN_DIM,\n",
+    "            group_size=GROUP_SIZE,\n",
+    "            n_gram=N_GRAM,\n",
+    "            batch_size=BATCH_SIZE,\n",
+    "            lr=LR,\n",
+    "            max_steps=MAX_STEPS,\n",
+    "            device=DEVICE,\n",
+    "        )\n",
+    "        print(f\"  done: final_loss={runs[key]['final_loss']:.4f}\")\n",
+    "        if str(DEVICE).startswith(\"cuda\"):\n",
+    "            torch.cuda.empty_cache()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## 5. 결과 표 — arch × seed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import statistics\n",
+    "\n",
+    "print(f\"{'arch':>16}  {'seed':>5}  {'final_loss':>11}\")\n",
+    "print(\"-\" * 40)\n",
+    "for arch in ARCHS:\n",
+    "    for seed in SEEDS:\n",
+    "        r = runs[(seed, arch)]\n",
+    "        print(f\"{arch:>16}  {seed:>5}  {r['final_loss']:>11.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== arch 별 mean ===\")\n",
+    "for arch in ARCHS:\n",
+    "    fls = [runs[(s, arch)][\"final_loss\"] for s in SEEDS]\n",
+    "    print(f\"  {arch:>16}: mean={statistics.mean(fls):.4f}, range={max(fls) - min(fls):.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11",
+   "metadata": {},
+   "source": [
+    "## 6. adjacency 학습 진화 — fc1 / fc2 의 학습된 adj heatmap"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "fig = plt.figure(figsize=(14, 9))\n",
+    "gs = fig.add_gridspec(2, 4, width_ratios=[1, 1, 1, 1])\n",
+    "\n",
+    "# row 0: group_full 의 fc1, fc2 (seed 42)\n",
+    "# row 1: group_identity 의 fc1, fc2 (seed 42)\n",
+    "for row_i, arch in enumerate([\"group_full\", \"group_identity\"]):\n",
+    "    r = runs[(SEEDS[0], arch)]\n",
+    "    if r[\"final_adj\"] is None:\n",
+    "        continue\n",
+    "    for col_i, layer_name in enumerate([\"fc1\", \"fc2\"]):\n",
+    "        ax = fig.add_subplot(gs[row_i, col_i * 2 : col_i * 2 + 2])\n",
+    "        adj = r[\"final_adj\"][layer_name].numpy()\n",
+    "        vmax = max(abs(adj).max(), 1e-6)\n",
+    "        im = ax.imshow(adj, cmap=\"RdBu_r\", vmin=-vmax, vmax=vmax, aspect=\"auto\")\n",
+    "        ax.set_title(f\"{arch} — {layer_name} adj (seed={SEEDS[0]})  shape={adj.shape}\")\n",
+    "        ax.set_xlabel(\"group_in\")\n",
+    "        ax.set_ylabel(\"group_out\")\n",
+    "        fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)\n",
+    "\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"adjacency_heatmaps.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## 7. loss curve 비교 (arch 별 mean ± σ across 2 seeds)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "window = 30\n",
+    "colors = {\"plain\": \"#1f77b4\", \"group_full\": \"#2ca02c\", \"group_identity\": \"#ff7f0e\"}\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(13, 5))\n",
+    "for arch in ARCHS:\n",
+    "    seed_curves = []\n",
+    "    for seed in SEEDS:\n",
+    "        losses = runs[(seed, arch)][\"losses\"]\n",
+    "        smoothed = np.convolve(losses, np.ones(window) / window, mode=\"valid\")\n",
+    "        seed_curves.append(smoothed)\n",
+    "    arr = np.array(seed_curves)\n",
+    "    steps = np.arange(window - 1, window - 1 + arr.shape[1])\n",
+    "    mean = arr.mean(axis=0)\n",
+    "    std = arr.std(axis=0, ddof=1)\n",
+    "    ax.plot(steps, mean, color=colors[arch], lw=1.5, label=arch)\n",
+    "    ax.fill_between(steps, mean - std, mean + std, color=colors[arch], alpha=0.15)\n",
+    "ax.set_xlabel(\"step\")\n",
+    "ax.set_ylabel(f\"loss (smoothed window={window})\")\n",
+    "ax.set_title(f\"Phase 9 — plain Linear vs GroupGraphLinear (mean ± σ over {len(SEEDS)} seeds)\")\n",
+    "ax.legend(loc=\"upper right\", fontsize=9)\n",
+    "ax.grid(alpha=0.3)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"loss_curves.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "## 결과 요약 / Phase 10 권장 방향\n",
+    "\n",
+    "확인 포인트:\n",
+    "- §5 final_loss — group_full 이 plain 과 비슷? (function preservation 가설 입증)\n",
+    "- §5 group_identity vs plain — block-diagonal sparse init 의 학습 성능 (inductive bias 효과)\n",
+    "- §6 adjacency heatmap — group_full 학습 후 어떤 group 간 연결이 강해졌나? group_identity 의 off-diagonal 학습 활성도?\n",
+    "- §7 loss curve — 세 arch 의 수렴 속도 비교\n",
+    "\n",
+    "**판정 시나리오**:\n",
+    "- **A. group_full ≈ plain** ⭐ — function preservation 입증. group routing 학습 가능성 확인\n",
+    "- **B. group_full < plain** — adjacency 학습이 plain Linear 보다 능력 강화 (drop-in 대체 후보)\n",
+    "- **C. group_identity ≈ group_full** — block-diagonal sparse init 도 충분 (Phase 10 의 sparsification 권장)\n",
+    "- **D. group_identity 명확 열위** — sparse 시작이 학습 능력 제한 — adjacency growth 메커니즘 필요\n",
+    "\n",
+    "**참고**:\n",
+    "- 아키텍처 구성 계획 (Notion): https://www.notion.so/36ce8b70b7aa818cbf1fe71687b449b8\n",
+    "- ML 용어집: https://www.notion.so/36ce8b70b7aa812298bbe1388e61b753"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "GraphLM (uv .venv)",
+   "language": "python",
+   "name": "graphlm-uv"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
@@ -11,10 +11,12 @@
     NeuronGrowingDecoder,
     SinusoidalAlpha,
 )
+from graphlm.neuron.graph_group import GroupGraphLinear
 from graphlm.neuron.growable import GrowableEmbedding, GrowableLayerNorm, GrowableLinear
 from graphlm.neuron.growth import add_attn_function_preserving, add_attn_smooth_start
 
 __all__ = [
+    "GroupGraphLinear",
     "GrowableEmbedding",
     "GrowableLayerNorm",
     "GrowableLinear",