[FEAT#65] neuron Phase 12 — hierarchical hybrid graph hidden layer (outer group + inner channel)

notebooks/02-function-level/11-phase12-hybrid-graph-foundations.ipynb

@@ -0,0 +1,387 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# 11-phase12-hybrid-graph-foundations\n",
+    "\n",
+    "**neuron Phase 12** — paradigm 의 ultimate 단계. Phase 9 group + Phase 10/11 channel 의\n",
+    "계층적 hybrid (사용자 vision: 히든 레이어 = graph 의 ultimate 구현).\n",
+    "\n",
+    "핵심 가설:\n",
+    "1. **function preservation** — hybrid_full_full (outer=full, inner=full) ≈ plain Linear?\n",
+    "2. **Phase 9/10/11 통합 표현** — hybrid 의 init 조합으로 이전 phase 결과 재현 가능?\n",
+    "   - outer=identity + inner=full ≈ Phase 9 group_identity (+0.14 열위)\n",
+    "   - outer=full + inner=around_one ≈ Phase 11 channel_around_one (≈ plain)\n",
+    "3. **계층적 routing 효과** — 두 adj 의 학습된 패턴 분리 시각화?\n",
+    "4. **paradigm 안정성** — magnitude rule 자동 적용으로 모든 init 조합 안전?\n",
+    "\n",
+    "설계: 4 × 2 sweep = 8 run, max_steps=1500.\n",
+    "\n",
+    "데이터: TinyShakespeare (char-LM)\n",
+    "시드: [42, 123]\n",
+    "작성일: 2026-05-26\n",
+    "연관: Issue [#65](https://github.com/EinSofINTEREST/GraphLM/issues/65) / Phase 11 baseline PR [#64](https://github.com/EinSofINTEREST/GraphLM/pull/64)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## 1. 환경"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "import sys\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "import graphlm\n",
+    "from graphlm.data.tinyshakespeare import (\n",
+    "    CharTokenizer,\n",
+    "    TinyShakespeareDataset,\n",
+    "    load_tinyshakespeare_text,\n",
+    ")\n",
+    "from graphlm.neuron.graph_hybrid_demo import train_hybrid_graph_mlp\n",
+    "from graphlm.utils import repo_root\n",
+    "\n",
+    "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "logging.basicConfig(\n",
+    "    level=logging.WARNING, format=\"%(asctime)s %(levelname)s %(message)s\", datefmt=\"%H:%M:%S\"\n",
+    ")\n",
+    "print(\"python  :\", sys.version.split()[0])\n",
+    "print(\"graphlm :\", graphlm.__version__)\n",
+    "print(\"torch   :\", torch.__version__)\n",
+    "print(\"device  :\", DEVICE)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## 2. 실험 설정"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import math\n",
+    "\n",
+    "ROOT = repo_root()\n",
+    "DATA_PATH = ROOT / \"data\" / \"tinyshakespeare.txt\"\n",
+    "OUT_DIR = ROOT / \"runs\" / \"notebook-neuron-phase12\"\n",
+    "OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "SEEDS = [42, 123]\n",
+    "ARCHS = [\n",
+    "    \"plain\",\n",
+    "    \"hybrid_full_full\",\n",
+    "    \"hybrid_identity_full\",\n",
+    "    \"hybrid_full_around_one\",\n",
+    "]\n",
+    "EMB_DIM = 64\n",
+    "HIDDEN_DIM = 256  # 256 / 16 = 16 groups\n",
+    "GROUP_SIZE = 16\n",
+    "N_GRAM = 4\n",
+    "BATCH_SIZE = 32\n",
+    "LR = 3e-4\n",
+    "MAX_STEPS = 1500\n",
+    "\n",
+    "# Phase 9/10/11 baseline (직접 비교)\n",
+    "PHASE9_GROUP_FULL = 2.1391\n",
+    "PHASE9_GROUP_IDENTITY = 2.2797  # 0-init vanishing 사례\n",
+    "PHASE10_CHANNEL_FULL = 2.1339\n",
+    "PHASE11_CHANNEL_AROUND_ONE = 2.1456\n",
+    "\n",
+    "print(f\"SEEDS      = {SEEDS}\")\n",
+    "print(f\"ARCHS      = {ARCHS}\")\n",
+    "print(\n",
+    "    f\"HIDDEN_DIM = {HIDDEN_DIM}, GROUP_SIZE = {GROUP_SIZE} → n_groups = {HIDDEN_DIM // GROUP_SIZE}\"\n",
+    ")\n",
+    "print(f\"MAX_STEPS  = {MAX_STEPS}\")\n",
+    "print(f\"전체 run   = {len(SEEDS) * len(ARCHS)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## 3. 데이터 로드"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = load_tinyshakespeare_text(DATA_PATH)\n",
+    "tokenizer = CharTokenizer(text)\n",
+    "dataset = TinyShakespeareDataset(text, tokenizer)\n",
+    "V = tokenizer.vocab_size\n",
+    "\n",
+    "V_PADDED = math.ceil(V / GROUP_SIZE) * GROUP_SIZE\n",
+    "print(f\"vocab_size : {V}, padded to {V_PADDED} (GROUP_SIZE {GROUP_SIZE} 배수)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## 4. sweep 학습"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "runs = {}\n",
+    "for seed in SEEDS:\n",
+    "    for arch in ARCHS:\n",
+    "        key = (seed, arch)\n",
+    "        print(f\"--- seed={seed}, arch={arch} ---\")\n",
+    "        runs[key] = train_hybrid_graph_mlp(\n",
+    "            dataset=dataset,\n",
+    "            vocab_size=V_PADDED,\n",
+    "            seed=seed,\n",
+    "            arch=arch,\n",
+    "            emb_dim=EMB_DIM,\n",
+    "            hidden_dim=HIDDEN_DIM,\n",
+    "            group_size=GROUP_SIZE,\n",
+    "            n_gram=N_GRAM,\n",
+    "            batch_size=BATCH_SIZE,\n",
+    "            lr=LR,\n",
+    "            max_steps=MAX_STEPS,\n",
+    "            device=DEVICE,\n",
+    "        )\n",
+    "        print(f\"  done: final_loss={runs[key]['final_loss']:.4f}\")\n",
+    "        if str(DEVICE).startswith(\"cuda\"):\n",
+    "            torch.cuda.empty_cache()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## 5. 결과 표 + Phase 9/10/11 baseline 비교"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import statistics\n",
+    "\n",
+    "print(f\"{'arch':>28}  {'seed':>5}  {'final_loss':>11}\")\n",
+    "print(\"-\" * 55)\n",
+    "for arch in ARCHS:\n",
+    "    for seed in SEEDS:\n",
+    "        r = runs[(seed, arch)]\n",
+    "        print(f\"{arch:>28}  {seed:>5}  {r['final_loss']:>11.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== arch 별 mean ===\")\n",
+    "agg = {}\n",
+    "for arch in ARCHS:\n",
+    "    fls = [runs[(s, arch)][\"final_loss\"] for s in SEEDS]\n",
+    "    agg[arch] = dict(mean=statistics.mean(fls), range=max(fls) - min(fls))\n",
+    "    print(f\"  {arch:>28}: mean={agg[arch]['mean']:.4f}, range={agg[arch]['range']:.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== Phase 9/10/11 baseline (직접 비교 — 통합 표현 검증) ===\")\n",
+    "print(f\"  Phase 9  group_full          : {PHASE9_GROUP_FULL:.4f}\")\n",
+    "print(f\"  Phase 9  group_identity      : {PHASE9_GROUP_IDENTITY:.4f}  (0-init vanishing 사례)\")\n",
+    "print(f\"  Phase 10 channel_full        : {PHASE10_CHANNEL_FULL:.4f}\")\n",
+    "print(f\"  Phase 11 channel_around_one  : {PHASE11_CHANNEL_AROUND_ONE:.4f}\")\n",
+    "print()\n",
+    "print(f\"  Phase 12 hybrid_full_full         : {agg['hybrid_full_full']['mean']:.4f}\")\n",
+    "print(\n",
+    "    f\"  Phase 12 hybrid_identity_full     : {agg['hybrid_identity_full']['mean']:.4f}  (expected ≈ Phase 9 group_identity)\"\n",
+    ")\n",
+    "print(\n",
+    "    f\"  Phase 12 hybrid_full_around_one   : {agg['hybrid_full_around_one']['mean']:.4f}  (expected ≈ Phase 11 channel_around_one)\"\n",
+    ")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== 자동 verdict ===\")\n",
+    "plain = agg[\"plain\"][\"mean\"]\n",
+    "full_full = agg[\"hybrid_full_full\"][\"mean\"]\n",
+    "identity_full = agg[\"hybrid_identity_full\"][\"mean\"]\n",
+    "full_around = agg[\"hybrid_full_around_one\"][\"mean\"]\n",
+    "range_max = max(agg[a][\"range\"] for a in ARCHS)\n",
+    "print(f\"  range_max = {range_max:.4f}\")\n",
+    "print(f\"  hybrid_full_full vs plain         : {full_full - plain:+.4f}\")\n",
+    "print(\n",
+    "    f\"  hybrid_identity_full vs plain     : {identity_full - plain:+.4f} (expected ~+0.14 like Phase 9)\"\n",
+    ")\n",
+    "print(f\"  hybrid_full_around_one vs plain   : {full_around - plain:+.4f} (expected ≈ 0)\")\n",
+    "if abs(full_full - plain) < range_max:\n",
+    "    print(\"  ✅ hybrid_full_full ≈ plain — function preservation 작동\")\n",
+    "if identity_full - plain > 0.05:\n",
+    "    print(\"  ✅ hybrid_identity_full 열위 — Phase 9 group_identity 패턴 재현\")\n",
+    "if abs(full_around - plain) < range_max:\n",
+    "    print(\"  ✅ hybrid_full_around_one ≈ plain — Phase 11 channel_around_one 패턴 재현\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11",
+   "metadata": {},
+   "source": [
+    "## 6. 학습된 adj_outer + adj_inner 시각화 (hybrid_full_full)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "r = runs[(SEEDS[0], \"hybrid_full_full\")]\n",
+    "adj_outer = r[\"final_adj\"][\"fc1\"][\"outer\"].numpy()\n",
+    "adj_inner = r[\"final_adj\"][\"fc1\"][\"inner\"].numpy()\n",
+    "print(f\"adj_outer shape: {adj_outer.shape}\")\n",
+    "print(f\"adj_inner shape: {adj_inner.shape}\")\n",
+    "\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
+    "\n",
+    "# (a) adj_outer heatmap\n",
+    "vmax_o = max(abs(adj_outer).max(), 1e-6)\n",
+    "im0 = axes[0].imshow(adj_outer, cmap=\"RdBu_r\", vmin=-vmax_o, vmax=vmax_o, aspect=\"auto\")\n",
+    "axes[0].set_xlabel(\"input group\")\n",
+    "axes[0].set_ylabel(\"output group\")\n",
+    "axes[0].set_title(\n",
+    "    f\"adj_outer (group-level)\\nmean={adj_outer.mean():.3f}, std={adj_outer.std():.3f}\"\n",
+    ")\n",
+    "fig.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.04)\n",
+    "\n",
+    "# (b) adj_inner mean abs by (G_out, G_in) — block-aggregated heatmap\n",
+    "inner_block_mean = np.abs(adj_inner).mean(axis=(2, 3))\n",
+    "vmax_i = max(inner_block_mean.max(), 1e-6)\n",
+    "im1 = axes[1].imshow(inner_block_mean, cmap=\"viridis\", vmin=0, vmax=vmax_i, aspect=\"auto\")\n",
+    "axes[1].set_xlabel(\"input group\")\n",
+    "axes[1].set_ylabel(\"output group\")\n",
+    "axes[1].set_title(f\"adj_inner |mean| per block\\noverall mean={inner_block_mean.mean():.3f}\")\n",
+    "fig.colorbar(im1, ax=axes[1], fraction=0.046, pad=0.04)\n",
+    "\n",
+    "fig.suptitle(f\"Phase 12 — hybrid_full_full 학습된 adj (seed={SEEDS[0]}, fc1)\", fontsize=11)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"hybrid_adj.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## 7. loss curve 4 arch 비교 (mean ± σ)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "window = 30\n",
+    "colors = {\n",
+    "    \"plain\": \"#1f77b4\",\n",
+    "    \"hybrid_full_full\": \"#2ca02c\",\n",
+    "    \"hybrid_identity_full\": \"#d62728\",\n",
+    "    \"hybrid_full_around_one\": \"#ff7f0e\",\n",
+    "}\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(13, 5))\n",
+    "for arch in ARCHS:\n",
+    "    seed_curves = []\n",
+    "    for seed in SEEDS:\n",
+    "        losses = runs[(seed, arch)][\"losses\"]\n",
+    "        smoothed = np.convolve(losses, np.ones(window) / window, mode=\"valid\")\n",
+    "        seed_curves.append(smoothed)\n",
+    "    arr = np.array(seed_curves)\n",
+    "    steps = np.arange(window - 1, window - 1 + arr.shape[1])\n",
+    "    mean = arr.mean(axis=0)\n",
+    "    std = arr.std(axis=0, ddof=1)\n",
+    "    ax.plot(steps, mean, color=colors[arch], lw=1.5, label=arch)\n",
+    "    ax.fill_between(steps, mean - std, mean + std, color=colors[arch], alpha=0.15)\n",
+    "\n",
+    "ax.set_xlabel(\"step\")\n",
+    "ax.set_ylabel(f\"loss (smoothed window={window})\")\n",
+    "ax.set_title(\"Phase 12 — 4 arch loss curve (mean ± σ across 2 seeds)\")\n",
+    "ax.legend(loc=\"upper right\", fontsize=9)\n",
+    "ax.grid(alpha=0.3)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"loss_curves.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "## 결과 요약 / Phase 13 권장\n",
+    "\n",
+    "확인 포인트:\n",
+    "- §5 hybrid_full_full ≈ plain (function preservation)\n",
+    "- §5 hybrid_identity_full vs Phase 9 group_identity (+0.14 패턴 재현)\n",
+    "- §5 hybrid_full_around_one vs Phase 11 channel_around_one (≈ plain 패턴 재현)\n",
+    "- §6 학습된 adj_outer + adj_inner 의 분리 표현\n",
+    "- §7 4 arch 의 발산/수렴 패턴\n",
+    "\n",
+    "**판정 시나리오**:\n",
+    "- **A. 4 arch 모두 expected 와 일치** ⭐ — hybrid 가 Phase 9/10/11 의 통합 표현 입증 → Phase 13 (Transformer 통합) 진입\n",
+    "- **B. 일부 불일치** — hybrid 의 dual adj 의 interaction effect 가 단순 합 이상의 dynamics\n",
+    "- **C. hybrid_full_full 도 plain 보다 명확 우위** — 계층적 routing 의 추가 표현력 발견 (surprise positive)\n",
+    "\n",
+    "**참고**:\n",
+    "- Phase 11 결과: https://www.notion.so/36ce8b70b7aa81b4b684f9c559788a46\n",
+    "- 아키텍처 구성 계획: https://www.notion.so/36ce8b70b7aa818cbf1fe71687b449b8"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "GraphLM (uv .venv)",
+   "language": "python",
+   "name": "graphlm-uv"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

pyproject.toml

@@ -100,6 +100,9 @@ markers = [
 source = ["src/graphlm"]
 omit = [
     "*/tests/*",
+    # demo modules — 노트북 orchestration 전용, unit-test 대상 아님 (Phase 8 ~ 12)
+    "*/neuron/*_demo.py",
+    "*/neuron/positional_analysis.py",
 ]
 
 [tool.coverage.report]