[FEAT#61] neuron Phase 10 — channel-as-node graph hidden layer foundations (A-safe)

notebooks/02-function-level/09-phase10-channel-graph-foundations.ipynb

@@ -0,0 +1,386 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# 09-phase10-channel-graph-foundations\n",
+    "\n",
+    "**neuron Phase 10** — channel-as-node graph hidden layer foundations (사용자 vision 의 본질 axis).\n",
+    "\n",
+    "핵심 가설:\n",
+    "1. **function preservation** — channel_full (adj=full) 가 plain Linear 와 forward 동치?\n",
+    "2. **adjacency 학습** — adj 파라미터에 gradient 흐름 + 학습된 per-edge importance?\n",
+    "3. **0-init 금지 + sweet spot 적용** — channel_uniform_small (adj ∈ [0.05, 0.15]) 시작이 plain 과 비교?\n",
+    "4. **post-training adj 분포** — Phase 5 의 implicit pruning 패턴이 edge-level 에서 재현?\n",
+    "5. **Phase 9 group 과의 비교** — channel-level granularity 가 group-level (Phase 9: 2.1378~2.1391) 보다 우위?\n",
+    "\n",
+    "설계: 3-way sweep × 2 seed = 6 run, max_steps=1500.\n",
+    "- arch ∈ {plain, channel_full, channel_uniform_small}\n",
+    "- seed ∈ {42, 123}\n",
+    "\n",
+    "데이터: TinyShakespeare (char-LM)\n",
+    "시드: [42, 123]\n",
+    "작성일: 2026-05-26\n",
+    "연관: Issue [#61](https://github.com/EinSofINTEREST/GraphLM/issues/61) / Phase 9 baseline PR [#60](https://github.com/EinSofINTEREST/GraphLM/pull/60) / [Phase 9 결과 PR #60](https://github.com/EinSofINTEREST/GraphLM/pull/60) — 0-init 금지 규칙"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## 1. 환경"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "import sys\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "import graphlm\n",
+    "from graphlm.data.tinyshakespeare import (\n",
+    "    CharTokenizer,\n",
+    "    TinyShakespeareDataset,\n",
+    "    load_tinyshakespeare_text,\n",
+    ")\n",
+    "from graphlm.neuron.graph_channel_demo import train_channel_graph_mlp\n",
+    "from graphlm.utils import repo_root\n",
+    "\n",
+    "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "logging.basicConfig(\n",
+    "    level=logging.WARNING, format=\"%(asctime)s %(levelname)s %(message)s\", datefmt=\"%H:%M:%S\"\n",
+    ")\n",
+    "print(\"python  :\", sys.version.split()[0])\n",
+    "print(\"graphlm :\", graphlm.__version__)\n",
+    "print(\"torch   :\", torch.__version__)\n",
+    "print(\"device  :\", DEVICE)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## 2. 실험 설정"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ROOT = repo_root()\n",
+    "DATA_PATH = ROOT / \"data\" / \"tinyshakespeare.txt\"\n",
+    "OUT_DIR = ROOT / \"runs\" / \"notebook-neuron-phase10\"\n",
+    "OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "SEEDS = [42, 123]\n",
+    "ARCHS = [\"plain\", \"channel_full\", \"channel_uniform_small\"]\n",
+    "EMB_DIM = 64\n",
+    "HIDDEN_DIM = 256  # channel granularity — group 무관, 어떤 dim 도 가능\n",
+    "N_GRAM = 4\n",
+    "BATCH_SIZE = 32\n",
+    "LR = 3e-4\n",
+    "MAX_STEPS = 1500\n",
+    "\n",
+    "# Phase 9 baseline (PR #60 결과)\n",
+    "PHASE9_PLAIN_MEAN = 2.1378\n",
+    "PHASE9_GROUP_FULL_MEAN = 2.1391\n",
+    "PHASE9_GROUP_IDENTITY_MEAN = 2.2797  # 0-init vanishing 사례\n",
+    "\n",
+    "print(f\"SEEDS      = {SEEDS}\")\n",
+    "print(f\"ARCHS      = {ARCHS}\")\n",
+    "print(f\"HIDDEN_DIM = {HIDDEN_DIM} (channel granularity)\")\n",
+    "print(f\"MAX_STEPS  = {MAX_STEPS}\")\n",
+    "print(f\"전체 run   = {len(SEEDS) * len(ARCHS)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## 3. 데이터 로드"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = load_tinyshakespeare_text(DATA_PATH)\n",
+    "tokenizer = CharTokenizer(text)\n",
+    "dataset = TinyShakespeareDataset(text, tokenizer)\n",
+    "V = tokenizer.vocab_size\n",
+    "print(f\"vocab_size : {V}\")  # padding 불필요 — channel granularity 는 모든 dim 지원"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## 4. sweep 학습\n",
+    "\n",
+    "각 (seed, arch) 에 대해 1 run.\n",
+    "- plain: standard nn.Linear baseline\n",
+    "- channel_full: adj=full (function preservation, 'free' graph)\n",
+    "- channel_uniform_small: adj ∈ [0.05, 0.15] (Phase 2 sweet spot 패턴 적용, 0-init 회피)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "runs = {}\n",
+    "for seed in SEEDS:\n",
+    "    for arch in ARCHS:\n",
+    "        key = (seed, arch)\n",
+    "        print(f\"--- seed={seed}, arch={arch} ---\")\n",
+    "        runs[key] = train_channel_graph_mlp(\n",
+    "            dataset=dataset,\n",
+    "            vocab_size=V,\n",
+    "            seed=seed,\n",
+    "            arch=arch,\n",
+    "            emb_dim=EMB_DIM,\n",
+    "            hidden_dim=HIDDEN_DIM,\n",
+    "            n_gram=N_GRAM,\n",
+    "            batch_size=BATCH_SIZE,\n",
+    "            lr=LR,\n",
+    "            max_steps=MAX_STEPS,\n",
+    "            device=DEVICE,\n",
+    "        )\n",
+    "        print(f\"  done: final_loss={runs[key]['final_loss']:.4f}\")\n",
+    "        if str(DEVICE).startswith(\"cuda\"):\n",
+    "            torch.cuda.empty_cache()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## 5. 결과 표 — arch × seed + Phase 9 baseline 비교"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import statistics\n",
+    "\n",
+    "print(f\"{'arch':>26}  {'seed':>5}  {'final_loss':>11}\")\n",
+    "print(\"-\" * 50)\n",
+    "for arch in ARCHS:\n",
+    "    for seed in SEEDS:\n",
+    "        r = runs[(seed, arch)]\n",
+    "        print(f\"{arch:>26}  {seed:>5}  {r['final_loss']:>11.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== arch 별 mean ===\")\n",
+    "agg = {}\n",
+    "for arch in ARCHS:\n",
+    "    fls = [runs[(s, arch)][\"final_loss\"] for s in SEEDS]\n",
+    "    agg[arch] = dict(mean=statistics.mean(fls), range=max(fls) - min(fls))\n",
+    "    print(f\"  {arch:>26}: mean={agg[arch]['mean']:.4f}, range={agg[arch]['range']:.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== Phase 9 (group-level) baseline 비교 ===\")\n",
+    "print(f\"  Phase 9 plain               : {PHASE9_PLAIN_MEAN:.4f}\")\n",
+    "print(f\"  Phase 9 group_full          : {PHASE9_GROUP_FULL_MEAN:.4f}\")\n",
+    "print(f\"  Phase 9 group_identity      : {PHASE9_GROUP_IDENTITY_MEAN:.4f}  (0-init vanishing 사례)\")\n",
+    "print()\n",
+    "print(f\"  Phase 10 plain (재현)        : {agg['plain']['mean']:.4f}\")\n",
+    "print(f\"  Phase 10 channel_full       : {agg['channel_full']['mean']:.4f}\")\n",
+    "print(f\"  Phase 10 channel_uniform    : {agg['channel_uniform_small']['mean']:.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11",
+   "metadata": {},
+   "source": [
+    "## 6. 학습된 adj 분포 분석 — implicit pruning at edge-level?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "# channel_* 의 fc1 adj 분포 vs init\n",
+    "fig, axes = plt.subplots(2, 2, figsize=(13, 8))\n",
+    "\n",
+    "for col_i, arch in enumerate([\"channel_full\", \"channel_uniform_small\"]):\n",
+    "    r = runs[(SEEDS[0], arch)]\n",
+    "    if r[\"final_adj\"] is None:\n",
+    "        continue\n",
+    "    for row_i, layer in enumerate([\"fc1\", \"fc2\"]):\n",
+    "        ax = axes[row_i, col_i]\n",
+    "        adj = r[\"final_adj\"][layer].numpy().flatten()\n",
+    "        ax.hist(adj, bins=80, alpha=0.7, color=\"#1f77b4\")\n",
+    "        ax.set_xlabel(\"adj value\")\n",
+    "        ax.set_ylabel(\"count\")\n",
+    "        ax.set_title(f\"{arch} — {layer} adj distribution (n={len(adj)})\")\n",
+    "        ax.axvline(0, color=\"red\", linestyle=\"--\", lw=0.8, alpha=0.5)\n",
+    "        ax.axvline(adj.mean(), color=\"green\", linestyle=\":\", lw=1, label=f\"mean={adj.mean():.3f}\")\n",
+    "        ax.legend(fontsize=8)\n",
+    "        ax.grid(alpha=0.3)\n",
+    "\n",
+    "fig.suptitle(f\"Phase 10 — channel-level adj 분포 (seed={SEEDS[0]})\", fontsize=11)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"adj_distribution.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## 7. adj heatmap (fc1, channel × channel)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
+    "\n",
+    "for i, arch in enumerate([\"channel_full\", \"channel_uniform_small\"]):\n",
+    "    r = runs[(SEEDS[0], arch)]\n",
+    "    if r[\"final_adj\"] is None:\n",
+    "        continue\n",
+    "    adj = r[\"final_adj\"][\"fc1\"].numpy()\n",
+    "    vmax = max(abs(adj).max(), 1e-6)\n",
+    "    im = axes[i].imshow(adj, cmap=\"RdBu_r\", vmin=-vmax, vmax=vmax, aspect=\"auto\")\n",
+    "    axes[i].set_xlabel(\"input channel\")\n",
+    "    axes[i].set_ylabel(\"output channel\")\n",
+    "    axes[i].set_title(f\"{arch} — fc1 adj (shape={adj.shape})\")\n",
+    "    fig.colorbar(im, ax=axes[i], fraction=0.046, pad=0.04)\n",
+    "\n",
+    "fig.suptitle(f\"Phase 10 — channel adj heatmap (seed={SEEDS[0]}, fc1)\", fontsize=11)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"adj_heatmaps.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "## 8. loss curve 비교 (arch × mean ± σ across 2 seeds)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "16",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "window = 30\n",
+    "colors = {\n",
+    "    \"plain\": \"#1f77b4\",\n",
+    "    \"channel_full\": \"#2ca02c\",\n",
+    "    \"channel_uniform_small\": \"#ff7f0e\",\n",
+    "}\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(13, 5))\n",
+    "for arch in ARCHS:\n",
+    "    seed_curves = []\n",
+    "    for seed in SEEDS:\n",
+    "        losses = runs[(seed, arch)][\"losses\"]\n",
+    "        smoothed = np.convolve(losses, np.ones(window) / window, mode=\"valid\")\n",
+    "        seed_curves.append(smoothed)\n",
+    "    arr = np.array(seed_curves)\n",
+    "    steps = np.arange(window - 1, window - 1 + arr.shape[1])\n",
+    "    mean = arr.mean(axis=0)\n",
+    "    std = arr.std(axis=0, ddof=1)\n",
+    "    ax.plot(steps, mean, color=colors[arch], lw=1.5, label=arch)\n",
+    "    ax.fill_between(steps, mean - std, mean + std, color=colors[arch], alpha=0.15)\n",
+    "ax.axhline(\n",
+    "    PHASE9_GROUP_FULL_MEAN,\n",
+    "    color=\"gray\",\n",
+    "    linestyle=\":\",\n",
+    "    lw=1,\n",
+    "    alpha=0.7,\n",
+    "    label=f\"Phase 9 group_full ({PHASE9_GROUP_FULL_MEAN})\",\n",
+    ")\n",
+    "ax.set_xlabel(\"step\")\n",
+    "ax.set_ylabel(f\"loss (smoothed window={window})\")\n",
+    "ax.set_title(f\"Phase 10 — plain Linear vs ChannelGraphLinear (mean ± σ over {len(SEEDS)} seeds)\")\n",
+    "ax.legend(loc=\"upper right\", fontsize=9)\n",
+    "ax.grid(alpha=0.3)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"loss_curves.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "17",
+   "metadata": {},
+   "source": [
+    "## 결과 요약 / Phase 11 권장 방향\n",
+    "\n",
+    "확인 포인트:\n",
+    "- §5 channel_full vs plain — function preservation 입증 (~0)?\n",
+    "- §5 channel_uniform_small vs plain — 0-init 회피한 sweet spot 패턴이 잘 작동? (열위 ≤ 0.05 면 OK)\n",
+    "- §6 adj 분포 — 학습 후 spread? 일부 edge 가 자연 약화 (implicit pruning)?\n",
+    "- §7 heatmap — sparse/structured 패턴 emerge?\n",
+    "- §8 Phase 9 group_full (2.1391) 와의 비교 — channel granularity 가 group 보다 우위/동등?\n",
+    "\n",
+    "**판정 시나리오**:\n",
+    "- **A. channel_full ≈ plain + uniform_small 도 비슷** ⭐ — function preservation + 0-init 회피 둘 다 입증, Phase 11 (hybrid) 진입\n",
+    "- **B. channel_full ≈ plain, channel_uniform_small 열위** — sweet spot 의 small init 이 fine-grained edge 에는 부족, 추가 sweep 필요\n",
+    "- **C. channel_uniform_small 우위** — implicit pruning at edge-level 이 실제 효과 입증 (sparsification 학습 motivation 강화)\n",
+    "\n",
+    "**참고**:\n",
+    "- 아키텍처 구성 계획 (Notion): https://www.notion.so/36ce8b70b7aa818cbf1fe71687b449b8\n",
+    "- Phase 9 결과: https://www.notion.so/36ce8b70b7aa8100b0acf756686d2e9f"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "GraphLM (uv .venv)",
+   "language": "python",
+   "name": "graphlm-uv"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}