[FEAT#63] neuron Phase 11 — channel adj 의 scale-corrected init (uniform_around_one)

notebooks/02-function-level/10-phase11-scale-corrected-init.ipynb

@@ -0,0 +1,365 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# 10-phase11-scale-corrected-init\n",
+    "\n",
+    "**neuron Phase 11** — Phase 10 의 결정적 발견 (\"sweet spot magnitude 는 자유도의 의미적 위치에\n",
+    "의존\") 의 빠른 검증. ChannelGraphLinear 의 adj 에 **scale-corrected init** 적용.\n",
+    "\n",
+    "핵심 가설:\n",
+    "1. **uniform_around_one ≈ channel_full** — 1.0 근처 noise 도 magnitude 균형 유지 + adj 학습 활성\n",
+    "2. **uniform_around_one > uniform_small** — magnitude rule 직접 입증 (small=10% scale vs around_one=100% scale)\n",
+    "3. **plain ↔ channel_full ↔ uniform_around_one 모두 ≈** — graph 구조의 free 성 + scale-corrected init 동등성\n",
+    "\n",
+    "설계: 4-way × 2 seed = 8 run, max_steps=1500.\n",
+    "- arch ∈ {plain, channel_full, channel_uniform_small, channel_uniform_around_one}\n",
+    "- seed ∈ {42, 123}\n",
+    "\n",
+    "데이터: TinyShakespeare (char-LM)\n",
+    "시드: [42, 123]\n",
+    "작성일: 2026-05-26\n",
+    "연관: Issue [#63](https://github.com/EinSofINTEREST/GraphLM/issues/63) / Phase 10 baseline PR [#62](https://github.com/EinSofINTEREST/GraphLM/pull/62)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## 1. 환경"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "import sys\n",
+    "\n",
+    "import torch\n",
+    "\n",
+    "import graphlm\n",
+    "from graphlm.data.tinyshakespeare import (\n",
+    "    CharTokenizer,\n",
+    "    TinyShakespeareDataset,\n",
+    "    load_tinyshakespeare_text,\n",
+    ")\n",
+    "from graphlm.neuron.graph_channel_demo import train_channel_graph_mlp\n",
+    "from graphlm.utils import repo_root\n",
+    "\n",
+    "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "logging.basicConfig(\n",
+    "    level=logging.WARNING, format=\"%(asctime)s %(levelname)s %(message)s\", datefmt=\"%H:%M:%S\"\n",
+    ")\n",
+    "print(\"python  :\", sys.version.split()[0])\n",
+    "print(\"graphlm :\", graphlm.__version__)\n",
+    "print(\"torch   :\", torch.__version__)\n",
+    "print(\"device  :\", DEVICE)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3",
+   "metadata": {},
+   "source": [
+    "## 2. 실험 설정"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ROOT = repo_root()\n",
+    "DATA_PATH = ROOT / \"data\" / \"tinyshakespeare.txt\"\n",
+    "OUT_DIR = ROOT / \"runs\" / \"notebook-neuron-phase11\"\n",
+    "OUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "SEEDS = [42, 123]\n",
+    "ARCHS = [\"plain\", \"channel_full\", \"channel_uniform_small\", \"channel_uniform_around_one\"]\n",
+    "EMB_DIM = 64\n",
+    "HIDDEN_DIM = 256\n",
+    "N_GRAM = 4\n",
+    "BATCH_SIZE = 32\n",
+    "LR = 3e-4\n",
+    "MAX_STEPS = 1500\n",
+    "\n",
+    "# Phase 10 baseline (PR #62)\n",
+    "PHASE10_PLAIN_MEAN = 2.1487\n",
+    "PHASE10_CHANNEL_FULL_MEAN = 2.1339\n",
+    "PHASE10_CHANNEL_UNIFORM_SMALL_MEAN = 2.3268  # +0.18 열위 — magnitude scale 함정\n",
+    "\n",
+    "print(f\"SEEDS      = {SEEDS}\")\n",
+    "print(f\"ARCHS      = {ARCHS}\")\n",
+    "print(f\"HIDDEN_DIM = {HIDDEN_DIM}\")\n",
+    "print(f\"MAX_STEPS  = {MAX_STEPS}\")\n",
+    "print(f\"전체 run   = {len(SEEDS) * len(ARCHS)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "## 3. 데이터 로드"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = load_tinyshakespeare_text(DATA_PATH)\n",
+    "tokenizer = CharTokenizer(text)\n",
+    "dataset = TinyShakespeareDataset(text, tokenizer)\n",
+    "V = tokenizer.vocab_size\n",
+    "print(f\"vocab_size : {V}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "## 4. sweep 학습"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "runs = {}\n",
+    "for seed in SEEDS:\n",
+    "    for arch in ARCHS:\n",
+    "        key = (seed, arch)\n",
+    "        print(f\"--- seed={seed}, arch={arch} ---\")\n",
+    "        runs[key] = train_channel_graph_mlp(\n",
+    "            dataset=dataset,\n",
+    "            vocab_size=V,\n",
+    "            seed=seed,\n",
+    "            arch=arch,\n",
+    "            emb_dim=EMB_DIM,\n",
+    "            hidden_dim=HIDDEN_DIM,\n",
+    "            n_gram=N_GRAM,\n",
+    "            batch_size=BATCH_SIZE,\n",
+    "            lr=LR,\n",
+    "            max_steps=MAX_STEPS,\n",
+    "            device=DEVICE,\n",
+    "        )\n",
+    "        print(f\"  done: final_loss={runs[key]['final_loss']:.4f}\")\n",
+    "        if str(DEVICE).startswith(\"cuda\"):\n",
+    "            torch.cuda.empty_cache()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9",
+   "metadata": {},
+   "source": [
+    "## 5. 결과 표 + Phase 10 baseline 비교"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import statistics\n",
+    "\n",
+    "print(f\"{'arch':>32}  {'seed':>5}  {'final_loss':>11}\")\n",
+    "print(\"-\" * 60)\n",
+    "for arch in ARCHS:\n",
+    "    for seed in SEEDS:\n",
+    "        r = runs[(seed, arch)]\n",
+    "        print(f\"{arch:>32}  {seed:>5}  {r['final_loss']:>11.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== arch 별 mean ===\")\n",
+    "agg = {}\n",
+    "for arch in ARCHS:\n",
+    "    fls = [runs[(s, arch)][\"final_loss\"] for s in SEEDS]\n",
+    "    agg[arch] = dict(mean=statistics.mean(fls), range=max(fls) - min(fls))\n",
+    "    print(f\"  {arch:>32}: mean={agg[arch]['mean']:.4f}, range={agg[arch]['range']:.4f}\")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== Phase 10 baseline (PR #62) 비교 ===\")\n",
+    "print(f\"  Phase 10 plain                        : {PHASE10_PLAIN_MEAN:.4f}\")\n",
+    "print(f\"  Phase 10 channel_full                 : {PHASE10_CHANNEL_FULL_MEAN:.4f}\")\n",
+    "print(\n",
+    "    f\"  Phase 10 channel_uniform_small        : {PHASE10_CHANNEL_UNIFORM_SMALL_MEAN:.4f}  (+0.18 열위 — magnitude 함정)\"\n",
+    ")\n",
+    "print()\n",
+    "print(f\"  Phase 11 plain (재현)                  : {agg['plain']['mean']:.4f}\")\n",
+    "print(f\"  Phase 11 channel_full (재현)           : {agg['channel_full']['mean']:.4f}\")\n",
+    "print(f\"  Phase 11 channel_uniform_small (재현)  : {agg['channel_uniform_small']['mean']:.4f}\")\n",
+    "print(\n",
+    "    f\"  Phase 11 channel_uniform_around_one   : {agg['channel_uniform_around_one']['mean']:.4f}  ← 핵심 검증\"\n",
+    ")\n",
+    "\n",
+    "print()\n",
+    "print(\"=== 자동 verdict ===\")\n",
+    "plain = agg[\"plain\"][\"mean\"]\n",
+    "full = agg[\"channel_full\"][\"mean\"]\n",
+    "around_one = agg[\"channel_uniform_around_one\"][\"mean\"]\n",
+    "small = agg[\"channel_uniform_small\"][\"mean\"]\n",
+    "range_max = max(agg[a][\"range\"] for a in ARCHS)\n",
+    "print(f\"  range_max = {range_max:.4f}\")\n",
+    "print(f\"  around_one vs plain          : {around_one - plain:+.4f}\")\n",
+    "print(f\"  around_one vs channel_full   : {around_one - full:+.4f}\")\n",
+    "print(f\"  around_one vs uniform_small  : {around_one - small:+.4f}  (음수면 magnitude rule 입증)\")\n",
+    "if around_one - small < -0.05:\n",
+    "    print(\"  ✅ magnitude rule 입증 — around_one 이 uniform_small 보다 명확히 우위\")\n",
+    "if abs(around_one - full) < range_max:\n",
+    "    print(\"  ✅ around_one ≈ channel_full — scale balance 작동\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11",
+   "metadata": {},
+   "source": [
+    "## 6. 학습된 adj distribution 비교 — 4 arch"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "# channel_* 3 arch 의 fc1 adj 분포 비교\n",
+    "fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
+    "arch_names = [\"channel_full\", \"channel_uniform_small\", \"channel_uniform_around_one\"]\n",
+    "colors = [\"#1f77b4\", \"#ff7f0e\", \"#2ca02c\"]\n",
+    "\n",
+    "for i, arch in enumerate(arch_names):\n",
+    "    r = runs[(SEEDS[0], arch)]\n",
+    "    if r[\"final_adj\"] is None:\n",
+    "        continue\n",
+    "    adj = r[\"final_adj\"][\"fc1\"].numpy().flatten()\n",
+    "    axes[i].hist(adj, bins=80, alpha=0.7, color=colors[i])\n",
+    "    axes[i].set_xlabel(\"adj value\")\n",
+    "    axes[i].set_ylabel(\"count\")\n",
+    "    axes[i].set_title(f\"{arch}\\nmean={adj.mean():.3f}, std={adj.std():.3f}\")\n",
+    "    axes[i].axvline(0, color=\"red\", linestyle=\"--\", lw=0.8, alpha=0.5)\n",
+    "    axes[i].axvline(adj.mean(), color=\"green\", linestyle=\":\", lw=1)\n",
+    "    axes[i].grid(alpha=0.3)\n",
+    "\n",
+    "fig.suptitle(f\"Phase 11 — channel adj 분포 비교 (seed={SEEDS[0]}, fc1)\", fontsize=11)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"adj_dist_compare.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## 7. loss curve 비교 — 4 arch mean ± σ"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "window = 30\n",
+    "colors = {\n",
+    "    \"plain\": \"#1f77b4\",\n",
+    "    \"channel_full\": \"#2ca02c\",\n",
+    "    \"channel_uniform_small\": \"#ff7f0e\",\n",
+    "    \"channel_uniform_around_one\": \"#d62728\",\n",
+    "}\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(13, 5))\n",
+    "for arch in ARCHS:\n",
+    "    seed_curves = []\n",
+    "    for seed in SEEDS:\n",
+    "        losses = runs[(seed, arch)][\"losses\"]\n",
+    "        smoothed = np.convolve(losses, np.ones(window) / window, mode=\"valid\")\n",
+    "        seed_curves.append(smoothed)\n",
+    "    arr = np.array(seed_curves)\n",
+    "    steps = np.arange(window - 1, window - 1 + arr.shape[1])\n",
+    "    mean = arr.mean(axis=0)\n",
+    "    std = arr.std(axis=0, ddof=1)\n",
+    "    ax.plot(steps, mean, color=colors[arch], lw=1.5, label=arch)\n",
+    "    ax.fill_between(steps, mean - std, mean + std, color=colors[arch], alpha=0.15)\n",
+    "ax.axhline(\n",
+    "    PHASE10_CHANNEL_FULL_MEAN,\n",
+    "    color=\"gray\",\n",
+    "    linestyle=\":\",\n",
+    "    lw=1,\n",
+    "    alpha=0.7,\n",
+    "    label=f\"Phase 10 channel_full ({PHASE10_CHANNEL_FULL_MEAN})\",\n",
+    ")\n",
+    "ax.set_xlabel(\"step\")\n",
+    "ax.set_ylabel(f\"loss (smoothed window={window})\")\n",
+    "ax.set_title(\"Phase 11 — 4 arch loss curve (mean ± σ across 2 seeds)\")\n",
+    "ax.legend(loc=\"upper right\", fontsize=9)\n",
+    "ax.grid(alpha=0.3)\n",
+    "fig.tight_layout()\n",
+    "fig.savefig(OUT_DIR / \"loss_curves.png\", dpi=120)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "## 결과 요약 / Phase 12 권장\n",
+    "\n",
+    "확인 포인트:\n",
+    "- §5 around_one vs uniform_small (마이너스 시 magnitude rule 입증)\n",
+    "- §5 around_one vs channel_full (range 보다 작은 차이 시 scale balance 작동)\n",
+    "- §6 around_one 의 adj 분포 — 1.0 근처 spread 가 학습 후 어떻게 변화?\n",
+    "- §7 loss curve — around_one 이 uniform_small 보다 명확히 낮은 위치?\n",
+    "\n",
+    "**판정 시나리오**:\n",
+    "- **A. around_one ≈ channel_full + uniform_small 명확 열위 유지** ⭐ — magnitude rule 확정, Phase 12 (hybrid) 진입\n",
+    "- **B. around_one > channel_full** — noise 가 학습 가속 (implicit pruning 활성), surprise positive\n",
+    "- **C. around_one ≈ uniform_small** — magnitude 만으로 부족, 다른 요인 (예: weight 와의 동시 학습 dynamics)\n",
+    "\n",
+    "**참고**:\n",
+    "- Phase 10 결과: https://www.notion.so/36ce8b70b7aa81ff82a6edd3e2d03770\n",
+    "- 아키텍처 구성 계획: https://www.notion.so/36ce8b70b7aa818cbf1fe71687b449b8"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "GraphLM (uv .venv)",
+   "language": "python",
+   "name": "graphlm-uv"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}