diff --git a/docs/figures/neuron-phase18/loss_curves.png b/docs/figures/neuron-phase18/loss_curves.png new file mode 100644 index 0000000..fe8eb01 Binary files /dev/null and b/docs/figures/neuron-phase18/loss_curves.png differ diff --git a/docs/figures/neuron-phase18/phase16_17_18_progression.png b/docs/figures/neuron-phase18/phase16_17_18_progression.png new file mode 100644 index 0000000..abf196a Binary files /dev/null and b/docs/figures/neuron-phase18/phase16_17_18_progression.png differ diff --git a/notebooks/02-function-level/18-phase18-large-long.ipynb b/notebooks/02-function-level/18-phase18-large-long.ipynb new file mode 100644 index 0000000..470256e --- /dev/null +++ b/notebooks/02-function-level/18-phase18-large-long.ipynb @@ -0,0 +1,514 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": "# 18-phase18-large-long\n\n**neuron Phase 18** — Phase 17 의 핵심 한계 (**plateau 미도달**) 해소 + **larger model + longer training** 동시 적용.\n\n**배경**: Phase 17 sweep (~6M params, 5000 step) 의 핵심 발견:\n- scale-up 의 절대적 효과 명확 (loss 0.5-0.6 ↓, ppl 6.7 → 3.8)\n- dynamic method advantage 부분 진전 (grown 격차 +0.043 → +0.026 축소)\n- **plateau 미도달** — 모든 mode 가 drift -0.006 (still learning)\n\n→ **5000 step 도 부족**. Phase 18 은 **larger model (2x) + longer training (3x)** 동시 적용으로 *진정한 plateau* 도달 시도.\n\n**핵심 가설**:\n1. **plateau 도달** — 충분한 학습 시간으로 진정한 final loss 측정 (|drift| < 0.001 — 양/음 방향 모두 작은 변화만 plateau, 감소든 발산이든 절댓값 기준)\n2. **DST advantage 발현** — 더 큰 model + 긴 학습으로 swap cost amortize 가능?\n3. **grown ≈ dense** — 더 긴 post-grow 학습 (12500 step) 으로 0-init activation 충분?\n4. **Phase 17 → 18 trend** — dynamic advantage 가설의 추가 진전?\n\n설계: 5 mode × 2 seed = 10 run at **large model + long training**.\n\n| 항목 | Phase 17 | Phase 18 | 배수 |\n|---|---|---|---|\n| hidden_dim | 256 | **512** | 2x |\n| n_heads | 8 | **16** (head_dim=32 유지) | 2x |\n| n_layers | 6 | **12** | 2x |\n| ffn_dim (large) | 512 | **1024** | 2x |\n| max_steps | 5000 | **15000** | 3x |\n| approx params | ~6.4M | **~30-40M** | ~5-6x |\n\n5 mode (Phase 17 와 동일 구성, hp 만 scale 맞춤):\n- `dense_large`: 큰 모델 baseline (target reference)\n- `static_prune_50`: dense 학습 후 step 7500 에서 50% prune\n- `DST_RigL_p16hp`: Phase 16a 동일 hp (period=50, swap=0.1) — 순수 scale 효과 분리 control\n- `DST_RigL_p17hp`: Phase 17 보수적 hp (period=200, swap=0.05) — scale + hp 효과\n- `grown`: ffn=512 시작 → step 7500 에서 1024 로 grow\n\narch 고정: `hybrid_around_one_around_one` + `use_full_graph=True`\n데이터: TinyShakespeare (char-LM, block_size=64 유지 — vocab/data 동일, scale 만 model + train)\n시드: [42, 123]\nCompute 추정: 단일 GPU 4-6 시간\n작성일: 2026-05-29\n연관: Issue [#82](https://github.com/EinSofINTEREST/GraphLM/issues/82) / Phase 17 PR [#81](https://github.com/EinSofINTEREST/GraphLM/pull/81)" + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## 0. 환경 / 의존성" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "from __future__ import annotations\n", + "\n", + "import math\n", + "import statistics\n", + "from pathlib import Path\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import torch\n", + "\n", + "from graphlm.data.tinyshakespeare import (\n", + " CharTokenizer,\n", + " TinyShakespeareDataset,\n", + " load_tinyshakespeare_text,\n", + ")\n", + "from graphlm.neuron.hybrid_transformer_demo import (\n", + " HybridTransformerTrainConfig,\n", + " train_hybrid_transformer_lm,\n", + ")\n", + "from graphlm.utils import safe_perplexity\n", + "\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(f\"device: {device}\")\n", + "print(f\"torch: {torch.__version__}\")\n", + "if device == \"cpu\":\n", + " print(\"⚠️ Phase 18 은 ~30-40M params + 15K step — GPU 강력 권장\")" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## 1. Config — large model + long training" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "text = load_tinyshakespeare_text()\n", + "tokenizer = CharTokenizer(text)\n", + "dataset = TinyShakespeareDataset(text, tokenizer)\n", + "vocab_size = tokenizer.vocab_size\n", + "print(f\"vocab_size = {vocab_size}, dataset size = {len(dataset)}\")\n", + "\n", + "# Phase 18: large model (2x Phase 17) + long training (3x Phase 17)\n", + "HIDDEN_DIM = 512\n", + "N_HEADS = 16 # head_dim = 32 (512/16)\n", + "N_LAYERS = 12\n", + "GROUP_SIZE = 16\n", + "BLOCK_SIZE = 64\n", + "BATCH_SIZE = 32\n", + "LR = 3e-4\n", + "MAX_STEPS = 15000\n", + "DYNAMIC_AT_STEP = MAX_STEPS // 2 # 7500\n", + "# Phase 16a hyperparameter (control — scale 효과 only 분리)\n", + "DST_PERIOD_P16 = 50\n", + "DST_SWAP_FRACTION_P16 = 0.1\n", + "# Phase 17 보수적 hyperparameter\n", + "DST_PERIOD_P17 = 200\n", + "DST_SWAP_FRACTION_P17 = 0.05\n", + "# 마지막 1500 step stabilize (Phase 17 의 500 의 3x)\n", + "DST_END_STEP = max(1, MAX_STEPS - 1500)\n", + "SEEDS = [42, 123]\n", + "ARCH = \"hybrid_around_one_around_one\"\n", + "FFN_SMALL = 512\n", + "FFN_LARGE = 1024\n", + "\n", + "MODES = [\n", + " {\n", + " \"name\": \"dense_large\",\n", + " \"ffn_dim\": FFN_LARGE,\n", + " \"prune_at_step\": None,\n", + " \"prune_fraction\": 0.0,\n", + " \"regrow_method\": None,\n", + " \"dst_period\": None,\n", + " \"dst_swap_fraction\": 0.0,\n", + " \"grow_at_step\": None,\n", + " \"grow_ffn_target\": None,\n", + " },\n", + " {\n", + " \"name\": \"static_prune_50\",\n", + " \"ffn_dim\": FFN_LARGE,\n", + " \"prune_at_step\": DYNAMIC_AT_STEP,\n", + " \"prune_fraction\": 0.5,\n", + " \"regrow_method\": None,\n", + " \"dst_period\": None,\n", + " \"dst_swap_fraction\": 0.0,\n", + " \"grow_at_step\": None,\n", + " \"grow_ffn_target\": None,\n", + " },\n", + " {\n", + " \"name\": \"DST_RigL_p16hp\",\n", + " \"ffn_dim\": FFN_LARGE,\n", + " \"prune_at_step\": DYNAMIC_AT_STEP,\n", + " \"prune_fraction\": 0.5,\n", + " \"regrow_method\": \"rigl\",\n", + " \"dst_period\": DST_PERIOD_P16,\n", + " \"dst_swap_fraction\": DST_SWAP_FRACTION_P16,\n", + " \"grow_at_step\": None,\n", + " \"grow_ffn_target\": None,\n", + " },\n", + " {\n", + " \"name\": \"DST_RigL_p17hp\",\n", + " \"ffn_dim\": FFN_LARGE,\n", + " \"prune_at_step\": DYNAMIC_AT_STEP,\n", + " \"prune_fraction\": 0.5,\n", + " \"regrow_method\": \"rigl\",\n", + " \"dst_period\": DST_PERIOD_P17,\n", + " \"dst_swap_fraction\": DST_SWAP_FRACTION_P17,\n", + " \"grow_at_step\": None,\n", + " \"grow_ffn_target\": None,\n", + " },\n", + " {\n", + " \"name\": \"grown\",\n", + " \"ffn_dim\": FFN_SMALL,\n", + " \"prune_at_step\": None,\n", + " \"prune_fraction\": 0.0,\n", + " \"regrow_method\": None,\n", + " \"dst_period\": None,\n", + " \"dst_swap_fraction\": 0.0,\n", + " \"grow_at_step\": DYNAMIC_AT_STEP,\n", + " \"grow_ffn_target\": FFN_LARGE,\n", + " },\n", + "]\n", + "print(\"\\nPhase 18 config (large + long):\")\n", + "print(\n", + " f\" hidden={HIDDEN_DIM}, n_heads={N_HEADS}, n_layers={N_LAYERS}, ffn(L)={FFN_LARGE}, group={GROUP_SIZE}\"\n", + ")\n", + "print(f\" max_steps={MAX_STEPS}, dynamic_at={DYNAMIC_AT_STEP}, dst_end={DST_END_STEP}\")\n", + "print(f\" DST p16hp: period={DST_PERIOD_P16}, swap={DST_SWAP_FRACTION_P16}\")\n", + "print(f\" DST p17hp: period={DST_PERIOD_P17}, swap={DST_SWAP_FRACTION_P17}\")\n", + "print(f\" post-dynamic 학습 시간: {DYNAMIC_AT_STEP} step (Phase 17 의 {2500} 의 3x)\")" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## 2. Sweep 실행 (5 mode × 2 seed = 10 run, GPU 강력 권장)\n", + "\n", + "추정 시간: 단일 GPU 4-6 시간. 각 run ~ 25-35 분 (30M params × 15K step)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "results = {}\n", + "for mode in MODES:\n", + " for seed in SEEDS:\n", + " key = (mode[\"name\"], seed)\n", + " print(f\"\\n== mode={mode['name']} seed={seed} ==\")\n", + " cfg = HybridTransformerTrainConfig(\n", + " dataset=dataset,\n", + " vocab_size=vocab_size,\n", + " hidden_dim=HIDDEN_DIM,\n", + " n_heads=N_HEADS,\n", + " ffn_dim=mode[\"ffn_dim\"],\n", + " n_layers=N_LAYERS,\n", + " group_size=GROUP_SIZE,\n", + " arch=ARCH,\n", + " use_full_graph=True,\n", + " block_size=BLOCK_SIZE,\n", + " batch_size=BATCH_SIZE,\n", + " lr=LR,\n", + " max_steps=MAX_STEPS,\n", + " prune_at_step=mode[\"prune_at_step\"],\n", + " prune_fraction=mode[\"prune_fraction\"],\n", + " regrow_method=mode[\"regrow_method\"],\n", + " dst_period=mode[\"dst_period\"],\n", + " dst_swap_fraction=mode[\"dst_swap_fraction\"],\n", + " dst_end_step=DST_END_STEP if mode[\"regrow_method\"] else None,\n", + " grow_at_step=mode[\"grow_at_step\"],\n", + " grow_ffn_target=mode[\"grow_ffn_target\"],\n", + " seed=seed,\n", + " device=device,\n", + " )\n", + " out = train_hybrid_transformer_lm(cfg)\n", + " results[key] = out\n", + " n_cycles = len(out[\"dst_cycles\"])\n", + " print(\n", + " f\" final_loss = {out['final_loss']:.4f} (ppl = {safe_perplexity(out['final_loss']):.2f})\"\n", + " f\" params = {out['final_param_count']:,}\"\n", + " f\" sparsity = {out['final_sparsity']:.3f} dst_cycles = {n_cycles}\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## 3. 결과 표 + 자동 verdict (plateau 도달 검증 추가)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "print(\n", + " f\"{'mode':>16s} {'seed':>6s} {'final_loss':>12s} {'perplexity':>12s} \"\n", + " f\"{'params':>12s} {'sparsity':>10s}\"\n", + ")\n", + "print(\"-\" * 80)\n", + "for (name, seed), out in results.items():\n", + " fl = out[\"final_loss\"]\n", + " print(\n", + " f\"{name:>16s} {seed:>6d} {fl:>12.4f} {safe_perplexity(fl):>12.2f} \"\n", + " f\"{out['final_param_count']:>12,} {out['final_sparsity']:>10.3f}\"\n", + " )\n", + "\n", + "# mode 별 평균\n", + "print(\"\\n== Mode summary (mean ± σ across seeds) ==\")\n", + "summary = {}\n", + "for mode in MODES:\n", + " name = mode[\"name\"]\n", + " vals = [results[(name, s)][\"final_loss\"] for s in SEEDS]\n", + " params = [results[(name, s)][\"final_param_count\"] for s in SEEDS]\n", + " sparsities = [results[(name, s)][\"final_sparsity\"] for s in SEEDS]\n", + " m = statistics.mean(vals)\n", + " sd = statistics.stdev(vals) if len(vals) > 1 else 0.0\n", + " summary[name] = (m, sd, statistics.mean(params), statistics.mean(sparsities))\n", + " print(\n", + " f\" {name:>16s} {m:.4f} ± {sd:.4f} (ppl ≈ {safe_perplexity(m):.2f}) \"\n", + " f\"params={statistics.mean(params):,.0f} sparsity={statistics.mean(sparsities):.3f}\"\n", + " )\n", + "\n", + "# 자동 verdict\n", + "print(\"\\n== Verdict ==\")\n", + "dense_loss = summary[\"dense_large\"][0]\n", + "static_loss = summary[\"static_prune_50\"][0]\n", + "rigl_p16_loss = summary[\"DST_RigL_p16hp\"][0]\n", + "rigl_p17_loss = summary[\"DST_RigL_p17hp\"][0]\n", + "grown_loss = summary[\"grown\"][0]\n", + "\n", + "all_finite = all(math.isfinite(out[\"final_loss\"]) for out in results.values())\n", + "verdict_1 = \"PASS\" if all_finite else \"FAIL\"\n", + "print(f\"1. all-finite: {all_finite} [{verdict_1}]\")\n", + "\n", + "diff_rigl_static = rigl_p16_loss - static_loss\n", + "verdict_2 = \"PASS\" if diff_rigl_static <= 0.02 else \"FAIL\"\n", + "print(\n", + " f\"2. DST_RigL_p16hp ≤ static + 0.02 (순수 scale + long-train 효과): \"\n", + " f\"diff = {diff_rigl_static:+.4f} [{verdict_2}]\"\n", + ")\n", + "if diff_rigl_static < 0:\n", + " print(\" → DST advantage 발현! Phase 16a 의 scale-limit 가설 입증.\")\n", + "elif diff_rigl_static < 0.005:\n", + " print(\" → DST 가 static 근방까지 도달 — 부분 advantage.\")\n", + "else:\n", + " print(\" → DST 여전히 static 보다 약간 열위.\")\n", + "diff_hp = rigl_p17_loss - rigl_p16_loss\n", + "print(f\" 2b. hp 효과: p17hp - p16hp = {diff_hp:+.4f}\")\n", + "\n", + "diff_grown_dense = grown_loss - dense_loss\n", + "verdict_3 = \"PASS\" if diff_grown_dense <= 0.02 else \"FAIL\"\n", + "print(\n", + " f\"3. grown ≤ dense_large + 0.02 (16b 재검증, long post-grow): \"\n", + " f\"diff = {diff_grown_dense:+.4f} [{verdict_3}]\"\n", + ")\n", + "\n", + "# 4. plateau 도달 검증 — Phase 18 의 핵심 verdict\n", + "# 절댓값 기준: |drift| < 0.001 만 plateau. 음수 큰 값 = 계속 학습 중, 양수 = 발산/과적합.\n", + "PLATEAU_EPS = 0.001\n", + "print(f\"\\n4. plateau 도달 검증 (|drift| < {PLATEAU_EPS} = true plateau):\")\n", + "all_plateau = True\n", + "for name in [m[\"name\"] for m in MODES]:\n", + " drifts = []\n", + " for seed in SEEDS:\n", + " losses = results[(name, seed)][\"losses\"]\n", + " last_500 = losses[-500:]\n", + " half_len = len(last_500) // 2\n", + " if half_len > 0:\n", + " first_half = sum(last_500[:half_len]) / half_len\n", + " second_half = sum(last_500[half_len:]) / (len(last_500) - half_len)\n", + " drifts.append(second_half - first_half)\n", + " else:\n", + " drifts.append(0.0)\n", + " mean_drift = sum(drifts) / len(drifts)\n", + " plateau_reached = abs(mean_drift) < PLATEAU_EPS\n", + " all_plateau = all_plateau and plateau_reached\n", + " if plateau_reached:\n", + " status = \"plateau\"\n", + " elif mean_drift < 0:\n", + " status = \"still learning\"\n", + " else:\n", + " status = \"diverging\"\n", + " marker = \"✓\" if plateau_reached else \"✗\"\n", + " print(f\" {name:>16s} mean drift = {mean_drift:+.4f} {marker} ({status})\")\n", + "verdict_4 = \"PASS\" if all_plateau else \"FAIL\"\n", + "print(f\" → all-mode plateau 도달: {all_plateau} [{verdict_4}]\")\n", + "if not all_plateau:\n", + " print(\n", + " \" → 일부 mode 미수렴 (still learning) 또는 발산 (diverging) — 본 sweep 도 수렴 영역 미달 가능성\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "## 4. Loss curve 시각화 — 15K step 전체" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(1, 1, figsize=(14, 6))\n", + "colors = {\n", + " \"dense_large\": \"tab:green\",\n", + " \"static_prune_50\": \"tab:red\",\n", + " \"DST_RigL_p16hp\": \"tab:purple\",\n", + " \"DST_RigL_p17hp\": \"tab:blue\",\n", + " \"grown\": \"tab:orange\",\n", + "}\n", + "window = 200 # Phase 18 의 longer training 에 맞춰 window 도 키움\n", + "\n", + "for mode in MODES:\n", + " name = mode[\"name\"]\n", + " losses_per_seed = [results[(name, s)][\"losses\"] for s in SEEDS]\n", + " smoothed = []\n", + " for losses in losses_per_seed:\n", + " smoothed.append(\n", + " [\n", + " sum(losses[max(0, i - window + 1) : i + 1]) / min(i + 1, window)\n", + " for i in range(len(losses))\n", + " ]\n", + " )\n", + " arr = torch.tensor(smoothed)\n", + " mean = arr.mean(dim=0)\n", + " std = arr.std(dim=0) if arr.size(0) > 1 else torch.zeros_like(mean)\n", + " steps = list(range(len(mean)))\n", + " ax.plot(steps, mean, label=name, color=colors[name], linewidth=1.5)\n", + " ax.fill_between(steps, mean - std, mean + std, color=colors[name], alpha=0.12)\n", + "\n", + "ax.axvline(\n", + " DYNAMIC_AT_STEP,\n", + " color=\"black\",\n", + " linestyle=\":\",\n", + " alpha=0.5,\n", + " label=f\"prune/grow @ {DYNAMIC_AT_STEP}\",\n", + ")\n", + "ax.axvline(DST_END_STEP, color=\"gray\", linestyle=\":\", alpha=0.4, label=f\"DST end @ {DST_END_STEP}\")\n", + "\n", + "ax.set_xlabel(\"step\")\n", + "ax.set_ylabel(f\"loss (rolling mean w={window})\")\n", + "ax.set_title(\n", + " \"Phase 18 large+long (hidden=512, n_layers=12, ffn(L)=1024, steps=15000) — 5 mode (mean ± σ over 2 seeds)\"\n", + ")\n", + "ax.legend(loc=\"upper right\")\n", + "ax.grid(alpha=0.3)\n", + "plt.tight_layout()\n", + "\n", + "out_dir = Path(\"../../runs/notebook-neuron-phase18\")\n", + "out_dir.mkdir(parents=True, exist_ok=True)\n", + "fig.savefig(out_dir / \"loss_curves.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(f\"saved: {out_dir / 'loss_curves.png'}\")" + ] + }, + { + "cell_type": "markdown", + "id": "11", + "metadata": {}, + "source": [ + "## 5. Phase 16/17/18 3단계 progression 비교" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "# Phase 16/17 baseline 수치 (Notion 정리 페이지 출처)\n", + "# Phase 16a: https://www.notion.so/36de8b70b7aa819aaa54d561fc222fa1\n", + "# Phase 16b: https://www.notion.so/36ee8b70b7aa811bb354c4fc98a484ed\n", + "# Phase 17: https://www.notion.so/36fe8b70b7aa8168a61ce9fd27cf8625\n", + "p16 = {\n", + " \"dense\": 1.8999,\n", + " \"static_prune_50\": 1.9777,\n", + " \"DST_RigL_p16hp\": 1.9831, # 16a 의 DST_RigL_50 (period=50, swap=0.1)\n", + " \"grown\": 1.9428,\n", + "}\n", + "p17 = {\n", + " \"dense\": 1.3326,\n", + " \"static_prune_50\": 1.3496,\n", + " \"DST_RigL_p16hp\": 1.3588,\n", + " \"grown\": 1.3582,\n", + "}\n", + "p18 = {\n", + " \"dense\": summary[\"dense_large\"][0],\n", + " \"static_prune_50\": summary[\"static_prune_50\"][0],\n", + " \"DST_RigL_p16hp\": summary[\"DST_RigL_p16hp\"][0],\n", + " \"grown\": summary[\"grown\"][0],\n", + "}\n", + "\n", + "fig, ax = plt.subplots(1, 1, figsize=(11, 5))\n", + "modes_plot = [\"dense\", \"static_prune_50\", \"DST_RigL_p16hp\", \"grown\"]\n", + "x = list(range(len(modes_plot)))\n", + "width = 0.27\n", + "p16_vals = [p16[m] for m in modes_plot]\n", + "p17_vals = [p17[m] for m in modes_plot]\n", + "p18_vals = [p18[m] for m in modes_plot]\n", + "ax.bar([i - width for i in x], p16_vals, width, label=\"Phase 16 (~1M, 1.5K step)\", color=\"#bcd5e6\")\n", + "ax.bar(x, p17_vals, width, label=\"Phase 17 (~6M, 5K step)\", color=\"#5b8db8\")\n", + "ax.bar([i + width for i in x], p18_vals, width, label=\"Phase 18 (~30M, 15K step)\", color=\"#1e3a5f\")\n", + "ax.set_xticks(x)\n", + "ax.set_xticklabels(modes_plot, rotation=15)\n", + "ax.set_ylabel(\"final loss (mean over 2 seeds)\")\n", + "ax.set_title(\"Phase 16 → 17 → 18 — scale progression (same hp)\")\n", + "ax.legend()\n", + "ax.grid(alpha=0.3, axis=\"y\")\n", + "plt.tight_layout()\n", + "fig.savefig(out_dir / \"phase16_17_18_progression.png\", dpi=150, bbox_inches=\"tight\")\n", + "plt.show()\n", + "print(f\"saved: {out_dir / 'phase16_17_18_progression.png'}\")\n", + "\n", + "# 격차 변화 정량화\n", + "print(\"\\n== dynamic advantage trend (Phase 16 → 17 → 18) ==\")\n", + "for label, a_key, b_key in [\n", + " (\"DST - static\", \"DST_RigL_p16hp\", \"static_prune_50\"),\n", + " (\"grown - dense\", \"grown\", \"dense\"),\n", + "]:\n", + " d16 = p16[a_key] - p16[b_key]\n", + " d17 = p17[a_key] - p17[b_key]\n", + " d18 = p18[a_key] - p18[b_key]\n", + " print(f\" {label:>16s}: P16={d16:+.4f} P17={d17:+.4f} P18={d18:+.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "13", + "metadata": {}, + "source": [ + "## 6. 결론 / 다음 단계\n", + "\n", + "(셀 출력 보고 사용자가 채울 영역)\n", + "\n", + "**핵심 질문**:\n", + "1. **plateau 도달했나?** verdict 4 결과\n", + "2. **DST advantage 발현?** verdict 2 의 diff 가 음수 또는 0.005 이내?\n", + "3. **grown 이 dense 수준까지 도달?** verdict 3 의 diff 가 0.02 이내?\n", + "4. **Phase 16 → 17 → 18 의 trend**: dynamic advantage 의 progression?\n", + "\n", + "**Phase 19 후보** (사용자 결정):\n", + "- (plateau + dynamic advantage 발현) → **결과 정리 + paper writing**\n", + "- (plateau but dynamic 여전히 미실현) → method 개선 (Net2WiderNet duplication, RigL scheduler) 또는 real dataset\n", + "- (plateau 미도달) → 더 긴 학습 또는 더 큰 model" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "GraphLM (uv .venv)", + "language": "python", + "name": "graphlm-uv-venv" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file