feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(cli): posthog feature flags + fast_provision experiment - #3366

Merged
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision
Apr 28, 2026
Merged

feat(cli): posthog feature flags + fast_provision experiment#3366
la14-1 merged 2 commits into
OpenRouterLabs:mainfrom
AhmedTMM:feat/feature-flags-fast-provision

Conversation

@AhmedTMM

Copy link
Copy Markdown
Collaborator

Summary

Wires PostHog `/decide` into the CLI so we can A/B-test provisioning behaviors with feature flags. First experiment: `fast_provision` — for users who didn't pass `--beta` or `--fast` manually, the `test` variant turns on `tarball + images` by default to see if faster provisioning lifts the late-funnel conversion rate.

The PostHog experiment was already created in the dashboard; this PR is the code side of it.

Design calls

Why `tarball,images` and not the full `--fast` set (`+parallel,docker`)? Clean attribution. The hypothesis is specifically about tarball/image; if we ship the full `--fast` bundle we can't tell which feature moved the metric. `--fast` stays as the power-user knob.

Why share `distinct_id` with telemetry? PostHog identity needs to match across telemetry events and flag decisions, otherwise the experiment's exposure events don't line up with the funnel events they're supposed to attribute. Telemetry already had a persistent user-id at `~/.config/spawn/.telemetry-id` — moved that into a shared `install-id.ts` module so feature flags reuse it. Existing users keep their bucket.

On-disk cache with 1h TTL. Without a cache, every `spawn` invocation pays a 1.5s network call. Stale-while-revalidate via the cache file means cold starts get a near-instant variant, refreshes happen lazily.

User-wins. If the user passes `--beta tarball` or `--fast`, the flag is bypassed entirely. `SPAWN_FEATURE_FLAGS_DISABLED=1` is a hard kill switch.

Files

  • `shared/install-id.ts` (new) — UUID generation/read with disk-failure fallback
  • `shared/feature-flags.ts` (new) — hand-rolled `/decide` POST, 1.5s timeout, fail-open, on-disk cache, exposure events
  • `shared/telemetry.ts` — `distinct_id` now sourced from `install-id.ts`
  • `shared/paths.ts` — adds `getInstallIdPath()` (returns existing telemetry-id path)
  • `index.ts` — `await initFeatureFlags()` early in `main()`; applies `fast_provision` test variant after `--beta`/`--fast` composition (so they win)
  • 14 new unit tests across `install-id.test.ts` and `feature-flags.test.ts`

Rollout

Recommend ramping the PostHog flag at 5% → 25% → 50% → 100% on the `test` variant with 24h between bumps. The 1.5s fail-open timeout is itself a soft kill switch — if PostHog is down, every user gets control.

Test plan

  • `bunx @biomejs/biome check src/` — 0 errors over 199 files
  • `bunx tsc --noEmit -p .` — 0 production errors
  • `bun test` — 2183 pass, same 4 pre-existing failures as upstream/main
  • New tests: install-id roundtrip + format guard; feature-flags fetch/HTTP500/malformed/disabled/idempotent/stale-cache; exposure event capture
  • End-to-end: spawn with experiment flag set to `test` in PostHog → confirm `SPAWN_BETA=tarball,images` is set
  • Verify `$feature_flag_called` events arrive in PostHog tagged correctly to the experiment

Bumps CLI to 1.0.23.

🤖 Generated with Claude Code

AhmedTMMand others added 2 commits April 27, 2026 16:27
Wires PostHog `/decide` into the CLI so we can A/B-test provisioning
behaviors. First experiment: `fast_provision` — for users who didn't
pass --beta or --fast manually, the `test` variant turns on
`tarball + images` by default. Hypothesis: faster provisioning →
fewer drop-offs in the "VM ready → install completed" leg of the
funnel.
What's added:
- `shared/install-id.ts` — stable per-machine UUID, persisted at
~/.config/spawn/.telemetry-id. Reuses telemetry's existing path
so existing users keep their PostHog identity. Falls back to an
ephemeral UUID on disk-write failure.
- `shared/feature-flags.ts` — hand-rolled POST to PostHog /decide
(no SDK dep). 1.5s timeout, fail-open. On-disk cache at
$SPAWN_HOME/feature-flags-cache.json with 1h TTL so cold starts
don't pay the network cost. SPAWN_FEATURE_FLAGS_DISABLED=1 kill
switch. Captures `$feature_flag_called` exposure events for both
arms so PostHog can compute conversion.
- `shared/telemetry.ts` — moves user-id loading into install-id.ts
so flags and events share the same `distinct_id`.
- `index.ts` — `await initFeatureFlags()` at the top of `main()`,
then applies `fast_provision`'s `test` variant by appending
`tarball,images` to SPAWN_BETA — but only if the user didn't
pass --beta or --fast (those always win, so opt-out is free).
Why tarball+images and not all four (`+parallel,docker`):
clean attribution. The hypothesis is about tarball/image; if we
ship the full --fast bundle we can't tell which feature moved the
metric. Keep --fast as the user-facing power-user knob.
Tests: 14 new (install-id roundtrip + format guard, feature-flags
fetch/timeout/HTTP500/malformed/disabled/idempotent/stale-cache,
exposure-event behavior). Full suite: 2183 pass, same 4 pre-existing
failures as upstream/main.
Bumps CLI to 1.0.23.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt real SWR
Two review-fix commits from PR feedback squashed into one:
1. Move `await initFeatureFlags()` below the `spawn pick` and
`spawn feedback` bypass clauses in `main()`. Both commands are called
from bash scripts and must stay fast; neither gates on a flag, so
there's no reason to pay up to 1.5s of network latency on cold cache.
2. Implement real stale-while-revalidate in `shared/feature-flags.ts`.
The prior implementation did a synchronous fetch on stale cache,
which contradicted the docstring and PR description. Now:
- fresh cache (<TTL) → use cache, no network
- stale cache (>=TTL) → use cache immediately, refresh in background
- no cache → await sync fetch (first run only)
Adds `_awaitBackgroundRefreshForTest()` so tests can deterministically
wait for the background refresh before asserting. Updated the existing
"stale cache" test to verify SWR semantics (stale served first, fresh
lands next invocation) and added a "fresh cache does not fetch" test.
All 2127 tests pass; biome clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@la14-1

Copy link
Copy Markdown
Collaborator

Applied the two must-fix items from review in d2ec13d:

  1. Fast-path guard: moved await initFeatureFlags() below the spawn pick and spawn feedback bypass clauses in main(). Shell-invoked fast paths no longer pay the up-to-1.5s flag-fetch cost.
  2. Real SWR: initFeatureFlags() now serves stale cache immediately and refreshes in the background (fire-and-forget), matching the docstring and PR description. No cache → still awaits a bounded sync fetch for the first-run case.

Test coverage:

  • Renamed the >1h stale cache re-fetches test to assert SWR semantics (stale served first, fresh lands next invocation) via a new _awaitBackgroundRefreshForTest() helper.
  • Added does NOT fetch when cache is fresh (<1h old) to pin down the no-network path.

All 2127 tests pass, biome clean.

@la14-1la14-1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review fixes applied: fast-path skip + real SWR. All checks green.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@AhmedTMM@la14-1@claude