Skip to content

fix(core): reduce Windows CPU under parallel sessions - #43769

Draft
Hona wants to merge 9 commits into
anomalyco:v2from
Hona:windows-cpu-final
Draft

fix(core): reduce Windows CPU under parallel sessions#43769
Hona wants to merge 9 commits into
anomalyco:v2from
Hona:windows-cpu-final

Conversation

@Hona

@HonaHona commented Aug 21, 2026

Copy link
Copy Markdown
Member

Cutting Windows server CPU without weakening snapshots

TL;DR

Snapshot CPU fell from 2,250 ms to 515 ms across ten clean completed steps (-77.1%). Git launches fell from 70 to 40. A stale-lock retry burst fell from 1,438 ms to 31 ms (-97.8%).

The baseline is upstream/v2 commit 212139ff95bb725d698bfe657d6fda37cbbfa01b. This replaces the closed draft that was incorrectly based on dev.

CommitChangeDirect evidenceCumulative target
2df9fa7Combine snapshot candidate scans, make filtering linear, and skip known-equal step comparisonsClean-step Git launches: 70 -> 40; CPU: 2,250 -> 1,219 ms (-45.8%)Snapshot process count
cc9cebeResolve Git once and directly spawn absolute native Windows executablesClean-step CPU: 1,219 -> 438 ms (-64.1% incremental, -80.5% from base)36.2% live statSync hotspot
9446514Remove tracing only from high-frequency provider deltasProvider fragment throughput: 63,464 -> 82,481 Events/s (+30%); CPU: 5,937 -> 4,688 ms (-21%)Delta telemetry overhead
d46b8efStream/discard ripgrep indexing, build path prefixes once, and reuse prepared target arraysTargets the 10.22% cold-start filesystem-index pathWindows startup and autocomplete
97e540cCache durable Event codecs by immutable definitionOne Schema adapter per definition instead of per durable Event8.09% Schema ancestry
ab08e20Cache immutable Effect tool schema compilation and codecsDefinitions/s: 19,115 -> 54,530 (+185%); CPU: 578 -> 94 ms (-83.7%)Repeated physical model attempts
ba040efCoordinate snapshot writers across processes and back off lock failuresTen stale-lock calls: 30 Git launches -> 3; CPU: 312 -> 32 ms (-89.7% incremental)Cross-channel snapshot safety
bb43664Keep a mixed file-search target cache current during an active scanRegression coverage for partial initial indexesSearch correctness
85347fePreserve boundary contracts found in reviewRetries failed initialization, keeps Git validation, traces non-deltas, and avoids caching mutable Standard schemasReview hardening

Percentages from CPU profiles are overlapping sampled-stack categories. Do not add them.

What the real beta server does

A 30-second profiler plugin ran inside the installed beta server process (0.0.0-beta-17759, PID 34464) under the real application workload. It did not use a generated Event benchmark.

MetricResult
Process CPU8,609 ms over 30 seconds
Average CPU28.7% of one core
Git launches in the profile window73
Ripgrep launches2
PowerShell launches1
statSync self samples36.2% of active samples
Native spawn self samples14.3% of active samples
SQLite values self samples2.5% of active samples

The profile's representative statSync stack enters the bundled cross-spawn command-resolution path. The matching server log confirms that 73 of 76 child launches were Git. The server was resolving and launching about 2.43 Git processes per second during this window.

The earlier 81.7-second profile (0.0.0-beta-17498) showed the same shape:

  • 70.45% post-start process ancestry;
  • 52.11% post-start native spawn leaf samples;
  • 11.81% post-start executable resolution;
  • 36 snapshot failures against one private index.lock;
  • an older server/channel process accessing the same snapshot repository.

This is primarily a Windows process-launch and snapshot-amplification problem, not model inference or EventFeed encoding.

Snapshot flow before this branch

Each physical model attempt captures the filesystem before llm.stream(...). A terminal attempt captures it again and compares the two trees.

One clean capture used:

git diff-files --name-only
git ls-files --others --exclude-standard
git write-tree

The completed step then ran git diff --name-only even when the two captured tree IDs were equal.

start capture: 3 Git processes
end capture: 3 Git processes
comparison: 1 Git process
total: 7 Git processes

Ten clean completed steps therefore launched 70 Git processes.

Snapshot flow after this branch

Candidate discovery now uses one tagged, NUL-delimited git ls-files call:

git ls-files --modified --others --exclude-standard -t -z

The tag keeps tracked and untracked paths distinct, so the 2 MiB limit still applies only to untracked files. Tests cover tracked changes, deletion, untracked additions, status-like filename prefixes, and oversized untracked files.

The Session runner now skips comparison only when its two freshly captured IDs are equal. Git.tree.files still validates arbitrary IDs, including equal invalid IDs.

start capture: 2 Git processes
end capture: 2 Git processes
comparison: 0 Git processes when freshly captured IDs match
total: 4 Git processes

Snapshots remain Git trees. This PR does not replace snapshot storage or correctness with raw filesystem watchers. Restore, diff, ignored-file filtering, external edits, and the persistent private index keep their existing semantics.

Windows executable resolution

The V2 spawner currently sends every command through cross-spawn. On Windows, each bare git launch can repeat PATH, PATHEXT, which, isexe, statSync, and shebang work before native process creation.

The Git service now resolves Git once. A relative lookup result is normalized once against the server startup directory. The process spawner bypasses cross-spawn only when all of these conditions hold:

  • Windows;
  • an absolute executable path;
  • .exe or .com extension;
  • no shell option.

Shell commands, .cmd, .bat, shebang scripts, and relative commands keep the existing cross-spawn behavior. Focused Windows tests cover the direct native executable and spaced .cmd paths.

This commit does not reduce Git command count. Its direct value is removing repeated synchronous executable discovery from every remaining Git launch.

Snapshot ownership and failure control

V2 already serializes one process's private-index mutations with a KeyedMutex. It did not coordinate separate beta, local, or replacement server processes that share the same deterministic snapshot path.

Snapshot repository initialization, capture, and restore now use the existing EffectFlock boundary. Its heartbeat is renewed every staleMs / 3 until release. Different worktrees still use different lock keys and run independently.

Lock-related capture failures use bounded backoff:

5s -> 10s -> 20s -> 40s -> 60s

The code does not delete Git lock files. A failed lazy initialization invalidates its cached fiber, so a later call can recover after contention clears.

This changes only an already-failed best-effort capture path. Successful snapshots do not back off.

Provider delta path

Normal text and reasoning output is already batched before public Session delta publication. The raw provider dispatcher still entered a named traced Effect.fn for every provider chunk, before batching.

The final implementation keeps SessionRunner.publishLLMEvent spans for:

  • step boundaries;
  • text/reasoning starts and ends;
  • tool calls and results;
  • provider errors;
  • finish Events.

Only text, reasoning, and tool-input deltas use the untraced dispatcher. Anthropic's content-block delta handler is also untraced.

The benchmark reconstructs provider fragment events from 583 real V2 Session fragment Events. Each run processes 291,500 provider Events. Results use seven runs, remove the fastest and slowest complete runs by wall time, then take the median of the remaining five.

ModeEvents/sCPU timeCPU change
Traced63,4645,937 msbaseline
Delta-only untraced82,4814,688 ms-21%

File indexing

Windows uses the ripgrep/fuzzysort fallback because FFF is disabled there by default. The cold profile attributed 10.22% of pre-cutoff samples to filesystem indexing.

The old scan:

  • retained every parsed ripgrep row even though search consumed only the callback;
  • repeatedly used slice(...).join(...) for every directory prefix;
  • rebuilt prepared target arrays for every query.

The new Ripgrep.scan operation has a void result and requires onEntry, so callers cannot silently request discarded results through an optional boolean. File search builds prefixes in one pass, retains one prepared target per path, and invalidates its mixed list when new scan entries arrive.

Event and tool schemas

Durable Bus data codecs are cached by immutable Event definition. Replay and publication retain the same Schema validation.

Tool runtime caching is deliberately narrower:

  • Effect Schemas are immutable and cache JSON Schema compilation plus codec adapters;
  • every request receives a deep fresh JSON-schema value for mutable plugin hooks;
  • Standard-schema converters are not cached because they may reflect mutable plugin state;
  • raw JSON schemas keep their existing behavior.

The tool benchmark registers 40 nested Effect-schema tools and renders 4,000 definitions per run. It uses the same seven-run trimmed-median rule.

ModeDefinitions/sCPU timeCPU change
Uncached19,115578 msbaseline
Cached54,53094 ms-83.7%

Snapshot benchmark

The snapshot fixture has 200 tracked files. Each run measures:

  • ten complete clean steps (start capture, end capture, comparison);
  • one changed complete step;
  • ten capture calls while a stale Git index.lock exists.

Every commit was run seven times. For each scenario, the fastest and slowest complete runs were removed by wall time, then the median of the remaining five was reported.

Clean completed steps

RevisionGit launchesWall timeCPU timeCPU change
V2 base705,434 ms2,250 msbaseline
Combined scan + equal skip404,147 ms1,219 ms-45.8%
Absolute native Git403,021 ms438 ms-80.5% cumulative
Final branch403,029 ms515 ms-77.1% cumulative

Changed completed step

RevisionGit launchesWall timeCPU timeCPU change
V2 base10883 ms359 msbaseline
Combined scan8849 ms235 ms-34.5%
Absolute native Git8699 ms94 ms-73.8% cumulative

Stale-lock calls

RevisionGit launchesWall timeCPU timeCPU change
V2 base403,335 ms1,438 msbaseline
Process optimizations302,345 ms280 ms-80.5%
Flock + backoff3222 ms32 ms-97.8% cumulative

Event replay

A separate real trace contained 1,961 public V2 Events. The long four-subscriber replay processed 49,025 source Events and 196,100 delivered frames per run.

The full branch changed delivered throughput from 31,437 to 35,608 Events/s (+13.3%) and CPU from 49.72 to 47.57 ms per thousand delivered Events (-4.3%). This benchmark is reported only as an aggregate regression check. It is not used to attribute snapshot, spawn, search, or tool-schema commits because those paths are outside EventFeed.

Verification

  • Pre-push workspace typecheck: 34/34 tasks passed.
  • Core, AI, Util, and Server package typechecks passed.
  • Git tree tests: 2 passed.
  • Snapshot tests: 7 passed.
  • Bus, ripgrep, filesystem search, tool schema, and registry tests: 93 passed.
  • Provider-delta Session runner tests: 2 passed.
  • Anthropic protocol tests: 50 passed.
  • Windows absolute executable test: passed.
  • Windows spaced .cmd routing test: passed.
  • One unchanged cross-spawn echo assertion remains platform-specific when the full file runs: Windows returns quoted output. Focused changed routes pass.

Compatibility

  • Public Protocol and Server HttpApi contracts do not change.
  • Snapshot IDs, Git trees, diff behavior, restore behavior, and user-index isolation do not change.
  • No Git lock file is deleted automatically.
  • Different worktrees retain parallel snapshot execution.
  • Non-delta operational spans remain.
  • Standard-schema converter mutability remains observable.
  • No generated client files change.

The result keeps the normal flow direct: capture with fewer Git commands, launch Git without repeated Windows lookup, compare only when tree IDs differ, and stop retrying an already-failed lock on every model boundary.

Hona added 9 commits August 21, 2026 13:28
Combine tracked and untracked candidate discovery, skip equal-tree comparisons, and make snapshot filtering linear. A clean completed model step now needs four Git processes instead of seven.
Resolve Git once and spawn absolute native Windows executables directly. Shell commands, scripts, cmd files, and shebang routing continue through cross-spawn.
Keep model-call and step spans while making raw provider event dispatch and Anthropic content-delta handling untraced.
Stream ripgrep entries without retaining a duplicate result, build directory prefixes once, and reuse prepared target arrays between fuzzy queries.
Cache Schema data encoders and decoders by immutable event definition instead of rebuilding adapters for every publish and replay.
Cache immutable JSON Schema compilation and Effect codec adapters by schema identity while returning fresh request-local schema objects for plugin hooks.
Use the existing stale-safe process lock around snapshot repository mutation. Lock-related capture failures back off from five to sixty seconds and recover without deleting Git lock files.
Invalidate the cached mixed target list as new entries arrive so searches during the initial scan observe later files and directories.
Retry transient snapshot initialization, keep equal-tree validation at the Git boundary, trace non-delta provider events, cache only immutable Effect schemas, and expose discard-mode ripgrep scans as a distinct operation.
@Enough1122

Copy link
Copy Markdown

AI code review — automated review for reference; please use your judgment.

  • packages/core/src/git.ts:12 — which("git") now runs unconditionally on every platform at module import, changing POSIX behavior too: command resolution moves from spawn-time PATH lookup to import-time (stale if PATH/env changes later), adds startup cost even when VCS is unused, and an exception inside which() would break importing the module — gate it behind process.platform === "win32" like the spawner predicate, wrap in try/catch, and prefer lazy memoized resolution inside the layer.
  • packages/core/src/snapshot.ts:96 — All locks share LOCK_TIMEOUT_MS = 30s, including capture, yet the retry/backoff machinery (retryAt/retryMs) only helps AFTER a timeout fires — under the parallel-session load this PR targets, a second session's capture can now block up to 30s waiting for the flock instead of failing fast and skipping; a much shorter capture-specific timeout (e.g., 100-500ms) feeding the existing backoff would match the stated goal of reducing cross-session interference.
  • packages/core/src/snapshot.ts:130 — tapError now invalidates the cached repository fiber on ANY initialization failure, so permanent errors like "Project is not a Git repository" are fully re-run (discover + lock + create attempt) on every subsequent capture — reasonable for transient recovery, but consider distinguishing permanent failures (no invalidation) from transient/lock ones, or bounding consecutive re-init attempts so a misconfigured project doesn't churn locks forever.
  • packages/core/src/filesystem/search.ts:88 — combinedTargets invalidation happens only in the file-entry branch, never when directories are added; correctness silently relies on the invariant that every directory push occurs in the same onEntry callback as (and after) a file push — true today because directories are derived from file paths, but a future entry type (symlink dir, bare directory scan) would break it invisibly; either invalidate in both branches or pin the invariant with a loud comment plus a regression test.
  • packages/core/src/git.ts:430 — The refresh rewrite collapses two git calls into one ls-files --modified --others -t -z pass — nice win — but the tracked/untracked split depends entirely on undocumented -t tag formatting ("? " prefix, 2-char tags with -z); add a brief comment citing the git docs tags (H/S/M/R/C/K/?) and keep the deletion coverage (R-tagged worktree deletions) pinned by the new test so a git output change surfaces immediately.
  • packages/core/src/tool/runtime.ts:122 — Cached JSON schemas are structuredClone'd on every definition() call to preserve the "fresh object per call" contract the new test enforces; for large tool schemas invoked per-request this clone may rival the toJsonSchema cost being optimized away — consider Object.freeze on the cached value instead (and asserting immutability rather than freshness), or measure to confirm the clone is cheap enough.

— AI code review (automated)

renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 23, 2026
…0984)
Repeated idle writes republished Status and Idle events to every
connected client. SessionStatus.set now publishes only when
transitioning from a non-idle state; repeated idle writes are no-ops.
README: ported-table row, deferred-ports section for anomalyco#43769/anomalyco#40698
(tree shapes absent at v1.18.21), enriched deleted-session-cleanup row.
Co-authored-by: zcxGGmu <zcxGGmu@users.noreply.github.com>
renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 26, 2026
- pin note, seam-lookup range, and deferred-port re-verification now
reference v1.18.23 (post-split packages/ai|util tree still absent,
anomalyco#43769 stays blocked)
- watch-list gains the 2026-08-26 sweep candidates (anomalyco#39930, anomalyco#38939,
anomalyco#41950, anomalyco#33713, anomalyco#44631)
- fork-build.sh BASE lookup now excludes *-lowmem.* tags: after a fresh
upstream merge the previous fork tag ties the new base tag on commit
distance with a newer date and git describe stamps the OLD base
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Hona@Enough1122