perf(ai): unify inference pipeline LRU + dispose-on-evict (Phase 2.3) - #69
Merged
Conversation
Both inference workers carried byte-identical pipeline-LRU logic and neither disposed the evicted pipeline -> VRAM/RAM leak (same class as the WebLLM eviction fix, AUDIT 2026-06-01 #1). Extract services/ai/pipelineLruCache.ts: dispose-on-evict, in-flight load dedup (no double-load of multi-MB models), injectable clock for deterministic tests. Rewire workers/inference.worker.ts and workers/v2/inference.worker.ts onto it; remove the duplication. Self-review of same-day commits logged in AUDIT.md (aiRetry property tests + useLoraView selector nit -> Phase 3). Latency telemetry already exists at the facade, so no new worker->main hop added. lint + typecheck green; 9 new cache tests + 7 existing worker tests pass (behavior-preserving). Coverage/smoke:prod via CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The latest updates on your projects. Learn more about Vercel for GitHub.
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
CodeAnt AI finished reviewing your PR. |
…on (Phase 3) Self-review hardening of the same-day commits: - aiRetry.ts: add property/invariant tests for computeRetryDelayMs (exponential, non-decreasing, capped at AI_RETRY_MAX_DELAY_MS, full-jitter in [0,capped)) and parseRetryAfterMs (ms/seconds/string/header forms, hostile-value clamp), plus an integration test asserting a server Retry-After beats the computed backoff. 19 tests. - useLoraView.ts: selectDatasetForProject(projectId) is a createSelector factory that returned a fresh memoized selector each render, defeating memoization. Wrap in useMemo keyed on projectId; module-level stable empty selector for the no-project path. 12 existing hook tests stay green (behavior-preserving). lint + typecheck green; 31 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address 3 CodeAnt findings on the pipeline LRU cache: - set() overwriting a live key now disposes the previous value (was a leak); a no-op when the value is identical. - PipelineLruCache.safeDispose() centrally swallows synchronous throws and async rejections from the dispose callback, so a failing backend disposal can never surface as an unhandled rejection in worker/main contexts. Both worker dispose callbacks simplified to return the (possibly async) result; the cache catches it. +4 tests (replace-disposes-old, identical-no-dispose, sync-throw-safe, async-reject-safe). lint + typecheck green; 13 cache + 7 worker tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ASM (Phase 2.4) Correction: sileroVadEngine.ts and kokoroTtsEngine.ts already had tests since 2026-05-31 — the TODO 'covered 0 tests' note was stale. Real gap was the Kokoro playback-control surface: add cancel() (no-op + stops an in-flight source), pause()/resume() + dispose() AudioContext delegation, and the no-WebAssembly isAvailable() branch. +4 tests (10 total). Inference-worker LRU is covered via pipelineLruCache.test.ts. typecheck + lint green; 10 Kokoro + 5 Silero tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nd Quick Start (Phase 4) - docs/adr/0001-state-management-boundaries.md — formalizes Redux (persisted/undo) vs Zustand transientUiStore (ephemeral) so the recurring 'why two state libs?' audit question has a decision record instead of prose. - docs/adr/0002-local-ai-stack-layering.md — WebLLM->ONNX->Transformers.js->heuristic fallback chain, shared infra (pipelineLruCache, aiInferenceCache, aiRetry, worker-bus), and the honest-degradation contract. - docs/adr/README.md — ADR index; linked from README Documentation Hub. - README: '⚡ Quick Start (60 seconds)' section + TOC entry — browser-first, no-key onboarding path to first value. Docs-only; no code or i18n keys changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After Phase 2.3/2.4 tests, CI (PR #69, Node 22 & 24) measured coverage at 75.15 L / 61.23 B / 67.84 F / 73.14 S. Bump vitest.config.ts thresholds from L72/F64/B58/S70 to L74/F66/B60/S72 — ~1pt under measured so the gate absorbs Node 22/24 variance without flaking. C-7 target remains L85/B75/F80. Update README coverage badge (L75/B61/F68) and AUDIT.md per the CI-first post-merge metric-update policy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Phase 2.3 — Pipeline-cache unification (perf hardening sprint)
Problem
Both inference workers carried byte-identical pipeline-LRU logic and neither disposed the evicted pipeline → VRAM/RAM leak (same bug-class as the WebLLM eviction fix, AUDIT 2026-06-01 #1):
workers/inference.worker.ts:34-98workers/v2/inference.worker.ts:21-56Fix
Extract
services/ai/pipelineLruCache.ts—PipelineLruCache<T>:dispose?.(), closing the leakSelf-review (meta-review of same-day commits)
aiRetry.ts— sound; property-based invariant tests scheduled (Phase 3)useLoraView.ts:70-72— selector-recreation nit; fix scheduled (Phase 3)localWorkerBus.recordResult) → no new hop addedVerification
pipelineLruCache.test.ts(9) +inferenceWorker.test.ts(7) ✅ — existing worker tests unchanged ⇒ behavior-preservingsmoke:prod(worker-bundle rolldown guard) + E2E via CI🤖 Generated with Claude Code
CodeAnt-AI Description
Unify inference pipeline caching and stop evicted models from staying in memory
What Changed
Impact
✅ Lower GPU and RAM usage during model switching✅ Fewer duplicate model loads under concurrent requests✅ Fewer memory leaks after repeated inference💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.