Replace Espresso with a hardened ANE-LM backend - #86
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-01a03e89-5ac5-732e-9da8-9eafeb591994 Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a03e89-5ac5-732e-9da8-9eafeb591994 Co-authored-by: Amp <amp@ampcode.com> # Conflicts: # Package.resolved # Package.swift # Sources/UI/ModelManagementView.swift
Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a03e89-5ac5-732e-9da8-9eafeb591994
Amp-Thread-ID: https://ampcode.com/threads/T-01a03e89-5ac5-732e-9da8-9eafeb591994 Co-authored-by: Amp <amp@ampcode.com>
# Conflicts: # Sources/UI/ModelManagementFamilies.swift # Sources/UI/ModelManagementSections.swift
IchenDEV
left a comment
There was a problem hiding this comment.
I found three correctness/resource issues in the new ANE path and one defect in the environment-gated fallback test. The current smoke coverage can pass on partial Qwen reasoning output, long prompts silently exceed the native KV window, and a successful generation fallback leaves both local backends resident. These should be addressed before merge.
| if modelName.lowercased().contains("qwen") { | ||
| return "<|im_start|>system\n\(system)<|im_end|>\n" | ||
| + "<|im_start|>user\n\(user)<|im_end|>\n" | ||
| + "<|im_start|>assistant\n" |
There was a problem hiding this comment.
[P1] Disable Qwen3 thinking for direct-output requests
Qwen3 treats this bare assistant\n suffix as its default thinking mode; its chat template only inserts the empty <think>\n\n</think>\n\n prefix when thinking is disabled. The formatting and edit-command paths expect direct text/JSON, use budgets as low as 256 tokens, and often pass temperature 0, which the pinned sampler implements as greedy decoding. The model can therefore spend the whole budget inside <think> before emitting the final answer, while the current real-model test still passes on any non-empty reasoning fragment. Please use the non-thinking suffix (or an equivalent hard /no_think switch) and assert final user-facing text/valid JSON in the real-model test.
| ane_lm_generate( | ||
| model.runtime, | ||
| tokens.baseAddress, | ||
| tokens.count, |
There was a problem hiding this comment.
[P2] Reject prompts that overflow the 2,048-token ANE cache
The pinned Qwen3 runtime has a fixed 2,048-entry KV cache and overwrites cache.start once it fills, while ane_lm_generate accepts every prompt length. Passing tokens.count here therefore makes prompts over 2,048 tokens silently forget their beginning before the first generated token. Screen context, memory, or longer transcripts can cross this even though the selected model advertises a much larger context. Guard the prompt length (ideally reserving output headroom) and fall back to MLX or truncate explicitly; add a >2,048-token regression test.
| ) | ||
| } | ||
| ) | ||
| if result.usedMLX { |
There was a problem hiding this comment.
[P1] Unload ANE after a successful MLX fallback
If ANE loading succeeded but espressoLLM.generate throws, the actor still owns its native LoadedModel. This branch then returns a valid result after loading MLX, and the fallback policy only changes the saved backend to .mlx; it never runs the model lifecycle unload path. The process therefore keeps both full models resident until a later manual unload or exit, which can turn a recoverable ANE failure into memory pressure or an OOM. Unload espressoLLM before returning the successful MLX fallback (after preserving the failure message/outcome), and cover the generation-failure path with an isLoaded == false assertion.
| ) | ||
| if index == 0 { | ||
| baselineFootprint = currentMemoryFootprint() | ||
| options.localLLMBackend = .mlx |
There was a problem hiding this comment.
[P2] Preserve the fallback observation before switching iterations to MLX
With OPENTYPE_FALLBACK_ITERATIONS > 1, iteration 1 runs the .mlx branch, which calls clearEspressoOutcome(). Because the test consumes the tracker only after the loop, outcome is nil and XCTAssertEqual(outcome, .fallback) fails before the repeated-request memory check can validate anything. Consume/capture the fallback immediately after iteration 0, or keep a separate sawFallback flag.
Uh oh!
There was an error while loading. Please reload this page.
Summary
IchenDEV/ANE-LM@033472ec12ea796fc7ea4f8cefd7ed456f69900bSwiftPM runtimeM5 evidence
This evidence is limited to Qwen3-0.6B on the tested M5 Max / macOS 27 host. Other model sizes, Apple Silicon devices, and OS versions remain unverified. The backend uses private Apple APIs and is not suitable for Mac App Store distribution; MLX remains the stable fallback.
Verification
DEVELOPER_DIR=/Applications/Xcode-beta.app/Contents/Developer swift test— 592 XCTest tests passed, 10 environment-gated tests skipped, and the Swift Testing suite passed after merging currentmainUTTER_ANE_TEST_ITERATIONS=20— passed in 84.367 secondsbash scripts/sdlc-checks.sh— all 12 change bundles passed the strict shell gatebash scripts/ci-basic-checks.sh— passed after conflict resolution and all approval-state updatesDEVELOPER_DIR=/Applications/Xcode-beta.app/Contents/Developer bash scripts/build-app.sh— release App and DMG built; app and mounted-DMG signatures and DMG checksum passed after merging currentmainThe two active intents, specs, plans, and verification artifacts were approved by the user on 2026-08-31. Merge and release remain separate human gates.
Evidence
docs/research/2026-08-31-apple-neural-engine-m5-compatibility.mddocs/sdlc/changes/2026-08-31-ane-lm-runtime/docs/sdlc/changes/2026-08-31-settings-semantic-background/