feat(transcript): add semantic position snapshots and Runtime Host paging

[Sun-GLiang]

Summary

Lands the minimum coherent Storage + Runtime Host slice requested in the PR1 composition review: durable semantic transcript position snapshots together with their first production consumer.

This PR intentionally contains PR1 + PR2. It supersedes #4258 as the landing vehicle; #4258 should remain Draft and unmerged, then be closed as superseded only after this PR lands.

Refs #4123

• Base: apache/main at 29d02dc218078297215f6792de3afb55b76386da
• Head: 7e1e96d61458172daaba1b65d8fb03ad287406a4
• Review scope: cumulative Storage producer + Runtime Host consumer

Coherent production slice

• Storage normalizes semantic Turn/note membership and order into exact owner/shared position snapshots.
• Runtime Host exposes subscription-scoped semantic position paging and complete Turn windows through additive, strictly decoded protocol operations.
• ClientSessionSubscription consumes those operations without exposing Storage snapshot triples, leases, generations, sequences, recovery state, or hidden shared identities.
• Existing v34 transcript open remains lazy: it performs no semantic Storage work until a semantic request is made.

Storage mechanism	Production consumer/invariant
normalized membership/order metadata	dense position pages and exact per-key body probes
bounded legacy/imported/chunked recovery	one bounded Storage recovery step per Host request; persisted v34 chunk recovery resumes after reopen
persisted payload hashes and scanner state	recovery continues across owner replacement without retaining whole bodies
exact immutable snapshot key	signed Host handles/cursors remain bound to one exact projection
two retained generations	replacement is prepared while the current snapshot stays usable
opaque leases/refcounts	subscription, connection, access-revocation, and Host shutdown paths await release
capacity/backpressure	privacy-neutral typed results; no hidden eviction or fallback
monotonic generation	stale handle/cursor ABA is rejected after release, reopen, and reallocation

Protocol and bounds

• session.transcript.positions.query: acquire | page | continue | replace | release.
• session.transcript.turn_window.page: open | continue.
• Wire positions expose only dense { ordinal, key } values (turn | note | empty).
• Tokens and cursors are opaque, subscription-bound, incarnation-bound, domain-separated HMAC-SHA256 values.
• A logical window contains at most 10 positions, 256 records, and 16 MiB raw/encoded data; fragments are at most 512 KiB.
• Shared reads use the existing shared transcript sanitizer and do not reveal hidden identities or counts.

Out of scope

No Desktop/Preload/Renderer activation, continuous layout, navigation, export, compatibility-epoch switch, or legacy-path removal is included. Those remain PR3/PR4 work. The small Desktop/CLI edits here are compile/test fixture updates for the additive client interface, not production activation.

Verification

• Rebased onto the exact base above. The active guest grant conflict retained both the newer access.credential.finalize grant and the two semantic transcript grants.
• Main added schema v35 for Skill invocation persistence, so the semantic snapshot migration was moved additively to v36. Its migration assertion now follows the shared schema-version constant.
• Fresh post-rebase npm run build:test passed.
• Fresh post-rebase Storage suite: 1114 total, 1106 passed, 8 skipped, 0 failed.
• Fresh post-rebase PR-relevant Runtime Host suites: 187/187 passed.
• npm run typecheck, npm run lint, npm run format:check, ASF headers, Desktop/UI knip, and diff-check passed.
• Earlier whole-workspace validation passed Core (733/733), Runtime (3090 passed, 13 skipped), and all PR-relevant Runtime Host suites. Load-sensitive local UDS/FIFO/TUI timeouts were non-deterministic; the detached-launcher case passed 3/3 on both this branch and the exact main baseline after cooldown. GitHub CI remains the authoritative full-suite gate for this rebased head.
• Production integration covers real SQLite -> Runtime Host -> production client, owner/shared privacy, bodyless admissions, visible/hidden notes, two-slot capacity, lease refcounting, monotonic ABA rejection, and persisted v34 >4 MiB chunk recovery across Host reopen.

Benchmark evidence from both normal and full-semantic modes of npm --workspace @maka/runtime-host run benchmark:transcript:

• 10,000 alternating-visibility positions: owner 10,000 / shared 5,000; two generations contain 20,000 position rows; warm reads produced WAL delta 0 and data-version delta 0.
• 15 MiB single position: 31 fragments, wire amplification 1.335; 17 MiB single position returns position_too_large with zero fragments.
• Full 17 MiB / 64 MiB multi-position reads: wire amplification 1.383, peak retained accounting about 0.04 MiB, and retained accounting returns to zero after cleanup.

This PR should remain Draft until the rebased GitHub checks are green and a maintainer confirms the combined composition. Do not merge #4258 separately.

AI use

Select exactly one:

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex implemented and reviewed the Storage/Runtime Host changes, tests, benchmark coverage, rebase conflict resolution, and merge-preparation updates. Commits include the required Generated-by: OpenAI Codex trailer.

Checklist

• Tests cover the change and fail without it
• Lint, format, typecheck, dependency, and legal checks pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

[Astro-Han]

Thanks for putting so much work into this. Please pause here before spending more time on PR3 or PR4. I need to correct the direction first.

#4123 describes a real problem: the durable transcript is complete, but Desktop only exposes the current resident range to the Renderer. When the user jumps from the live tail to an old Turn, the range is replaced and the intermediate Turns lose their logical positions. They are still in Storage, but the conversation can look discontinuous.

I wrote #4123 before #4206 landed. At that time, the Renderer had its own Turn virtualizer, height index, spacers, scroll compensation, and observers. #4206 later removed those mechanisms because they created a second layout and scroll state machine. It moved the active transcript bound into Runtime Host, kept the Renderer projection small and stable, and left the existing chat scroll logic as the single scroll authority.

I should have updated #4123 after that change, but I didn’t. The issue still described the old virtualizer architecture, so it encouraged you to build the backend for a design that no longer exists on main. That is my mistake, not yours.

The current PR builds a much stronger system than the remaining product problem requires. It adds materialized position snapshots, two retained generations, durable leases, resumable legacy scanning, a second body paging protocol, and several new recovery and cursor state machines. Those pieces are carefully implemented, but there is still no Desktop consumer in this PR, so the user-visible problem remains unchanged.

Some of these guarantees also work against the performance goal. A new transcript watermark can require rebuilding the complete position snapshot, and every normal transcript append now synchronously maintains the new semantic index even though Desktop does not consume it yet. I don’t think it is responsible to ask you to keep polishing this design before the real Desktop composition proves that these mechanisms are necessary.

I’m going to revise #4123 around the architecture that exists after #4206. The smaller direction I want to validate is:

• Keep Runtime Host’s existing bounded transcript range and the current single scroll authority.
• Add one lightweight, incrementally maintained logical position index containing only the identity and ordering needed for layout and navigation.
• Page that index with a watermark or revision and a simple keyset cursor. A stale index can be re-anchored instead of retaining old materialized generations.
• Continue using the existing session.transcript.page path for message bodies rather than introducing another body window, fragment, cache, and cursor protocol.
• Start with one real Desktop owner path and an end-to-end regression covering old, intermediate, and live-tail navigation. That consumer should determine which backend guarantees are actually needed.
• Defer shared guest virtualization, bodyless durable positions, exact old-snapshot survival, and byte-level resumable scanning until a concrete product path requires them.

Please keep this PR as Draft for now. I don’t want you to throw away the branch yet because the tests and implementation are useful evidence, and parts of the normalized position index may still carry over. I first need to rewrite the issue and narrow the contract against current main.

If I have missed a constraint that makes this smaller design unworkable, please push back. You have spent more time inside these failure and recovery paths than I have, and I want that context before deciding what can be removed.

I used Codex to compare this PR with current main, #4123, and #4206, and independently checked the architecture and production paths discussed here.