feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(runtime): complete Phase 3A file-transaction recovery - #1346

Closed
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a
Closed

feat(runtime): complete Phase 3A file-transaction recovery#1346
zhiiw wants to merge 45 commits into
apache:mainfrom
zhiiw:codex/runtime-resume-phase3a

Conversation

@zhiiw

@zhiiwzhiiw commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR completes the Phase 3A operation plane for controlled recovery and now includes the file-transaction evidence required to make production Write and Edit recovery causal rather than heuristic.

  • keeps RuntimeEvent as the canonical recovery truth and SQLite tool rows as rebuildable projections
  • resolves recoverable operations through one deterministic, fail-closed RecoveryResolver
  • atomically commits reconcile observations, synthesized responses, decisions, and projections
  • wires the same recovery contract registry through planning, SessionManager recovery, and execution revalidation
  • adds a native per-file transaction checkpoint for Write and Edit, independent of Git
  • resumes across a real SQLite close/reopen boundary after an injected T2 failure
  • trims only interrupted model text/thinking suffixes from provider replay while preserving immutable RuntimeEvents and their high-water

Review blockers resolved

1. Production Write / Edit did not declare recoveryMode: 'reconcile'

The production-shaped path now marks the canonical builtin Write and Edit definitions as reconcile whenever a file checkpoint carrier is installed. ToolRuntime therefore persists the same recovery mode that the registered contracts expect instead of falling back to never_auto_retry.

This is covered by a real builtin-definition → ToolRuntime → SQLite dispatch test for both tools. The test reads the durable T1 facts back from SQLite, so it cannot pass by constructing reconcile mode manually.

2. countOccurrences could not prove that this particular Edit caused the current file state

Occurrence-based applied/not-applied inference has been replaced by a fine-grained file transaction protocol:

  1. Under the normal per-file lock, before T1, prepare a durable checkpoint containing the canonical target, exact before identity (missing or content hash), expected-after hash, and transform version.
  2. Generate the expected after-image with the same production Write/Edit transform used by normal execution; recovery does not carry a second simplified Edit implementation.
  3. Write a deterministic same-directory temp file, validate its hash, fsync it, revalidate the target against the before identity, atomically replace it, then fsync the parent directory.
  4. On recovery, use only the causal three-state decision:
    • current hash equals expected-after → mutation completed; synthesize the missing response
    • current hash equals before → deterministically redo the prepared transaction, then synthesize the response
    • anything else → external drift/conflict; park without overwriting

This protocol does not require Git. Git-backed workspace snapshots remain a separate Phase 3B/4A concern.

Crash and restart coverage

The file transaction suite covers interruption around checkpoint durability, temp write, temp fsync, before/after replace, parent fsync, external drift, oversized files, temp cleanup, and T2 failure.

A production-shaped restart test executes a real builtin Write, persists checkpoint + T1, atomically replaces the file, injects failure before T2, closes and reopens SQLite, invokes the real SessionManager recovery path, commits the atomic recovery bundle, synthesizes the missing tool response, and completes a continuation through FakeBackend.

Additional recovery fixes

  • A completed tool followed by interrupted model text or signed thinking no longer parks with provider_resume_boundary_unsupported. Only the provider replay view rolls back to the latest durable user/tool boundary; the ledger and high-water remain unchanged.
  • Recovery results expose authoritative recoveredOperations summaries without requiring hosts to reconstruct them from diagnostics.
  • Preflight-invalid Edit operations remain definitely-not-dispatched because checkpoint preparation happens before durable dispatch.

Safety boundary

  • Bash, remote APIs, manual-only tools, legacy operations without checkpoints, and unprovable side effects still park.
  • Planning and execution use the same recovery contract registry.
  • Drift never triggers overwrite or blind retry.
  • Reconciliation remains serial so canonical RuntimeEvent ordering is deterministic.
  • Native support stops at per-file transactions. Workspace-wide drift detection, Git snapshots, isolated restore, and rebaseline remain Phase 3B/4A.

Validation

Latest local validation on the split Phase 3A branch:

  • npm run typecheck — all workspaces passed
  • 81 targeted runtime tests — 80 passed, 0 failed, 1 platform-dependent symlink case skipped
  • targeted coverage includes production builtin dispatch mode, file crash matrix, SQLite T2 restart, SessionManager recovery, provider suffix replay, and atomic durable boundaries

GitHub CI on the latest upstream-synchronized head passes typecheck, test, and e2e.

@zhiiw
zhiiw marked this pull request as ready for review July 22, 2026 13:39
@likun666661

Copy link
Copy Markdown
Member

这里有一个我认为需要在合并前调整的 blocker:我不接受当前基于目标文件内容中 old_string/new_string 出现次数来判断 Edit 是否已经执行的方案。

当前判断:

oldMatches===0&&(newString.length===0||newMatches===1)

只能描述文件现在长什么样,不能证明当前状态是由这次 Edit 产生的。例如文件原本就包含一次 NEW、但不包含 OLD,操作完成 durable dispatch 后在真正执行前崩溃,恢复逻辑仍会判定 applied 并合成成功响应。删除操作(new_string === "")和 old_string === new_string 也有同类问题;后者与真实 Edit 实现会直接报错的语义甚至不一致。

这不是补几个字符串边界条件可以修好的问题,而是 recovery evidence 本身不足。以当前 live file 内容反推操作因果关系,复杂度和误判面都会持续扩大。

我建议把本地 Write/Edit 收敛为一个细粒度的、可恢复的文件事务:

  1. 在与正常文件操作相同的 per-file lock 内、durable dispatch 之前创建文件级 checkpoint,至少记录 canonical path、before state(missing 或 content hash/blob ref)、expected-after hash,以及 transform/algorithm version。
  2. 使用正常执行的同一份 Write/Edit transform,从 checkpoint 确定性生成 expected content;不要在 recovery 模块重新实现一套简化 Edit 逻辑。
  3. 在目标文件同目录创建临时文件,写入 expected content,校验 hash 并 fsync。
  4. replace 前再次确认目标仍匹配 checkpoint 的 before state;匹配后使用 atomic rename/replace,并 fsync parent directory。
  5. 恢复时只做三态判断:
    • current hash == expected-after hash:replace 已完成,只补 function_response
    • current hash == before hash:从 checkpoint 重新生成临时文件,校验后 atomic replace,再补结果;
    • 其他:发生 drift/conflict,必须 park,不能覆盖。
  6. SQLite 中现有的 reconcile_result + synthesized response + recovery_decision 原子 bundle 可以继续保留。

也就是说,建议删除基于 countOccurrences 的 applied/not-applied 推断;old_string/new_string 只用于从 durable checkpoint 确定性生成 expected content,不再作为“这次操作是否发生过”的证据。

建议补齐 checkpoint durable 前后、temp write、fsync、replace 前后、replace 后但 T2 前再次崩溃,以及外部 drift 等 crash matrix 测试后再合并。

@Astro-Han

Astro-Han commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Using RuntimeEvents as the recovery source and committing the reconcile result, synthesized response, and recovery decision in one SQLite bundle makes sense. The file observer is bounded by workspace, file type, size, and UTF-8 checks, and planning and execution share the same recovery contract registry. I would keep those parts.

Two blockers remain.

First, the production Write and Edit definitions do not declare recoveryMode: 'reconcile'. tool-runtime therefore records the default never_auto_retry, and the new contracts reject those durable facts as recovery_contract_mismatch. The focused tests construct reconcile mode manually, so they never cover the real builtin-to-dispatch-to-resume path. This needs to be wired on the canonical tool definitions and covered by one production-shaped integration test.

Second, I agree with the existing comment that countOccurrences cannot establish whether this particular file operation ran. A file can already match the proposed new content before dispatch, and the old_string === new_string branch disagrees with the real Edit implementation, which rejects that input. Reusing the normal Edit transform would fix semantic drift, but it would not fix the causal ambiguity. Recovery needs durable before-state and expected-after evidence, or an equivalent file transaction protocol, before it can safely synthesize success.

The atomic SQLite bundle can stay. The missing production wiring and insufficient file evidence should be resolved before merge.

zhiiw added 5 commits July 23, 2026 14:41
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
# Conflicts:
#	packages/runtime/src/__tests__/session-manager.test.ts
#	packages/runtime/src/runtime-kernel.ts
#	packages/runtime/src/runtime-resume.ts
# Conflicts:
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
@likun666661

Copy link
Copy Markdown
Member

Thanks for the update. I re-reviewed the new head (82ad92ac). The original concern is substantially addressed: the occurrence-count heuristic is gone, and Write/Edit now use a durable per-file checkpoint prepared before T1, the production transform to derive the expected after-image, same-directory temp-file write/hash/fsync/atomic replace, and a three-way recovery decision (after -> finalize, before -> redo, other -> park). This is the right direction and matches the checkpoint-based approach we discussed.

I still see three blocking correctness/security issues:

  1. Prepared Write/Edit bypass the filesystem worker.

    When a checkpoint carrier is installed, ToolRuntime executes durablePreparation.execute() instead of tool.impl() (tool-runtime.ts#L1385-L1387). The prepared Write/Edit closures call fileMutationCheckpointCarrier.apply() directly in the host process (builtin-tools.ts#L368-L415), while the normal implementation delegates to filesystemWorker.execute() with the permission profile, one-call grant, sandbox mode, and abort signal. Desktop production wiring installs both the carrier and filesystem worker (tool-assembly.ts#L208-L220), so enabling the durable path silently removes the existing sandboxed execution boundary for workspace writes.

    The checkpoint/atomic-replace protocol should either live inside the filesystem worker, or prepared execution should delegate the operation through that worker. Please also add an integration test with both fileMutationCheckpointCarrier and filesystemWorker installed that proves the worker is still used.

  2. Atomic replace changes existing file permissions.

    prepare() currently collapses every existing mode to either 0644 or 0755 (local-file-checkpoint-carrier.ts#L84-L100), then applies that normalized mode to the replacement inode (local-file-checkpoint-carrier.ts#L179-L204). For example, editing a 0600 file makes it 0644, and 0700 becomes 0755. The former is a concrete security regression compared with overwriting the existing file.

    Please preserve the exact permission bits (for example, info.mode & 0o7777) and add coverage for at least 0600, 0700, and a group-readable/writable mode. It would also be useful to state explicitly how ACLs/xattrs/hard-link semantics are handled by this replace strategy.

  3. A post-replace exception can be durably reported as failure even though the file changed.

    apply() renames the temp file onto the target before parent-directory fsync (local-file-checkpoint-carrier.ts#L198-L208). If rename succeeds but directory fsync throws, the generic ToolRuntime catch path calls writeSyntheticToolResult(), which commits an error T2 (tool-runtime.ts#L537-L565, tool-runtime.ts#L1499-L1585). The durable operation is then settled as failed, so restart reconciliation will not observe the after-hash and synthesize success, even though the side effect happened.

    Errors after T1 need effect-aware handling: definitely-before/no-effect failures may commit an error T2, while ambiguous or post-replace failures should fail-stop and remain unsettled so recovery can decide from the checkpoint. Please cover after_replace and parent-fsync failures through the real ToolRuntime, not only by invoking the carrier directly.

One additional hardening issue: inspect() ignores maxFileBytes and reads the entire current file into memory (local-file-checkpoint-carrier.ts#L132-L153). After T1, an external writer can replace the target with a very large file and make recovery perform an unbounded allocation (and Edit may read it twice). Please stat before reading or hash via a bounded stream.

So: the revised architecture resolves my previous objection, but I would still block approval until the three issues above are addressed.

Comment threadpackages/runtime/src/recovery-resolver.ts
@Astro-Han

Astro-Han commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Three additional P2 findings on the current head, separate from the file-transaction issues already covered above:

  1. A chained continuation can reintroduce a provider suffix that was previously omitted. The immediate source is trimmed through buildContinuationReplayRuntimeEvents, but continuation ancestors are rebuilt with buildResumeReplayRuntimeEvents (runtime-resume.ts#L473, #L730). If A’s interrupted model suffix is omitted when creating B, then B later continues into C, A’s old suffix can appear in C’s prior runtime context even though B never saw it. Please apply the same continuation replay policy to ancestor segments and cover a two-generation continuation with an interrupted text/thinking suffix.

  2. Valid absolute or otherwise equivalent paths can fail checkpoint matching. The checkpoint stores a canonical relative path, while recovery compares it against the raw operation argument through normalizeRelativePath (file-tool-recovery.ts#L170-L179). A cwd-local absolute path therefore produces a valid checkpoint but later parks as prepared_file_checkpoint_invalid. Both sides should compare the same canonical target identity.

  3. Projection rebuild discards canonical event order for the journal tail. The query reads RuntimeEvents in event_seq order, but does not retain event_seq; reconcile/outcome/decision events are then re-sorted by millisecond timestamp and random event ID (sqlite-runtime-store.ts#L496-L505, #L642-L646). Events committed in the same millisecond can therefore rebuild in a different order. Please carry event_seq through reconstruction and add a same-timestamp test.

I’m grouping these in one comment because they do not duplicate the existing blockers and do not need separate inline threads, but each should be covered by a focused regression test in the next revision.

zhiiw added 4 commits July 23, 2026 17:27
…e-phase3a
# Conflicts:
#	apps/desktop/src/main/app-lifecycle.ts
#	packages/cli/src/runtime-bootstrap.ts
#	packages/runtime/src/session-manager.ts
#	packages/storage/src/__tests__/sqlite-session-metadata-store.test.ts
#	packages/storage/src/agent-run-store.ts
@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Addressed in db263cec (current head 3f620f79 after syncing with main).

  1. Chained continuation replay: ancestor segments now use the same continuation replay materialization policy as the immediate source, so a text/thinking suffix omitted from A → B cannot reappear in B → C. A two-generation regression covers the interrupted model suffix.
  2. Canonical file identity: recovery no longer compares the checkpoint path with a normalized raw argument. The carrier resolves the operation argument through the same workspace-bounded canonical identity used during preparation, then compares it with fact.canonicalPath. A cwd-local absolute Write path is covered.
  3. Journal rebuild order: rebuild now retains each RuntimeEvent's event_seq and orders reconcile/outcome/decision tail facts by that canonical sequence only. A same-millisecond regression proves random event IDs cannot reorder the journal.
  4. Sticky corruption (inline blocker): all later resolver passes preserve corruption; no dispatch or recovery fact can restore automatic eligibility. The combined sequence regression covers the overwrite case.

The revision also makes continuation admission atomic per source boundary across store/process instances, keeps the interactive workspace writer exclusive, and prevents cancelled source runs from entering automatic file reconciliation.

Validation after merging the latest main: all-workspace typecheck passed, plus the focused runtime, storage, SQLite coexistence, Desktop, and CLI regression suites.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the review on head 82ad92ac: all four items are addressed in the current head 1f1ae1f9 (primarily 1eeb9f87 and 23d426c0).

  1. Filesystem-worker ownership: host assembly now selects a single prepared-mutation execution owner. When a filesystem worker is installed, the effective carrier is WorkerBackedFileCheckpointCarrier: prepare/inspect stay in the host control plane, while every apply/redo is sent as prepared_file_apply through the worker with cwd, permission mode/profile, one-call grant, and abort signal. There is deliberately no host-local apply fallback after worker ownership is selected. Production-shaped coverage proves a real prepared Write crosses T1/T2 through the worker, the host-local apply is not called, recovery redo remains on the worker, and an ambiguous worker crash leaves the operation unsettled.

  2. Metadata correctness: POSIX mode is preserved exactly as mode & 0o7777; tests cover 0600, 0700, and 0660. Recovery identity includes mode on POSIX, while Windows deliberately uses content identity rather than Node's simulated POSIX mode. Hard-linked targets are rejected before mutation so atomic replacement cannot silently break inode sharing. The architecture contract now states the boundary explicitly: this strategy preserves POSIX permission bits, but does not claim to preserve ACLs, xattrs, owner, or other inode metadata. Windows split-replace additionally keeps a deterministic before-image backup until the replacement is durable.

  3. Effect-aware failures after T1: once replace is attempted, carrier failures are converted to DurableToolExecutionUnsettledError (effect_may_have_started / effect_applied_not_durable). ToolRuntime fail-stops and emits tool_unsettled without committing an error response/T2, so restart reconciliation remains authoritative. Real ToolRuntime + SQLite tests cover after_replace, after_parent_fsync, an actual parent-directory fsync failure, and a worker crash; they assert that the file may contain the after-image while the journal remains prepared and no function response is committed.

  4. Bounded observation: prepare, inspect, recovery content reads, and Windows backup reads all go through readBoundedFile. It checks the opened file size before allocation, reads in bounded chunks, and fails with PreparedFileCheckpointLimitError if the file exceeds/grows beyond maxFileBytes. Coverage includes oversized before and current images.

Focused verification on the merged head: 36 file-transaction/ToolRuntime tests passed (31 executed on Windows, 5 POSIX-only mode/hard-link/symlink cases correctly skipped here and retained for Linux/macOS CI), with no failures.

@likun666661

Copy link
Copy Markdown
Member

Thanks for the follow-up. I re-reviewed the current head (8bfc5ed3). The four items from my previous review are now materially addressed in the implementation: prepared apply/redo is worker-owned when the worker is available, POSIX modes are preserved and hard links are rejected, post-replace failures remain unsettled, and file observation is bounded.

I found two new blocking issues and one Windows cleanup issue in the revised boundary.

  1. Blocking: the prepared checkpoint is not bound to the source run's trusted workspace.

    preparedFactMatchesOperation() resolves the operation path using fact.workspaceRoot and then compares the result with fact.canonicalPath (file-tool-recovery.ts#L171-L189). This is self-authenticating: operation.workspaceCwd is available but is not used.

    For example, an operation created in /workspace/A with path: \"notes.txt\" can be paired with an otherwise valid/corrupted fact declaring workspaceRoot: \"/workspace/B\" and canonicalPath: \"/workspace/B/notes.txt\". The args hash still matches because cwd is not part of it, and the current identity check succeeds because it resolves the argument against the fact's own root.

    Recovery then compounds the problem by defaulting the worker cwd to fact.workspaceRoot (worker-backed-file-checkpoint-carrier.ts#L87-L99), causing the worker permission profile to treat that fact-provided directory as the workspace root.

    Please canonicalize and trust operation.workspaceCwd, require it to equal fact.workspaceRoot, and resolve the operation argument against that trusted root. A regression should mutate only the fact to point at a second workspace and assert that recovery parks without inspecting or applying it.

  2. Blocking: readBoundedFile() follows symlinks and has removed the previous non-symlink guarantee.

    The old inspect() used lstat() and explicitly rejected symbolic links. The bounded replacement opens the path normally and then calls file.stat() (local-file-checkpoint-carrier.ts#L467-L507). A normal open(path, \"r\") follows the symlink, and file.stat() describes the referent, so inspect() no longer detects a target replaced by a symlink after T1 (local-file-checkpoint-carrier.ts#L160-L171).

    This allows checkpoint preparation/observation and Edit recovery reads to follow a workspace path outside the workspace. It can also synthesize success if the symlink referent happens to match the expected-after identity, even though the prepared target is no longer a regular non-symlink file.

    Please open with O_NOFOLLOW where supported and use an equivalent lstat/open/revalidation strategy elsewhere. Add a crash/recovery regression that prepares a regular file, replaces the target with a symlink after T1, and asserts park for both an in-workspace and out-of-workspace referent.

  3. P2: successful Windows recovery can leave the full before-image backup behind indefinitely.

    Windows creates a deterministic .maka-before-*.bak before replacement and normally removes it after parent durability (local-file-checkpoint-carrier.ts#L248-L263). apply() also removes it when its own initial state is already after (local-file-checkpoint-carrier.ts#L204-L213).

    However, the real recovery contract returns immediately and synthesizes success when its initial observation is after, without calling carrier.apply() (file-tool-recovery.ts#L99-L108). A Windows crash after replace but before backup removal therefore completes recovery while leaving a hidden file containing the entire old content. The current carrier crash test calls apply() again manually, so it does not cover this real recovery path.

    Please expose an idempotent finalize/cleanup operation and invoke it before synthesizing the recovered response, with a real recovery regression for the Windows backup case.

The latest typecheck, test, and e2e checks are green, and the earlier corruption/continuation/event-order fixes also look present. I would still block approval on items 1 and 2.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Follow-up to the latest blocking review on head 8bfc5ed: the two blockers and the Windows cleanup issue are addressed in 52b9ae9; the branch is now at dde9bb8 after merging current upstream/main and adapting the recovery tests to the new ToolRuntime settlement API.

  1. The checkpoint is now bound to the source Run's trusted workspace. SessionManager carries sourceRun.cwd and sourceRun.permissionMode into the unsettled operation. Recovery independently canonicalizes operation.workspaceCwd, requires that trusted identity to equal fact.workspaceRoot, resolves the operation argument against that trusted root, and only then compares the result with fact.canonicalPath. Validation happens before the per-file lock, inspect, apply, or finalize. Worker-backed apply/finalize now require an explicit execution context and no longer fall back to fact.workspaceRoot or an implicit permission mode. A regression mutates only the prepared fact to a second workspace and proves recovery parks with zero inspect/apply calls; a worker-backed recovery regression also proves finalize receives the source Run cwd and mode.

  2. Bounded observation no longer follows symlinks. readBoundedFile now lstat-checks the path before open, uses O_NOFOLLOW on supported non-Windows platforms, compares the pre-open path identity with the opened handle, performs the bounded read, then revalidates handle and path identity (device/inode plus size/mtime/ctime stability) before accepting the hash. The portable fallback therefore still detects replacement races where O_NOFOLLOW is unavailable. Recovery converts this conflict into a park decision. Linux/macOS regressions prepare a regular target, replace it after T1 with both an in-workspace and an out-of-workspace symlink whose referent already has the expected-after bytes, and prove recovery parks without touching the referent.

  3. Windows before-image cleanup is now part of the recovery commit protocol. PreparedFileMutationCarrier exposes an idempotent finalize operation, filesystem-worker protocol v4 carries prepared_file_finalize, and WorkerBackedFileCheckpointCarrier keeps cleanup on the same worker-owned boundary as apply/redo. Recovery must complete finalize before returning synthesize_response. Finalize first proves the target is still the after-image, removes only operation temp/backup files whose hashes match the durable fact, and fsyncs the parent directory; an ambiguous worker result leaves the operation unsettled. The real recovery regression crashes after replace and parent fsync but before backup removal, confirms the full before-image backup exists, invokes the actual Write recovery contract, and confirms synthesis occurs only after the backup is removed.

I also moved the before_replace failpoint ahead of the final state observation and added a drift regression, so an external mutation injected at that boundary parks instead of being overwritten.

Verification on the upstream-synced head:

  • npm run build:test passed for every workspace;
  • npm run typecheck passed for every workspace;
  • the focused file-transaction, SQLite boundary, durable T2 crash, resolver/replay, and full crash -> restart -> SessionManager continuation suites passed;
  • git diff --check passed before both commits.

The architecture contract now records the trusted-root rule, no-follow observation protocol, worker-owned finalize ordering, and the requirement that verified cleanup complete before a synthesized response is committed.

@zhiiw

Copy link
Copy Markdown
ContributorAuthor

Final verification update for the latest blocking review: the detailed resolution is in #1346 (comment), and the current head is 24bb5f33.

One Linux-only regression surfaced after that reply: recovery's trusted-path identity check was dereferencing the final path component, so a target replaced by a symlink parked early as prepared_file_checkpoint_invalid instead of reaching the no-follow observer and reporting prepared_file_became_symbolic_link. 24bb5f33 now defines the durable target identity as canonical trusted parent + final basename, without dereferencing that final component. Parent symlink escapes and fact/workspace mismatches are still rejected before observation; final-component symlinks are rejected by the bounded lstat/O_NOFOLLOW observation path.

Current CI is fully green on the updated head:

  • typecheck / lint / format / build / knip: pass
  • full Linux test suite, including both in-workspace and out-of-workspace post-T1 symlink regressions: pass
  • e2e + alignment audit: pass

The PR is mergeable against current upstream/main with no unresolved review threads.

@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the substantial work here. The overall direction is sound: recovery should be based on durable causal evidence, runtime events should remain the canonical record, SQLite projections should be rebuildable, and reconciliation should be serialized.

My main recommendation, however, is to convert #1346 back to Draft and replace it with a small sequence of focused PRs rather than continuing to layer fixes onto this diff.

This is not primarily about the line count. The current PR combines several independent correctness claims:

  1. the recovery fact model and its durable storage;
  2. projection rebuilding and evidence validation;
  3. continuation and replay semantics;
  4. prepared local-file mutation and crash recovery;
  5. runtime/provider error settlement;
  6. CLI interactive-owner lifecycle changes;
  7. prototype Git recovery and restricted-verification surfaces.

Each of these has a different authority boundary, failure model, and rollback unit. Reviewing them together means proving all of their interactions at once. It also means that a regression in one area could require reverting unrelated, otherwise-correct work.

I would keep this PR as the umbrella/design record, mark it Draft, and land the production work through sequential PRs based on the latest main:

  1. Canonical recovery persistence

    Introduce the minimum recovery fact model, one atomic bundle writer, strict evidence identity validation, and rebuildable SQLite projections.

    A completed decision must cite a matching durable outcome. Split fact writers and generic append paths must not be alternative authorities for reserved recovery facts.

  2. Continuation correctness

    Make immutable runtime events the sole source of continuation cursors and replay admission. Keep mutable partial snapshots limited to UI/diagnostic use. Include typed reference rewriting when copying conversations.

  3. Prepared local-file recovery

    Add Write/Edit checkpointing only after the first two contracts are stable. This slice should prove the complete invariant:

    If durable T1 evidence selects reconciliation, a valid checkpoint must exist, and recovery must never silently overwrite external drift.

    It should also preserve the existing file-operation result, permission, path, and metadata semantics.

  4. CLI owner/lifecycle changes

    Keep these separate unless they are strictly required by the file-recovery slice. They have their own resource-ownership and shutdown invariants.

I would not carry the Phase 4 Git carrier, restricted verifier, or speculative retry/reattach modes into these initial PRs until a production consumer requires them.

The slices should be divided by independently testable invariants, not mechanically by file or commit. Prefer flat PRs from the latest main; stack them only where a child genuinely cannot build or be reviewed without its parent.

Findings

P1 — The final drift check and file replacement are not atomic

In packages/runtime/src/local-file-checkpoint-carrier.ts, the implementation inspects the target around line 258 and later replaces it with rename around line 271.

Another process can modify the target after the final inspection but before the rename. The recovery operation then silently overwrites that newer external content and reports success.

I reproduced this by inserting an external write after the last inspection and before the rename; the external content was replaced by the agent’s prepared content.

This is the one finding I would keep at P1 because it can cause irreversible loss of a concurrent external update. Another stat or hash immediately before rename would only make the race window smaller; it would not close it.

The commit path needs a real compare-and-swap-style guarantee, such as an exchange/displaced-inode verification strategy that can restore or preserve the competing version. If the platform cannot provide that guarantee, automatic redo should stop safely instead of overwriting the target.

P2 — A completed decision can exist without a matching durable outcome

The public split writers in tool-recovery-fact-writer.ts, together with generic runtime-event append paths in sqlite-runtime-store.ts, can persist recovery facts independently of the atomic bundle.

The resolver then treats a schema-valid completed decision as authoritative without requiring it to cite a matching tool outcome.

This is not currently demonstrated on the normal atomic production path, so I would classify it as P2 rather than P1. It is still a contract hole: a crash, importer, migration, or external caller using the exposed writers can create a state that is interpreted as safely completed even though the provider-facing outcome is missing.

There should be one authority for writing the reserved recovery bundle. Storage, rebuild, and resolver validation should reject mismatched execution identity and require:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — Preparation-time business errors bypass normal tool settlement

Prepared Write/Edit performs validation before the runtime enters its normal tool-error handling path.

Expected business errors—such as a missing Edit target, a missing or non-unique old_string, or a size-limit failure—can therefore reject provider execution instead of producing the same structured error tool result as the existing implementation.

These errors mean “the tool was definitely not dispatched”; they are not effect-uncertain runtime failures. They should settle through the normal provider-facing tool-error path. Only durable commit failures or genuinely uncertain effects should reject execution into recovery handling.

P2 — macOS path aliases can silently disable checkpointing

The checkpoint carrier compares lexical and canonical paths in a way that rejects valid aliases such as /var/... resolving to /private/var/....

supports() swallows that failure and returns false, after which the builtin can fall back to the old direct mutation path while the durable recovery contract still says reconciliation is required.

That breaks the central invariant: durable T1 evidence can request reconciliation even though no checkpoint was created.

The prepared operation also returns the canonicalized path, which changes the existing result contract for callers that supplied an alias.

Path identity should be resolved once through a single authority. An unsupported prepared target must fail before T1 is persisted; it must not silently fall back after reconciliation has already been selected.

P2 — Atomic replacement changes existing file semantics and drops metadata

The implementation creates a temporary file, applies a mode, and renames it over the target. That preserves content atomicity, but it does not preserve all existing file semantics.

On macOS, I verified that extended attributes disappear after a prepared Write. ACLs, ownership, group, Finder/quarantine metadata, and platform-specific streams may also be replaced with those of the temporary file. Read-only-target and parent-directory permission behavior can differ from the previous fs.writeFile implementation as well.

I would classify this as P2 because it is deterministic but limited to files carrying those attributes or permission arrangements. It nevertheless needs resolution before the prepared-file slice merges: either preserve the required metadata and semantics, or reject ineligible targets before durable preparation.

P2 — Continuation cursors include mutable partial snapshots

session-manager.ts obtains runtime events through readRuntimeEvents, while the SQLite implementation merges mutable partial snapshots into that result. The runtime then uses the resulting length as a high-water mark and continuation cursor.

A partial streaming snapshot is presentation state, not an immutable ledger position. If it is counted as durable history, later continuation or replay can skip, duplicate, or misclassify events.

Continuation identity should come exclusively from readImmutableRuntimeEvents. Partial snapshots should remain available only for rendering or diagnostics.

P2 — Recovery may redo the effect successfully but still fail to converge

In file-tool-recovery.ts, the recovery worker can apply the prepared mutation successfully and then lose or throw before returning its response.

A following inspection may already prove that the target matches the expected after-state, but the current path can still rethrow instead of finalizing and synthesizing the outcome immediately.

Once durable inspection proves the expected after-state, recovery should converge to completed regardless of whether the worker response survived.

P2 — Conversation copying does not rewrite recovery evidence references

Conversation copying rewrites outer event IDs but shallow-copies recovery fact payloads. Embedded evidenceEventIds and related operation identity remain pointed at the source conversation.

The copied conversation can therefore deterministically fail evidence validation or appear corrupted even though its outer events were copied successfully.

Copying needs an old-to-new event-ID map and typed rewriting of recovery facts. If recovery-bearing conversations are intentionally non-copyable, that restriction should instead be enforced explicitly.

Lower-priority follow-ups

I would treat the following as P3 or slice-local hardening rather than merge-level P1/P2 findings:

  • enforce causal event ordering for damaged or imported logs;
  • accept the legal POSIX mode 0000;
  • generate auxiliary temporary names from a fixed operation hash rather than the full basename, so valid near-NAME_MAX filenames do not fail with ENAMETOOLONG.

Validation

I built the affected core, storage, and runtime packages and ran both targeted and full runtime tests.

The targeted recovery set produced 115 passes and 1 failure. The full runtime suite produced 2,558 passes, 2 failures, and 7 skips.

The failures exposed:

  • a valid cwd-local absolute Write target being reported as escaping the workspace;
  • /var/... versus /private/var/... path canonicalization changing the expected result contract.

I also reproduced the concurrent-overwrite race, loss of macOS extended attributes, and ENAMETOOLONG for a legal long target basename.

Conclusion

The underlying architecture contains several good ideas, but I do not think the current combined diff is yet the simplest reviewable or safely reversible unit.

Converting #1346 to Draft and replacing it with invariant-focused PRs would make approval more meaningful, testing more targeted, and rollback much safer. It would also let us remove prototype and speculative surfaces instead of stabilizing APIs before they have a production consumer.

中文对照

感谢为这个 PR 投入的大量工作。整体方向是对的:恢复判断应该建立在持久化的因果证据上;Runtime Event 应该是事实的权威来源;SQLite 中的投影应当可以从事件重建;同一恢复过程中的判断与写入也应该串行完成。

不过,我最主要的建议是:把 #1346 转回 Draft,不要继续在这个大 PR 上叠加修复,而是将生产改动重新整理成一组边界清楚、可以独立审查和回滚的 PR。

原因并不只是改动行数太多。当前 PR 同时要求审查者证明多套彼此独立的正确性:

  1. recovery fact 的数据模型和持久化规则;
  2. SQLite 投影重建与证据校验;
  3. continuation 和 replay 的游标语义;
  4. 本地文件预备写入、提交和崩溃恢复;
  5. runtime 与 provider 之间的错误结算;
  6. CLI interactive owner 的生命周期;
  7. 尚未进入生产路径的 Git recovery 和 restricted verification 原型。

这些部分的事实权威、失败方式和回滚边界并不相同。把它们放在一次审批里,意味着审查者必须同时证明所有交互都是安全的。以后只要其中一个部分出现回归,也可能被迫回滚其他原本正确的能力。

我建议保留当前 PR 作为总设计和讨论记录,将它转成 Draft,然后从最新 main 开始,按下面的顺序逐步落地。

1. Recovery 持久化的唯一权威

第一批只引入最小的 recovery fact 模型、唯一的原子 bundle writer、严格的执行身份校验,以及可重建的 SQLite 投影。

completed 必须引用一条身份匹配、已经持久化的 outcome。拆开的 fact writer 和通用事件 append 接口不能成为写入保留 recovery fact 的第二套权威路径。

2. Continuation 正确性

不可变的 Runtime Event 应该是 continuation cursor 和 replay admission 的唯一来源。仍在变化的 partial snapshot 只能用于界面展示和诊断,不能充当账本位置。

如果复制 conversation,还需要同步重写 recovery fact 内部的事件引用和执行身份。

3. 本地文件的预备写入与恢复

前两项稳定以后,再接入 Write/Edit 的 checkpoint 和恢复能力。

这一批 PR 需要完整证明下面这条不变量:

一旦持久化的 T1 证据选择了 reconcile,就必须存在有效的 checkpoint;恢复过程也绝不能静默覆盖外部产生的新内容。

它还必须保持现有文件工具的返回结果、路径、权限和元数据语义。

4. CLI owner 和资源生命周期

除非文件恢复严格依赖这部分,否则 CLI interactive owner 的调整应当单独提交。资源所有权、初始化失败和退出清理本身就是另一套需要独立证明的不变量。

在真正出现生产消费者以前,我不建议把 Phase 4 Git carrier、restricted verifier 或预设的 retry/reattach 模式带入前面这些 PR。

拆分边界应该依据“能否独立表达和验证一个完整不变量”,而不是机械地按文件或现有 commit 切割。默认从最新 main 建立平铺 PR;只有后续 PR 确实无法独立构建、测试或审查时才使用堆叠 PR。

Findings

P1 — 最后的漂移检查与文件替换不是一个原子操作

packages/runtime/src/local-file-checkpoint-carrier.ts 中,实现先在大约第 258 行检查目标文件,然后在大约第 271 行通过 rename 替换文件。

另一个进程可以在最后一次检查结束后、rename 执行前修改目标文件。恢复操作仍然会用预备内容覆盖这次外部写入,并报告成功。

我通过在最后一次检查和 rename 之间插入外部写入复现了这个问题。外部刚写入的内容最终被 agent 的预备内容覆盖。

这是我认为唯一应该保留为 P1 的问题,因为它可能不可逆地丢失其他进程刚刚写入的数据。即使在 rename 前再做一次 stat 或 hash,也只是缩小竞争窗口,并不能从根本上消除它。

提交过程需要真正具有 compare-and-swap 性质的保证,例如使用交换操作或检查被替换的 inode,并且在发现竞争写入时保留或恢复外部版本。如果目标平台无法提供这种保证,就应该安全停止自动 redo,而不是冒险覆盖目标文件。

P2 — 没有对应 outcome,也可能写出 completed decision

tool-recovery-fact-writer.ts 暴露了拆分的 fact writer;sqlite-runtime-store.ts 中的通用 Runtime Event 写入路径也可以绕过原子 bundle,分别持久化 recovery fact。

resolver 随后会把格式合法的 completed decision 当成权威结论,却不强制它引用身份匹配的 tool outcome。

目前还没有证据表明正常的原子生产路径一定会生成这种状态,因此我会把它定为 P2,而不是 P1。但这仍然是一个真实的契约缺口:崩溃、导入器、迁移程序或使用公开 writer 的外部调用者,都可能生成“已经完成,但 provider outcome 不存在”的记录。

保留的 recovery fact 应当只有一个写入权威。Storage、投影重建和 resolver 都应该拒绝执行身份不一致的证据,并强制要求以下因果顺序:

call < preparation < dispatch < outcome/reconciliation < completed decision

P2 — 准备阶段的普通业务错误绕过了标准工具结算

预备 Write/Edit 会在 runtime 进入正常工具错误处理以前执行一部分校验。

因此,Edit 目标不存在、old_string 没找到或不唯一、文件超过大小限制等正常业务错误,可能直接让 provider execution reject,而不是像现有工具实现一样返回结构化的 tool error result。

这些错误表达的是“工具明确没有被执行”,并不属于“副作用是否发生已经无法确定”的恢复故障。它们应该走正常的 provider-facing tool error 结算流程。

只有持久化提交失败,或者副作用状态确实无法确定时,才应该中断 execution 并进入 recovery。

P2 — macOS 路径别名可能静默绕过 checkpoint

checkpoint carrier 对词法路径和 canonical path 的比较,会把 /var/... 实际解析为 /private/var/... 这样的合法 macOS 路径别名判为不支持。

supports() 吞掉异常并返回 false 后,builtin 可能退回原来的直接文件修改路径。但持久化的 recovery contract 仍然记录为需要 reconcile。

这破坏了最核心的不变量:T1 已经持久化并要求 reconcile,但真正恢复时却没有可用的 checkpoint。

另外,预备操作会返回 canonicalized path,这也改变了调用者传入路径别名时的原有返回值契约。

路径身份应该由一个统一的权威解析。如果目标不能使用预备写入,就必须在持久化 T1 之前失败;不能先选择 reconcile,再静默退回没有 checkpoint 的旧路径。

P2 — 原子替换改变了原有文件语义,并丢失元数据

当前实现会创建临时文件、设置 mode,然后通过 rename 覆盖目标文件。这可以保证文件内容不会只写入一半,但不能自动保持原文件的全部语义。

我在 macOS 上验证过:经过 prepared Write 后,原文件的 extended attributes 会消失。ACL、owner、group、Finder 或 quarantine 元数据,以及其他平台特有的附加数据,也可能被临时文件的属性替代。

只读目标文件、可写父目录等权限组合的行为,也可能与原来的 fs.writeFile 不同。

这个问题是确定发生的,但只影响带有相关元数据或特殊权限配置的文件,因此我倾向于定为 P2。它仍然需要在 prepared-file PR 合并前解决:要么保留必要的元数据和权限语义,要么在持久化 preparation 以前明确拒绝不适合原子替换的目标。

P2 — Continuation cursor 混入了仍会变化的 partial snapshot

session-manager.ts 通过 readRuntimeEvents 读取事件,而 SQLite 的实现会把 mutable partial snapshot 合并到返回结果里。runtime 随后又使用这个结果的长度作为 high-water mark 和 continuation cursor。

正在流式生成的 partial snapshot 属于展示状态,不是不可变账本中的位置。如果把它计入持久历史,后续 continuation 或 replay 就可能跳过事件、重复处理事件,或者错误判断事件所属的 continuation。

Continuation identity 应该只来自 readImmutableRuntimeEvents。Partial snapshot 可以继续用于渲染和诊断,但不应该进入 durable cursor 的计算。

P2 — Redo 已经成功,但响应丢失时不能立即收敛

file-tool-recovery.ts 中,recovery worker 可能已经成功应用预备文件修改,但在返回响应以前发生异常或丢失响应。

后续检查此时可能已经能够证明目标文件等于预期的 after-state,但当前逻辑仍可能继续抛出异常,而不是立即完成 recovery 并合成对应 outcome。

一旦持久化检查已经证明 after-state 成立,恢复过程就应该收敛到 completed,不应再依赖 worker response 是否成功返回。

P2 — 复制 conversation 时没有重写 recovery evidence 引用

复制 conversation 时,外层 Runtime Event ID 会被重写,但 recovery fact payload 只是浅复制。内部的 evidenceEventIds 和相关 operation identity 仍然指向原 conversation。

结果是:外层事件复制成功,但复制后的 conversation 会稳定地出现 evidence validation 失败,或者被判断为 recovery 数据损坏。

复制过程需要先建立旧事件 ID 到新事件 ID 的映射,然后按照 recovery fact 的类型重写内部引用。如果产品并不打算支持复制包含 recovery 状态的 conversation,也应该明确拒绝,而不是生成表面成功、内部引用失效的副本。

较低优先级的后续问题

下面这些问题更适合标为 P3,或者放到对应 slice 内处理,不需要定成当前 PR 的 P1/P2:

  • 对损坏或导入的日志严格检查 recovery 事件的因果顺序;
  • 接受合法的 POSIX mode 0000
  • 临时文件名只使用固定长度的 operation hash,不拼接完整 basename,避免合法但接近 NAME_MAX 的文件名触发 ENAMETOOLONG

验证结果

我构建了受影响的 core、storage 和 runtime package,并分别运行了 recovery 相关的定向测试和完整 runtime 测试。

定向测试结果是 115 个通过、1 个失败。完整 runtime 测试结果是 2,558 个通过、2 个失败、7 个跳过。

失败用例暴露了两个问题:

  • 合法的 cwd 内绝对 Write 路径被错误判断为逃逸 workspace;
  • /var/.../private/var/... 的 canonicalization 改变了原有返回路径契约。

除此之外,我还独立复现了:

  • 最终检查和 rename 之间的并发写入被静默覆盖;
  • macOS extended attributes 在 prepared Write 后丢失;
  • 合法的超长 basename 导致辅助临时文件触发 ENAMETOOLONG

结论

这套实现包含多项正确的架构思路,但当前合并后的大 diff 还不是最容易证明正确、最容易独立审批、也最容易安全回滚的交付单元。

#1346 转回 Draft,并用一组围绕明确不变量组织的 PR 逐步替代,会让每次审批更有意义、测试范围更清楚、出现回归时也更容易回滚。

这样还可以先删除没有生产消费者的原型和预设抽象,避免在真实需求出现以前,就把不必要的复杂度固化成公共契约。

@Astro-Han

Astro-Han commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Thanks for the design work and the documentation here — the thinking on file-transaction recovery still has value in the issue discussion.

I am closing this because it can no longer be reconnected to current main. It is 1487 commits behind, and 34 of the 76 files it touches no longer exist on main: the whole tool-recovery module (tool-recovery-coordinator.ts, tool-recovery-contract.ts, durable-tool-execution.ts, file-tool-recovery.ts and others), apps/desktop/src/main/app-lifecycle.ts, tool-assembly.ts, and the packages/runtime/src/index.ts / packages/core/src/index.ts barrel entrypoints. Those are the result of later architectural work, not something a rebase can bridge.

If you want to continue in this direction, opening a fresh branch against current main — ideally in smaller pieces — would be the way. If the documents under docs/architecture/ still hold, a standalone docs PR would be much easier to move.

简体中文

感谢你在这条上做的设计和文档整理,file-transaction recovery 这块的思路留在 issue 讨论里仍然有价值。

这条我准备关掉,原因是它已经没法接回现在的 main 了。它比当前 main 落后 1487 个提交,改动涉及的 76 个文件里有 34 个在 main 上已经不存在,其中包括整套 tool-recovery 模块、apps/desktop/src/main/app-lifecycle.tstool-assembly.ts,以及 packages/runtime/src/index.tspackages/core/src/index.ts 这两个 barrel 入口。这些都是后续架构重构的结果,不是简单 rebase 能对上的。

如果这个方向还要推进,建议基于当前 main 重开一条,规模也可以拆小一点。文档部分如果还成立,单独提一条 docs PR 会更容易推进。

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@zhiiw@likun666661@Astro-Han