feat(dynamic-workflow): opt-in content-addressed cross-run reuse

[modacker]

What changes

Implements opt-in content-addressed cross-run reuse for the dynamic-workflow plugin: a run can now adopt succeeded nodes from any prior run in the same workspace store when both the execution context and the call spec hash identically — not only within a repair lineage.

Design provenance: the derived-stable-identifier approach ("the key is the identity") follows our deterministic-addressing tool (sih-engine family, sih-tools/locator).

Design

• Opt-in at run level: reuseAcrossRuns: true on start/update (default false). The key is absent from the definition unless explicitly true — an always-present default key would change requestHash and break the legacy idempotent-replay path (caught by the "legacy limits … old idempotent requests still work" test; the fix follows the same absent-means-default pattern as DEFAULT_LIMITS).
• Key equality: context = hash({workspace, input, executor, fingerprints}) (same four factors as repair's contextHash, cached per run), spec = hash(spec) (prompt/schema/input included — a different schema is a different contract). Lookup joins steps × runs via json_extract, newest first, this run excluded, only succeeded agent nodes.
• Provenance: adopted steps carry reusedFrom — the original producer's lineage when present, otherwise {runId, stepId, endedAt, crossRun: true} — and emit step.reused. Reuse consumes no call budget (attempt: 0, before the budget gate).
• Schema revalidated per candidate against today's validator before adoption; a candidate that no longer validates is skipped (fresh call instead).
• Honest boundary (documented in README): cross-run reuse proves context identity and storage fidelity, not first-run semantic correctness. For critical nodes, put evidence fields in the schema and place an independent verification node downstream.

Test evidence

• New checks/cross-reuse.check.mjs: 11 tests — default-off, opt-in hit (zero new calls, provenance, event), input/executor/prompt/schema/fingerprint changes all miss, failed nodes never reused, newest-wins, pure reuse on a one-call budget, requestId idempotency across a flipped flag.
• TDD order: tests first (red 6/11 — the five positive feature cases failed on the missing API), implementation second → 11/11.
• Full plugin suite: 83/83 (72 baseline + 11 new). One regression was caught and fixed during development: an always-present default key broke the legacy idempotent replay — exactly what the legacy-compat test guards.
• Packaged MCP smoke passes; npm run build byte-reproducible; repository validator green on a clean checkout.
• macOS arm64, Node v26.7.0, based on main (00878c7).

Follow-up seam

Once #48 (integrity ledger) merges, a small follow-up can route accepted cross-run reuses through the ledgered surface so "what reuse trusts" is covered by the same tamper-evidence chain.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[hetaoBackend]

Request changes for exact current head 35223d3.

Blocking correctness/contract issues:

1. reuseAcrossRuns is implemented in engine.mjs:48,77, but is absent from the public MCP schemas in src/tools.mjs:13-14 (and the generated dist/main.mjs). Those schemas use additionalProperties:false, so a normal MCP client cannot discover or reliably send the documented feature. Add the field and description to workflow_start and workflow_update, rebuild dist, and add a packaged MCP contract test that enables it through the public tool surface.
2. Cross-run reuse does not track upstream lineage. engine.mjs:199-208 and store.mjs:52 match only the downstream node request hash and a run-level context hash; dependsOn is only the upstream ID. I reproduced this on the exact head: run 1 executed a(prompt=old) -> b(prompt=same), run 2 executed a(prompt=new) and then reused b from run 1, producing a=out:new with b=out:same and reusedFrom.crossRun=true. A changed/rerun upstream must invalidate downstream reuse, as the README promises. Persist/compare effective dependency lineage or upstream output/spec hashes, and add the R1/R2 regression test.
3. For MCode nodes without an explicit spec.model, the effective default model/provider/config comes from the CLI environment, but the cache identity contains only executor: 'mcode' and the node spec. A changed MCode default model/config/CLI can therefore reuse an old result. Require an explicit model for cross-run MCode reuse or include a resolved model/config/CLI identity in the cache key and test that a default-model change misses.
4. store.findCrossRunReuse() applies LIMIT 20 before filtering by context hash (store.mjs:52), so 20 newer same-node results from other contexts can hide an older valid candidate forever. Filter in SQL using a persisted/indexed context key or scan/paginate until a valid candidate is found.
5. Cross-run adoption of a repair-reused step preserves the old reusedFrom object (engine.mjs:205-206) while emitting a crossRun:true event (:207), so stored provenance can disagree with the event. Preserve immediate source and original producer as separate provenance fields and add a repair-to-cross-run test.

Current GitHub checks are green, but they do not cover the public MCP schema path or the reproduced stale-lineage case. Rebase/rebuild against the current main before requesting another review.

[hetaoBackend]

Review of current head 35223d31beaa19ce4aec9600fdf3f70e1d628b83: changes are still needed before merge. Current green checks do not cover these cases.

1. [P1] Changing an upstream node can reuse a stale downstream result. src/engine.mjs:199-208 matches the downstream spec and run context, but not the effective upstream execution lineage/output.

   Reproduced using the actual Engine/Store, real temporary files, and a deterministic executor:

   • Both runs begin with produced.txt = "same baseline", listed in input.files, so their approval-time fingerprints match.
   • Run 1: A writes old; B depends on A and reads the file, returning old.
   • Reset the file to the same baseline. Run 2 enables cross-run reuse and changes A's prompt to write new; B retains the same prompt/spec and dependsOn: ["a"].
   • Run 2 executes A but adopts B from Run 1: final result {a: "new", b: "old"}, while the actual file contains new. The recorded calls are A(old), B(read), A(new); B never executes again.

   Persist and compare effective dependency lineage/output identity before reuse; changes or reruns of relevant upstream nodes must invalidate downstream candidates. Add this two-run regression, including the case where fingerprints match before execution but an upstream node changes a tracked file during execution.

2. [P2] The advertised switch is absent from the public MCP contract. workflow_start and workflow_update in src/tools.mjs:13-14 omit reuseAcrossRuns and use additionalProperties: false. Validating the documented request against the advertised schema rejects that field. Add it to both public schemas/descriptions, rebuild the packaged runtime, and test discovery/use through the packaged MCP surface rather than only direct Engine calls.

3. [P2] Stored reuse provenance can disagree with the event. src/engine.mjs:205-206 preserves an existing reusedFrom object unchanged. A candidate previously reused by repair can therefore lack the documented crossRun: true, while the new step.reused event says crossRun: true and names a different immediate source. Preserve immediate source and original producer separately, and cover repair-to-cross-run adoption.

4. Reconcile the integrity dependency. This diff already includes feat(dynamic-workflow): tamper-evident integrity ledger for repair_cache and events #48's Store ledger implementation, although the description presents integrity integration as a follow-up. It also inherits the record-ownership verification problem described in my current feat(dynamic-workflow): tamper-evident integrity ledger for repair_cache and events #48 review: changing runId/id while retaining body/ledger data can still yield verified: true. Make the dependency and final diff explicit and include the corresponding fix or remove the unrelated implementation from this PR.

Validation: the new cross-reuse suite passes 11/11 locally. The failing lineage probe exercises orchestration with deterministic outputs and real temporary-file I/O; no paid model execution or model-quality claim is involved.

[modacker]

All four items addressed at exact head 8df53c9 (rebased on current main incl. #43/#44):

1. Upstream lineage (your two-run repro): each agent step now persists a lineageHash binding its own requestHash with, per succeeded dependency (in dependsOn order), that dependency's lineageHash and output hash. Cross-run lookup requires lineage equality in SQL. Your file-write variant is covered twice over: a changed upstream spec breaks lineage via requestHash, and an upstream that re-executes with a different output (the mcode-without-model escape, tracked-file divergence) breaks it via the output hash — both shapes are regression tests (checks/cross-reuse.check.mjs), including the mcode-no-model case where the upstream itself is never a candidate.
2. Public MCP contract: reuseAcrossRuns is now on workflow_start and workflow_update schemas; checks/cross-reuse-mcp.check.mjs drives the packaged MCP binary through the public tool surface (field advertised, true accepted and persisted, non-boolean rejected).
3. Provenance consistency: adoption always writes reusedFrom = immediate source with crossRun:true, plus a separate originalProducer field for the first producer — storage and the step.reused event can no longer disagree; chained repair→cross-run adoption is tested (originalProducer tracks back to run 1).
4. Integrity contamination: you were right — the previous round's store.mjs had accidentally carried feat(dynamic-workflow): tamper-evident integrity ledger for repair_cache and events #48's ledger code (our tooling copied sources from a tree that had it). store.mjs is rebuilt from clean main with only findCrossRunReuse; this PR no longer contains any ledger implementation, and the diff is addressing-only.

Also inherited from your first review and included here: mcode nodes without an explicit spec.model are never cross-run candidates; contextHash is stamped on steps and filtered in SQL (LIMIT applies after full match).

Validation at this head: 90/90 plugin checks (72 baseline + 15 cross-reuse incl. both repro shapes + MCP contract + package smoke), byte-reproducible rebuild, and the full 493-test repository gate on a local Ubuntu 24.04 box — 482 pass / 0 fail / 11 platform skips. Re-requesting review.

[modacker]

Correction: exact head is now af58a3a — two contributor-local CI helper scripts (local-ci.mjs, remote-ci.sh at repo root) were accidentally committed in the previous round via a broad git add -A; this commit removes them. Same class of slip as the ledger contamination you caught, and it prompted the same rule here: helper scripts live outside the clone, and branch commits enumerate paths explicitly. No plugin files changed in af58a3a — the addressing diff is otherwise identical to 8df53c9.

[hetaoBackend]

Re-reviewed at af58a3ac2f9136a10efe0617f9733bac86a13fa2. The previous upstream prompt/output invalidation, public MCP schema, and reuse provenance findings are fixed. The cross-reuse, packaged MCP, and repair suites pass (32/32); the previous real-file dependency reproduction now correctly returns the fresh downstream result.

One remaining issue before merging: cross-run adoption retains the source run's planId, so the dashboard duplicates/mislabels the node (src/engine.mjs:225). The adopted step spreads source without overriding planId; the repair-adoption path already assigns the current dispatch's planId.

Independent reproduction with the real Engine and workflowGraph:

1. Execute return await ctx.agent({id:'a',prompt:'a'});.
2. Start another run with the same input/spec and reuseAcrossRuns: true, only prepending a comment to the script.
3. Reuse correctly hits (one executor call total), but the current topology has plan:48 while the adopted step still has plan:13.
4. workflowGraph produces two nodes: plan:48 / not_run and live:a / succeeded, although there is only one agent node.

Please overwrite the adopted step's planId with the current dispatch value and add a regression asserting the graph contains one correctly matched successful node after source positions change.

Latest-head CI is green. This was reproduced with a deterministic executor and the actual graph-model function; it was not a browser screenshot or real-model test.

[modacker]

Fixed at exact head accae5c: cross-run adoption restamps the current dispatch's planId (matching the repair path) instead of keeping the source run's plan anchor. Regression mirrors your workflowGraph repro — same spec with a comment prepended, reuse hits with one call, and the adopted step maps onto the current topology's single node (planId equals the current node's, differs from run 1's, exactly one topology node for the agent). Full suite 91/91, packaged smoke 1/1, rebuild byte-reproducible.

[hetaoBackend]

Re-reviewed exact head accae5cb3f959e01b3c95452c68e52d2a5282ca4. The planId fix is verified with the actual workflowGraph: after shifting the script's source positions, the reused step maps to one successful node in the current topology. The earlier public MCP schema, full-key lookup filtering, and changed-upstream-prompt/output cases are addressed. Two correctness issues remain before merge.

1. [P1] An upstream rerun with unchanged returned text can still reuse a stale downstream result

src/engine.mjs:195 derives dependency identity from the upstream spec/lineage and output hash. It does not distinguish a fresh upstream execution whose filesystem effects changed but whose returned value stayed the same; :217 therefore accepts the old downstream candidate.

Independent reproduction with the actual Engine/Store, a deterministic executor, and real temporary-file I/O:

• Script: A has no explicit model; B has model:'m2', dependsOn:['a'], and an unchanged prompt to read produced.txt. Executor mode is mcode, with the test executor injected, so A is deliberately ineligible for cross-run reuse and runs again.
• Both runs begin with produced.txt = "same baseline", declared in input.files, so approval-time fingerprints match.
• Run 1: A writes old and returns "done"; B reads the file and returns old.
• Reset the file to the same baseline. Run 2 enables reuseAcrossRuns:true: A writes new but again returns "done".
• Actual result: Run 2 returns {a:"done", b:"old"} while the file contains new. Calls are A, B, A; B is incorrectly reused.

The changed-output regression does not cover this unchanged-output case. Please invalidate downstream candidates when a relevant upstream agent was freshly executed, or bind enough execution/artifact identity to establish equivalence safely. Add this same-output, changed-tracked-file regression with B's spec unchanged.

2. [P2] Multiple repairs followed by cross-run reuse lose the original producer

Repair adoption at src/engine.mjs:205-206 overwrites reusedFrom without initializing/preserving the first producer. The fallback at :231 later assumes that a repair step's immediate source is the original producer.

Independently reproduced:

1. R1 executes node A.
2. R2 repairs R1 and reuses A.
3. R3 repairs R2 and reuses A.
4. R4 enables cross-run reuse and adopts A from R3.

R4 correctly records reusedFrom.runId = R3, but incorrectly records originalProducer.runId = R2, even though R2 made zero calls and R1 actually produced the output. Please preserve original-producer provenance during repair adoption too, and add a mixed multi-repair-to-cross-run regression.

Validation at this exact head: plugin checks 91/91, packaged MCP smoke 1/1, byte-identical rebuild, and clean-tree repository validator pass. These additional reproductions use real orchestration/storage and deterministic execution; they do not involve real model calls.

Separately, current exact-head GitHub checks are 4 passed / 1 failed / 1 skipped. Ubuntu CI run 35318580789 fails a delegation cancelled while queued for the lock never starts its worker in plugins/Hylouis233/cli-agent-bridge/test/server.test.mjs:252 (assertion at line 266; false !== true). That failure is in another plugin; I am not claiming this diff caused it. Please retain fresh green exact-head repository validation before merging, independently of the two reproduced issues above.

[modacker]

On the red validate at accae5c (run 35318580789): the single failure is not ok 10 — a delegation cancelled while queued for the lock never starts its worker — cli-agent-bridge again, fifth distinct test name in the same treeTerminated/fail-closed family (evidence thread: #43 (comment)). This PR does not touch that plugin. The same head passes the complete 493-test repository gate on a local Ubuntu 24.04 box just now — 482 pass / 0 fail / 11 platform skips, including all 91 dynamic-workflow checks.

[modacker]

Rebased onto main after #50 (and #45's merge); exact head is now db2dafb. Validate now runs the stabilized suite — no cli-agent-bridge integration flakes in the default discovery set. Full plugin suite green at this head (100/100, incl. #45's claims/examples checks now in the baseline), packaged smoke 1/1, rebuild byte-reproducible.

[modacker]

Both findings fixed at exact head 0db3327:

1. Same-output / changed-tracked-file: adoption now requires every agent dependency to be itself adopted/reused. A freshly executed upstream has unproven execution identity — spec and output hashes cannot see changed filesystem effects — so downstream falls through to a fresh call; checkpoints stay eligible (deterministic recomputation, value hash already in lineage). Regression mirrors your repro end-to-end with real file I/O: A re-executes returning done both times while writing old→new, B re-executes and reads new (calls: a,b,a,b).
2. originalProducer across repairs: repair adoption now relays originalProducer (first producer) instead of letting the chain reset to the immediate repair source; your R1→R2(repair)→R3(repair)→R4(cross-run) chain now yields reusedFrom.runId=R3 and originalProducer.runId=R1 (regression included).

Suite 102/102, packaged smoke 1/1, rebuild byte-reproducible, rebased on the post-#50 main (validate deterministic).