[Follow-up] Implement an evidence adapter function integrated (PR #2757)

[stranske]

Why

PR #2757 established the read-only consumer-sync shadow handoff, but the current main runtime rail still needs one executable path for strict evidence ingestion, idempotency, safety controls, and machine-readable reporting. The current source of truth is scripts/orchestrator_runtime/; the former src/consumer_sync/* plan was never created and must not be used as a phantom target.

Scope

Extend the existing orchestrator runtime rail and its tests. Keep all consumer-sync effects read-only until the existing promotion policy is satisfied.

Non-Goals

Do not create a second capability ledger, accept raw prompt text as evidence, or enable consumer writes. Scaffold-only modules without executable tests and deliberate-break evidence do not count as completion.

Tasks

• Extend scripts/orchestrator_runtime/evidence_schema.py and scripts/orchestrator_runtime/completion_event_adapter.py to parse strict consumer-sync plan evidence and reject prose, malformed payloads, spoofed capability IDs, oversized refs, and missing provenance before state mutation.
• Extend scripts/orchestrator_runtime/capability_lifecycle.py with stable evidence identity, ledger-backed duplicate replay, counterexample preservation, and candidate retirement state.
• Extend scripts/orchestrator_runtime/runner_effect_bridge.py with read-only effect allowlists, TTL checks, kill-switch termination, and explicit blocked-reason output.
• Add scripts/orchestrator_runtime/consumer_sync_report.py and wire it to the existing runtime ledger so every candidate is classified as no-data, healthy-shadow-evidence, or failed-evidence-ingestion.
• Add validation, replay, safety, reporting, and retirement coverage in tests/orchestrator_runtime/test_completion_event_adapter.py and new tests/orchestrator_runtime/test_consumer_sync_runtime.py fixtures.

Acceptance Criteria

• python -m pytest tests/orchestrator_runtime/test_completion_event_adapter.py tests/orchestrator_runtime/test_consumer_sync_runtime.py -q passes with non-zero collection.
• Invalid or replayed evidence produces no additional capability mutation or effect application, and accepted counterexamples remain queryable across process boundaries.
• Shadow reporting returns exactly one of the three named evidence states and promotion remains blocked when rollback, TTL, harm, volume, or human-attention policy gates fail.
• Deliberate-break gate: bypass the duplicate-evidence short circuit in scripts/orchestrator_runtime/capability_lifecycle.py; tests/orchestrator_runtime/test_consumer_sync_runtime.py::test_duplicate_plan_reuses_ledger_result must fail, then restore the guard.

Implementation Notes

PR #2757 and issue #2753 are the shadow-handoff predecessor. Reuse the versioned runtime rail added under scripts/orchestrator_runtime/ and preserve the source-first Workflows ownership boundary.

[agents-workflows-bot]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Starting issue formatting...

Running formatter inline (not via label trigger).

[stranske-keepalive]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Analyzing issue for improvements...

Running optimizer inline (not via label trigger).

[stranske-keepalive]

Issue Optimization Suggestions

Review the suggestions below. If you want the agent to apply them, add the
agents:apply-suggestions label to this issue.

Task splitting

• Implement strict consumer-sync plan schema validation in the main processing rail in src/consumer_sync/rail.*, src/consumer_sync/schema.*, and src/consumer_sync/adapter.* so non-conforming payloads return a rejection result with an error log entry before downstream effects. (This task combines schema definition, validation logic implementation, adapter integration, and error handling across multiple modules. Each component requires separate implementation and testing.). Suggested split: Define the strict consumer-sync plan JSON schema in src/consumer_sync/schema.* with all required fields and validation rules, Implement schema validation logic in src/consumer_sync/rail.* that parses incoming payloads against the defined schema, Create rejection result types and error logging in src/consumer_sync/adapter.* for non-conforming payloads, Integrate validation into the main processing entry point to reject invalid inputs before downstream effects.
• Implement ledger-backed idempotency in the consumer-sync processing path in src/consumer_sync/rail.*, src/consumer_sync/idempotency.*, and src/ledger/* by persisting or reading a stable plan identity and replaying recorded results for duplicate submissions. (This task spans plan identity computation, ledger storage, retrieval logic, and replay mechanism across three different modules. Each aspect requires distinct implementation.). Suggested split: Implement stable plan identity computation in src/consumer_sync/idempotency.* using deterministic hashing of plan content, Create ledger storage methods in src/ledger/* to persist plan identities with execution outcomes, Implement ledger query methods in src/ledger/* to retrieve prior execution results by plan identity, Add replay logic in src/consumer_sync/rail.* to return recorded results for duplicate plan submissions.
• Extend durable ledger persistence in src/ledger/* and src/consumer_sync/heartbeat.* to store consumer-sync plan identities, execution outcomes, and capability heartbeat events for duplicate detection and reporting. (This task combines three distinct data types (plan identities, outcomes, heartbeats) with different schemas and persistence requirements.). Suggested split: Define ledger schema for consumer-sync plan identities and execution outcomes in src/ledger/*, Implement persistence methods in src/ledger/* to store plan identities and execution outcomes durably, Define capability heartbeat event schema in src/consumer_sync/heartbeat.* with required metadata, Implement heartbeat event persistence in src/consumer_sync/heartbeat.* to enable post-processing queries.
• Implement a machine-readable report command in src/consumer_sync/report_command.*, src/consumer_sync/reporting.*, and src/cli/* that reads storage/ledger evidence and classifies each candidate as no-data, healthy-shadow-evidence, or failed-evidence-ingestion. (This task combines CLI command creation, evidence retrieval logic, classification algorithm, and output formatting across multiple layers.). Suggested split: Implement evidence retrieval logic in src/consumer_sync/reporting.* to query storage and ledger for candidate data, Create classification logic in src/consumer_sync/reporting.* to categorize candidates into the three defined states, Define machine-readable output format for the report command with structured candidate classifications, Implement CLI command interface in src/cli/* and src/consumer_sync/report_command.* to invoke reporting logic.
• Implement promotion-gate evaluation logic in src/consumer_sync/promotion_gate.* and src/consumer_sync/policy.* that checks minimum volume, success rate, harm limit, rollback-path presence, TTL validity, and human-attention flags, and returns blocked decisions with unmet checks. (This task requires implementing six distinct validation checks, each with different data sources and threshold logic, plus decision aggregation.). Suggested split: Implement minimum evidence volume check in src/consumer_sync/promotion_gate.* with configurable threshold, Implement success rate validation in src/consumer_sync/promotion_gate.* against configured minimum percentage, Implement harm limit check in src/consumer_sync/promotion_gate.* to enforce maximum acceptable harm threshold, Implement rollback-path presence validation in src/consumer_sync/promotion_gate.* using rollback metadata, Implement TTL validity check in src/consumer_sync/promotion_gate.* to reject expired candidates, Implement human-attention flag validation in src/consumer_sync/promotion_gate.* for gated promotions, Create decision aggregation logic in src/consumer_sync/policy.* to return blocked status with unmet check identifiers.
• Implement persistent counterexample storage in src/consumer_sync/counterexamples.*, src/storage/*, and src/ledger/* so ingested counterexamples remain queryable after processing runs complete. (This task combines schema definition, storage implementation, ledger integration, and query interface across three modules.). Suggested split: Define counterexample data schema in src/consumer_sync/counterexamples.* with required fields and candidate linkage, Implement counterexample persistence methods in src/storage/* for durable storage, Add counterexample ledger integration in src/ledger/* to enable cross-session queries, Create query interface in src/consumer_sync/counterexamples.* to retrieve stored counterexamples by candidate.
• Add candidate retirement logic in src/consumer_sync/candidate_state.* and the reporting/query layer so candidates are marked retired when stored counterexamples meet the retirement condition and that state is exposed through queries. (This task combines retirement condition evaluation, state update logic, and query layer integration across multiple components.). Suggested split: Implement retirement condition evaluation in src/consumer_sync/candidate_state.* based on stored counterexample records, Create candidate state update methods in src/consumer_sync/candidate_state.* to mark candidates as retired, Integrate retirement state into the reporting layer to expose retired status in queries, Add retirement state filtering to query interfaces for candidate visibility.

Blocked tasks

• Ensure CI fully executes the relevant workflows and test suites for this work before merge. (Agent cannot retry CI pipelines or modify workflow configurations in .github/workflows/*.yml (protected files). Agent can only write tests that CI will execute. | Replace with: 'Add all new test files to existing test discovery patterns so CI automatically executes them' or mark as manual verification step for human reviewer.)

Objective acceptance criteria

• The promotion-gate evaluator returns allow only when all configured checks pass: minimum evidence volume, minimum success rate, maximum harm limit, rollback-path presence, non-expired TTL, and required human-attention conditions. (The term 'configured checks pass' is subjective without specifying what constitutes a passing check or the exact threshold values. | Specify: 'The promotion-gate evaluator returns allow only when: (a) evidence volume >= configured minimum_volume threshold, (b) success_rate >= configured minimum_success_rate, (c) harm_count <= configured maximum_harm_limit, (d) rollback_handler is defined and rollback_metadata is present, (e) current_time < TTL_expiry, and (f) human_attention_required flag matches approval state.')
• When stored counterexamples for a candidate meet the configured retirement condition, the candidate state is updated to retired and that state is exposed by the same reporting or query layer used for candidate visibility. (The 'configured retirement condition' is not defined, making it impossible to verify objectively when retirement should trigger. | Specify the exact retirement condition, e.g., 'When the count of stored counterexamples for a candidate exceeds the configured retirement_threshold value' or 'When any counterexample has severity level >= CRITICAL'.)

Missing sections

• None

Formatting issues

• The 'Implementation Notes' section uses prose paragraphs mixed with file path listings. Consider using consistent bullet points or subsections for better readability.
• File path patterns use wildcards (e.g., rail.*, schema.*) which may be ambiguous. Consider specifying exact file extensions or providing examples (e.g., rail.py, rail.ts).
• The 'Non-Goals' section could benefit from bullet formatting for the two distinct non-goals mentioned (dashboard interface and threshold values).

Notes

• This issue is well-structured with comprehensive sections including Why, Scope, Non-Goals, Tasks, Acceptance Criteria, and Implementation Notes. The main concerns are: (1) Several tasks are too broad and combine multiple implementation concerns that should be split for independent completion, (2) One task requires CI control beyond agent capabilities, (3) Some acceptance criteria reference 'configured' values without defining them, making objective verification difficult. The issue demonstrates good practices by explicitly listing file areas, providing context from prior work, and distinguishing implementation from design decisions. The acceptance criteria are generally detailed and testable, though a few would benefit from more precise threshold specifications.

[agents-workflows-bot]

🤖 Auto-pilot step $((STEP_COUNT + 1)): Applying optimization suggestions...

Running apply inline (not via label trigger).

[agents-workflows-bot]

🤖 Auto-pilot step 1: Issue prepared! Assigning to agent...

Adding agent:codex label. The capability check will run automatically.

⏳ Agent will create a PR shortly.

[stranske-automation-bot]

Belt dispatcher queued this issue and created branch codex/issue-2761.

[agents-workflows-bot]

🤖 Auto-pilot step 1: Branch ready, waiting for commits

Branch codex/issue-2761 exists but has no commits yet.
✅ Dispatched the belt worker to generate changes.

📊 Stall count: 1/5 (pauses after 5)

⏳ Auto-pilot will check again on next trigger.