Skip to content

Design: Separate Autonomous Task Maka from Interactive Maka #45

Description

@likun666661

upstream_issue_title

Design: Separate Autonomous Task Maka from Interactive Maka with first-class loop policy, verifier, isolation, budget, trace, and artifacts

upstream_issue_body

problem

Maka now has useful PR44 foundation for headless benchmark execution, but it should not be treated as the final autonomous agent loop product.

Confirmed evidence from upstream artifacts:

  • Current Maka baseline 9ca47663 / PR44 has RuntimeRunner, AgentRun ledger, RuntimeEvent read model, AiSdkFlow, AiSdkBackend, ToolRuntime, packages/headlessConfig x Task, real-backend isolation seam, isolated Bash executor seam, protected verifier assets, and JSONL result records.
  • SessionManager is still the public interactive facade; sendMessage, retry, regenerate, branch, turn state, and chat recovery are interactive-session semantics.
  • RuntimeRunner is a single invocation shell with stronger failure accounting after PR44, but it is not a persistent task controller.
  • AiSdkBackend still owns the actual model/tool loop, including provider stream, tool calling, permission parking, context budget, and tool availability.
  • packages/headless currently runs one task invocation plus verification command; it does not define task backlog, durable autonomous loop state, attempt retry, budgeted continuation, task inbox, or hidden verifier/product scoring as first-class concepts.
  • Pie evidence shows a useful split: the inner model/tool loop handles one assistant turn and tool-use closure, while the harness/controller handles cross-turn continuation, budget, audit, pause, and product delivery.

Design inference:

  • Maka should treat Interactive Maka and Autonomous Task Maka as two product shapes that share runtime/model/tool foundations, but do not share loop policy, state machine, verifier semantics, or result semantics.
  • Terminal-Bench and Harbor should be the first benchmark carriers, not the architecture boundary. The loop should work for Terminal-Bench, SWE-bench, local headless tasks, and future operational task runners.
  • A Harbor-only retry wrapper would collapse provider retry, attempt retry, task retry, verifier retry, and human handoff into one mechanism. That would not provide durable task state, fair benchmark accounting, permission scope, artifact audit, or resume semantics.

proposal

Add an autonomous task product shell above the existing runtime:

TaskProduct API / CLI / UI
-> TaskAgentController
-> AutonomousAgentLoop
-> TaskRunStore / TaskAttempt ledger / TaskArtifacts
-> RuntimeRunner
-> AiSdkFlow
-> AiSdkBackend / ModelAdapter / ToolRuntime

Core product rule:

  • Interactive Maka owns human chat sessions, user messages, permission UX, retry/regenerate/branch, and claim-to-chat handoff.
  • Autonomous Task Maka owns task definitions, task runs, attempts, verifier/scorer results, budgets, trace lineage, artifacts, retry policy, and task inbox.
  • Both may reuse RuntimeRunner, ToolRuntime, ModelAdapter, AgentRunStore, RuntimeEventStore, cost telemetry, and permission primitives.
  • Autonomous continuation prompts must be recorded as task observations/instructions, not as fake human user messages in an interactive transcript.

Recommended modules and contracts:

moduleresponsibility
TaskDefinitionGoal, input/case, workspace fixture, model policy, tool policy, verifier/scorer spec, budget, artifact retention, result schema. Extends the current headless Task without binding the design to Harbor.
TaskAgentControllerProduct facade: createTask, runTask, inspectRun, abortRun, resumeRun, retryFailed, export, claimInboxItem. It should not call SessionManager.sendMessage() for autonomous continuation.
AutonomousAgentLoopDurable controller: preflight, invoke runtime, persist attempt result, observe visible self-check, run verifier/feedback, decide continue/submit/pause/block/fail.
TaskRunDurable task envelope: status, root trace, creator/source, policy, budgets, workspace lease, isolation record, case matrix, current attempt.
TaskAttemptOne auditable try for one case/step: attempt number, runtime invocation id, workspace snapshot, permission/isolation policy, cost, failure class, artifacts, verifier result.
TaskRunStoreAppend-only ledger for task, attempt, decision, verifier, budget, permission, artifact, handoff, and status events. It may reuse Maka store patterns but needs task-specific event schema.
TaskEventTyped facts such as task_started, attempt_started, runtime_event_ref, self_check_observed, feedback_observed, decision_recorded, score_recorded, task_paused, task_completed.
SelfCheckObservationVisible agent feedback: claimed status, checked items, evidence refs, risks, recommended action, confidence, schema validity. It is not an authority to pass or approve.
FeedbackObservationStructured feedback from runtime failure, deterministic verifier, hidden verifier, tool failure, budget, permission, and human input. This becomes input to the next task step.
AutonomousDecisionStructured action: continue, submit, pause, needs_approval, budget_exhausted, failed, with reason, evidence refs, policy check ids, and next task instruction if continuing.
VerifierSpec / FinalScorerDefines and executes final benchmark/task scoring: command verifier, Terminal-Bench oracle, SWE-bench tests, LLM judge, or human approval. Output is structured ScoreResult, not assistant text.
TaskInboxHuman triage queue for approvals, ambiguous failures, budget extension, claim-to-chat, or findings. Background task output does not automatically enter the main chat.
ArtifactStoreStable refs for trajectory, logs, diff, screenshots, browser traces, verifier output, score report, workspace bundle, and reproducer command.
BudgetGuardEnforces USD, tokens, wall clock, tool calls, attempts, continuations, provider retries, verifier retries, and storage before every continuation/retry/verifier run.
TaskContextProjectorBuilds next model context from task ledger and artifact refs. Compression/synthesis must be based on durable event/artifact ids, not in-memory transcript indexes.

state machine

Minimum autonomous task state machine:

queued
-> admitted
-> attempt_started
-> preflight
-> model_tool_loop
-> turn_persisted
-> visible_self_check
-> feedback_observation
-> decision
-> continue -> next_attempt_or_continuation -> preflight
-> submit -> final_scoring -> completed | failed
-> pause | needs_approval -> paused
-> budget_exhausted -> blocked
-> abort_requested -> aborted

State semantics:

  • admitted: validate durable dedup key, root trace, cycle hop, workspace lease, isolation policy, baseline permission, and ledger writeability.
  • preflight: check run status, abort/pause markers, budget, deadline, attempt cap, continuation cap, tool-call cap, permission scope, and context projection.
  • model_tool_loop: call existing RuntimeRunner.run() for a single invocation. RuntimeRunner and AiSdkBackend continue to handle provider stream, tool calls, permission-denied events, failure accounting, and abort propagation.
  • turn_persisted: link RuntimeEvent / AgentRun facts into TaskAttempt, including terminal status, failure class, usage/cost, tool events, workspace diff summary, and artifact refs.
  • visible_self_check: record SelfCheckObservation. Missing schema, missing evidence, or contradiction with tool/verifier evidence should mark it invalid or needs_review; it cannot authorize pass, submit, promotion, or permission.
  • feedback_observation: run deterministic verifier, hidden verifier, runtime failure mapper, tool failure mapper, and budget/permission checks. The output is structured evidence for the decision step.
  • decision: choose continue, submit, pause, needs_approval, budget_exhausted, failed, or aborted. Every decision is a ledger fact with trace id, previous observation hash, proposed action hash, and budget state.
  • final_scoring: run FinalScorer over a frozen submitted artifact/workspace state. completed means the runner reached a terminal product state; passed means the scorer accepted the result.

Visible product states:

statemeaningresume path
queuedWaiting for runner/resource.Automatic.
runningCurrent runtime attempt is executing.Abort only.
verifyingRuntime turn ended; self-check/verifier/scorer is running.Abort only.
repairingDecision was continue; next task observation/instruction is being prepared.Automatic if budget/policy allows.
pausedNeeds user input, approval, or policy-controlled pause.resumeRun.
blockedBudget, permission, credential, isolation, or environment condition blocks progress.Change condition, then resumeRun.
completedTask runner finished and has final score/result.Export/retry.
failedNot automatically recoverable under current policy.Inspect/retry.
abortedUser/system cancellation.New run or explicit resume policy.

module boundaries

RuntimeRunner:

  • Keep single invocation semantics.
  • Accept autonomous lineage metadata such as taskId, runId, attemptId, caseId, rootTraceId, and source: 'autonomous_task'.
  • Return invocation facts and failure classes, but do not own task scoring, attempt caps, benchmark matrix semantics, or inbox.

SessionManager:

  • Remains the interactive product facade.
  • Autonomous loop must not use sendMessage() as its continuation API.
  • Claim-to-chat may create/link an interactive session, but it must include task provenance and source prefix. It must not masquerade as human input.

AiSdkBackend / future ModelToolLoop:

  • Short term: keep current backend loop; autonomous controller wraps around completed invocations.
  • Medium term: extract a clearer ModelToolLoop seam only after task product nouns stabilize.
  • Backend handles provider stream, tool execution, permission parking, context budget, and tool availability. It does not own benchmark pass/fail or task retry.

packages/headless:

  • Treat current runExperiment as an adapter or initial runOneCase, not the autonomous loop core.
  • Extend Task into TaskDefinition; extend ResultRecord with attempts, cost, artifact refs, verifier/scorer details, isolation policy, budgets, and taxonomy.
  • Keep CLI fail-closed: real/model-backed backend still requires explicit isolation. Operational autonomous mode should be a separate command path from the safe fake eval smoke path.

Storage:

  • Keep three ledgers separate:
    • TaskRunStore: task/attempt/decision/verifier/status ledger.
    • RuntimeEventStore / AgentRunStore: bottom-level invocation facts.
    • ArtifactStore: trajectory/log/diff/screenshot/verifier/score/workspace refs.
  • Dedup keys, cycle hops, admission outcomes, permission outcomes, budget checks, and isolation records must be durable. Rebuild runtime state from ledger after restart.

Execution isolation and permission policy:

  • Non-fake/model-backed backends require explicit isolation before backend registration and before each attempt.
  • Isolation record includes kind, label, executor identity, workspace lease, filesystem mount contract, env policy, network policy, timeout policy, secret policy, and artifact retention policy.
  • Permission grants bind to {taskId, runId, attemptId, toolName, normalizedArgsHash, resourceScope, expiry}.
  • CI/headless mode fails closed on approval prompts. Desktop mode may enter needs_approval, but must not continue while parked.
  • Assistant text and visible self-check are not permission records.

Trace and artifact policy:

  • Every task run has a root trace id. Attempts, verifier runs, handoffs, promotions, provider retries, tool retries, and continuation decisions join that trace.
  • Artifacts are first-class refs in task events and result rows, not markdown-only attachments.
  • Official benchmark result rows must link trajectory, diff/workspace snapshot, verifier output, score record, config, taxonomy class, budget policy, and isolation policy.

Result taxonomy:

classmeaningpass eligible
agent_successAgent submitted an artifact and final verifier/scorer passed.yes
agent_failureAgent made a complete submission, but verifier/scorer rejected it.no
agent_incompleteAgent did not reach valid submission, for example max tokens, missing terminal, repeated cycle, malformed required self-check, no artifact.no
verifier_failureVerifier/scorer contract failed independently of solution correctness.no official score
infra_failurePlatform, isolation, backend setup, storage, workspace, or transport failed before fair result.no official score
policy_deniedSafety/policy prevented execution, tool use, verification, continuation, or promotion.no
budget_exhaustedDeclared hard budget ran out before success.no
agent_abortedUser/system cancellation.no
blockedExternal input or condition needed; not final benchmark data.no

Classification rules:

  • passed=true is valid only with agent_success.
  • A verifier exit code that indicates wrong solution is agent_failure, not verifier_failure.
  • Verifier binary missing, invalid scorer schema, or judge service outage is verifier_failure or infra_failure, not agent_failure.
  • Permission denial is policy_denied even when surfaced as a runtime terminal event.
  • Placeholder, fake backend, dry run, skipped verifier, visible-only self-check, or unsupported real-backend CLI path must not enter official pass-rate denominators unless explicitly reported as a separate smoke metric.

phased implementation

Recommended PR sequence after PR44 is 6 small PRs:

  1. PR45: task contracts and ledger skeleton

    • Add TaskDefinition, TaskRun, TaskAttempt, TaskEvent, SelfCheckObservation, FeedbackObservation, AutonomousDecision, VerifierResult, ScoreResult.
    • Add append-only TaskRunStore and status projection.
    • Add top-level result taxonomy and compatibility adapter from current headless Task / ResultRecord.
    • Verify append/replay, status projection, taxonomy mapping, and result compatibility.
  2. PR46: TaskAgentController single-attempt facade

    • Add TaskAgentController.createTask/runTask/inspectRun/abortRun/resumeRun/retryFailed/export.
    • First implementation supports one case, one attempt, one RuntimeRunner.run(), plus current verification command.
    • Prove it does not use SessionManager.sendMessage() for autonomous continuation.
    • Persist isolation record, budget policy, runtime result, artifact refs, and score/result row.
  3. PR47: AutonomousAgentLoop v1

    • Implement durable sequence: preflight -> runtime invocation -> visible self-check -> feedback observation -> decision.
    • Enforce attempts, continuations, wall clock, token/USD, tool-call, provider retry, verifier retry, and storage budgets.
    • Add decision actions continue, submit, pause, needs_approval, budget_exhausted, failed.
    • Ensure visible self-check is recorded but cannot set pass/submit/approval.
  4. PR48: verifier/scorer and benchmark contract

    • Add VerifierSpec and FinalScorer abstractions for command verifier first, with Terminal-Bench/SWE-bench hooks left as adapters.
    • Freeze submitted workspace/artifacts before scoring; restore protected paths before verifier.
    • Split runner completed from score passed.
    • Add official result denominator rules and structured matrix-level errorClass.
  5. PR49: isolation, permission, and task inbox hardening

    • Promote RealBackendIsolation, env/network/secret policy, workspace lease, and tool executor identity into task ledger facts.
    • Add task-scoped permission grants with normalized args hash/resource scope/expiry.
    • Add TaskInbox for approvals, ambiguous failures, budget extension, and claim-to-chat.
    • Keep headless CI fail-closed; desktop may park in needs_approval.
  6. PR50: benchmark adapters, reporting, and operational CLI

    • Add Terminal-Bench adapter as first benchmark carrier without making Harbor the architecture.
    • Add result export with trajectory, diff/workspace snapshot, verifier output, score record, budget, isolation, and taxonomy.
    • Add maka task run/inspect/resume/retry-failed/export commands.
    • Add matrix resume/retry-failed basics; defer parallel pools/sharding/rich UI if needed.

acceptance criteria

Functional:

  • Interactive chat flows continue to use SessionManager; autonomous task flows use TaskAgentController.
  • Autonomous continuation creates task events/instructions, not fake human chat messages.
  • A task run can be inspected after restart from append-only task ledger and linked runtime/artifact refs.
  • Task results distinguish runner terminal state, final scoring execution, score pass/fail, and official benchmark eligibility.
  • Terminal-Bench/Harbor can be implemented as adapter/carrier; no core module is named or shaped as Harbor-only retry.

Safety and policy:

  • Real/model-backed headless execution without explicit isolation returns policy_denied.
  • Permission prompt in CI/headless fails closed; desktop parks as needs_approval.
  • Permission grant is scoped to task/run/attempt/tool/args/resource/expiry.
  • Protected verifier assets are restored before scoring, and tampering cannot produce passed=true.
  • Visible self-check success text cannot set pass, bypass verifier, approve a tool, claim inbox, or promote to chat.

Benchmark correctness:

  • passed=true requires final scorer/verifier actually ran under the declared contract and passed.
  • Fake, placeholder, skipped verifier, dry run, unsupported real-backend CLI path, infra failure, verifier failure, policy denied, and budget exhausted are excluded from official pass-rate denominators by default.
  • Solution failure maps to agent_failure; verifier contract failure maps to verifier_failure; platform/setup failure maps to infra_failure; permission denial maps to policy_denied.
  • Retry budgets are recorded separately for provider retries, tool retries, continuations, attempts, verifier retries, tokens, cost, wall clock, and human interventions.

risks

  • Semantic collapse: treating completed runner status as task success, visible self-check as verifier, or Harbor retry as autonomous loop would make benchmark results misleading.
  • Permission leakage: inheriting broad interactive approvals into headless/autonomous tasks would allow unintended tool use. Grants must be scoped and durable.
  • Isolation overclaim: PR44 has useful seams, but isolation is still partly caller-asserted. Official benchmark mode must record env/network/secret/filesystem policy as ledger facts.
  • Result taxonomy drift: if CLI, JSONL, UI, and reports each invent error strings, benchmark aggregation will become unreliable.
  • Hidden verifier leakage: official benchmark mode must not feed final scorer output back into the agent before submission, unless the benchmark explicitly defines that feedback loop.
  • Premature graph rewrite: extracting a full execution DAG before task nouns stabilize risks regressions in interactive chat, permissions, abort, telemetry, and context budget. Start with the conservative task shell.

architecture_decision

Decision: build a first-class Autonomous Task Maka product shell above the existing runtime, while keeping Interactive Maka as a separate product shape.

This explicitly rejects a Harbor-only retry wrapper:

  • Harbor/provider retry only addresses transient transport/provider failures.
  • Autonomous task execution needs durable task/run/attempt state, verifier/scorer contracts, permission scope, execution isolation, budget enforcement, trace lineage, artifact export, human handoff, and benchmark result taxonomy.
  • Terminal-Bench/Harbor is the first benchmark carrier, not the design boundary. The same task shell should support future SWE-bench adapters, local headless tasks, and operational task runners.

Confirmed basis:

  • PR44 already provides runtime and headless building blocks but no durable autonomous controller.
  • Pie’s useful lesson is the two-layer loop split: inner model/tool loop versus outer harness/controller policy.

Design inference:

  • The lowest-risk path is conservative: add task shell, ledger, verifier/scorer, and taxonomy first; only later extract a lower-level ModelToolLoop or graph stepper if real adapter needs justify it.

phased_plan

  1. PR45: Task Contracts + Ledger

    • Add task/run/attempt/event/decision/verifier/scorer types and append-only TaskRunStore.
    • Verify replay, status projection, taxonomy mapping, and compatibility with current headless result rows.
  2. PR46: TaskAgentController Single Attempt

    • Add task facade and run one current headless task through RuntimeRunner.
    • Persist runtime refs, artifacts, isolation, budget, and score result.
    • Prove task API does not use interactive sendMessage.
  3. PR47: AutonomousAgentLoop v1

    • Add preflight -> runtime -> self-check -> feedback -> decision loop.
    • Enforce budgets and continuation caps.
    • Make visible self-check non-authoritative in code and tests.
  4. PR48: Verifier/Scorer Contract

    • Add VerifierSpec, FinalScorer, ScoreResult, protected path restore, official denominator rules, and matrix-level structured errorClass.
    • Split runner completion from score pass/fail across JSONL, CLI, and reports.
  5. PR49: Isolation/Permission/Inbox

    • Ledger isolation/env/network/secret/workspace policies.
    • Add task-scoped permission grants and CI fail-closed behavior.
    • Add task inbox and claim-to-chat provenance.
  6. PR50: Benchmark Adapter + Operational Reporting

    • Add Terminal-Bench adapter without Harbor-only coupling.
    • Add task CLI inspect/resume/retry-failed/export and artifact-rich report rows.
    • Add basic matrix resume/retry-failed; defer sharding/parallel resource pools if needed.

acceptance_tests

Required unit tests:

  • Task ledger append/replay projects queued, running, verifying, paused, blocked, completed, failed, aborted.
  • Taxonomy mapper distinguishes runtime failure, verifier solution failure, verifier infrastructure failure, policy denial, budget exhaustion, abort, and setup failure.
  • SelfCheckObservation cannot set passed, cannot approve permissions, cannot bypass final scorer, and cannot hide verifier failure.
  • Budget guard consumes provider retry, tool retry, continuation, attempt, verifier retry, token/cost/wall-clock/tool-call/storage budgets separately.
  • Permission grant binds to normalized args/resource scope/task/run/attempt/expiry.
  • Path containment covers .., absolute paths, symlinks, protected path mutation, restored protected assets, and workspace-relative validation.

Required integration tests:

  • Fake backend happy path writes expected artifact; verifier passes; result is agent_success and passed=true.
  • Runtime completes and self-check claims success, but hidden verifier fails; result is agent_failure, passed=false.
  • Missing terminal, max tokens, tool-call cap, or repeated cycle maps to agent_incomplete.
  • Verifier binary missing or invalid scorer schema maps to verifier_failure and no official score.
  • Model-backed backend without explicit isolation maps to policy_denied.
  • Real backend with external executor records isolation label, workspace lease, env/network policy, and artifact refs.
  • Agent mutates protected grader; restore happens before verification; tampering cannot pass.
  • Restart/resume rebuilds dedup keys, budgets, cycle history, pending approvals, and current status from ledger.

Required smoke/benchmark contract tests:

  • Existing CLI fake eval still works without credentials and produces JSONL plus markdown report.
  • Programmatic real-backend smoke requires explicit isolation and records the isolation label.
  • Denied network/secret task becomes policy_denied or blocked without leaking host environment.
  • Budget-limited task stops as budget_exhausted with consumed budget evidence.
  • Hidden-verifier task proves visible self-check success text does not affect pass/fail.
  • Result denominator test excludes fake, dry-run, placeholder, skipped verifier, verifier failure, infra failure, policy denied, and budget exhausted rows from official pass rate by default.
  • Retry fairness test freezes submitted artifact across verifier infra retries and consumes continuation budget for repair attempts.
  • Artifact audit test ensures every scored row links trajectory, diff/workspace snapshot, verifier output, score record, config, budget policy, isolation policy, and taxonomy class.

source_artifacts

All required upstream artifacts were present and read:

  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/01-pie-loop-spine.md
    • Basis: Pie inner drive_loop, outer AgentHarness.run_turn_with_continuation, OnTurnEndHook, budget/compaction, session persistence, and lesson that cross-turn autonomous policy belongs outside the low-level model/tool loop.
  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/02-pie-harness-product.md
    • Basis: distinction between interactive chat, automation/task product, trigger delivery, inbox/handoff, budget/cost product surface, and why benchmark/headless needs task nouns rather than chat retry.
  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/03-pie-automation-subagent.md
    • Basis: trigger admission, dedup/cycle suppression, sub-agent delivery, promotion risks, structured details over free-form summary, and visible self-check as feedback rather than authorization.
  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/04-maka-current-runtime.md
    • Basis: Maka PR44 current runtime map, RuntimeRunner failure accounting, AgentRun/RuntimeEvent, AiSdkFlow, AiSdkBackend, headless runner, real backend isolation seam, protected path verification, and current gaps.
  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/05-design-delta.md
    • Basis: target architecture, module/interface list, autonomous task state machine, integration boundaries, conservative migration plan, and recommendation against interactive retry or Harbor-only wrapper.
  • /tmp/rive-maka-autonomous-agent-loop-design-20260618/output/06-risk-benchmark-contract.md
    • Basis: safety contract, benchmark contract, visible self-check versus hidden verifier, result taxonomy, verification plan, and non-negotiable benchmark-safe blockers.

Confirmed repository/input context from artifacts:

  • Maka repository: /Users/likun/Desktop/workspace-for-maka/maka-agent
  • Baseline: 9ca47663
  • Recent context: PR44-headless-foundation

Inference boundary:

  • The proposed module names and PR numbers are design recommendations for upstream discussion, not claims that those modules currently exist.
  • Terminal-Bench/Harbor integration details are intentionally adapter-level recommendations; the core design is benchmark-carrier agnostic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions