Skip to content

feat(headless): add experimental todo task-tool baseline - #561

Merged
Astro-Han merged 9 commits into
mainfrom
feat/317-crud-lite-task-tools
Jul 6, 2026
Merged

feat(headless): add experimental todo task-tool baseline#561
Astro-Han merged 9 commits into
mainfrom
feat/317-crud-lite-task-tools

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Default-off Harbor/headless task-tool experiment for #317, built on the task-system wiring from #537.

This PR ships one experimental baseline: todo_write with turn-tail guidance/replay. It records minimal task-tool usage in cell output/WAL/A/B reports and wires MAKA_MAX_STEPS through the Harbor host ai-sdk path.

The production default is unchanged: no task/todo tool is exposed unless the experiment env flag is set.

Why

Refs #317

We need a baseline for whether common task/todo tool surfaces help long-running benchmark tasks. Local A/B slices and trace review showed that CRUD-lite and bare todo_write mostly add schema cost without reliably changing model behavior, while todo_write with explicit turn-tail guidance makes the model externalize and update a task plan in most runs.

The evidence is still not strong enough for production enablement, so this PR keeps the path experimental, default-off, and removable.

Scope

Changed:

  • Adds a headless task-ledger experiment tool surface with only todo_write.
  • Enables the tool only under MAKA_CONTEXT_TASK_TOOLS=on, with MAKA_CONTEXT_TASK_REPLAY_MAX_CHARS for replay/guidance bounds.
  • Injects replay/guidance through the turn-tail prompt so the base system prompt hash stays stable.
  • Records minimal taskToolSummary through Harbor cell output, fixed-prompt WAL events, and A/B summaries/reports. Per-cell summary stores only todoWriteCalls; activation and activated attempt ids are derived from todoWriteCalls > 0.
  • Wires runtime-policy A/B arm-local context env through Harbor.
  • Wires host-side MAKA_MAX_STEPS into the Harbor ai-sdk backend path.
  • Adds focused tests for default-off behavior, todo_write registration, replay caps, safe rendering, Harbor env isolation, output/WAL/report summaries, runtime-policy env isolation, and max-steps parsing.

Not included:

  • Production/default task-tool enablement.
  • CRUD-lite shape, CRUD tools, CRUD env entrypoint, CRUD store methods, or CRUD summary fields. CRUD-lite evidence remains historical experiment artifact only.
  • Active tool prune behavior changes; this PR assumes the default-on active-prune setting validated in feat(headless): validate active-prune 2048 #340.
  • A full benchmark rollout or acceptance claim.
  • UI changes.
  • Docs, changelog, migrations, or breaking changes.
  • Automatically deleting the default-off experiment path if later evidence is negative.

Verification

  • npm run -w @maka/headless test passed on 2026-07-06: 726 tests, 725 pass, 0 fail, 1 skipped. The skipped test requires a clean execution checkout to prove prompt-repo safety ordering.
  • git diff --check passed.
  • Commit split check: review fixes are split by revert reason, including the minimal summary cleanup commit.
  • Local experiment artifact: /tmp/maka-task-tools-final/task-tools-final-1783289721020/summary.md.
  • Manual trace review compared runtimeEventsPath function-call sequences, model_stream_started tool schema metadata, and task/todo call timing for off/CRUD/bare-todo/guided-todo arms.

User-facing impact

None. This is a default-off headless/Harbor experiment. No UI, docs, changelog, migrations, or breaking changes.

Reviewer notes

Experiment evidence:

Primary evidence uses the default active-prune setting from #340. The treatment under test is still only task/todo tools off vs on. CRUD-lite and bare todo_write are historical shape-screening evidence and are not retained as production paths in this PR.

ShapeStatus in this PRCaseA passB passDeltaB activatedB callsCost deltaTrace missing
CRUD-liteexcludedcrud-vs-off-prune-on-highcap5/84/8-0.1251/81+65.3%4
bare todo_writeexcludedtodo-vs-off-prune-on-highcap5/83/8-0.252/86+15.0%0
todo_write + guidanceretainedtodo-guidance-vs-off-prune-on-highcap4/85/8+0.1257/821+18.8%4

Trace findings:

  • Historical CRUD-lite mostly behaved like extra schema, not a workflow change. It raised visible tool schema from about 2723 chars to 4442 chars, but only activated in 1/8 attempts in both high-cap cases.
  • Historical bare todo_write also mostly behaved like extra schema. It raised visible tool schema from about 2723 chars to 3545 chars, but activated in only 2/8 attempts and often appeared late.
  • Guided todo_write changed the trace shape: activation reached 7/8, the median first task-tool call position was 1, and many long tasks called todo_write within about 2-3 seconds.
  • In successful long-task traces such as mailman and financial-document-processor, the model repeatedly rewrote the todo list from in_progress to completed and used it as a process anchor through final verification.
  • The effect was not uniformly beneficial. In the default active-prune slice, guided todo_write improved large-scale-text-editing from budget-exhausted/fail to pass and reduced non-task calls slightly overall, but financial-document-processor and mteb-leaderboard became longer or more expensive.
  • The only score lift in the default active-prune slice came from large-scale-text-editing. So the trace supports better long-task behavior shape, not a stable score-improvement claim.
  • Failure traces still show false completion risk: overfull-hbox marked todo items complete but failed verification.

Interpretation:

  • This PR ships only todo_write plus turn-tail guidance as the experimental baseline.
  • CRUD-lite is excluded because adoption stayed at 1/8 in both high-cap cases.
  • Bare todo_write is excluded because adoption stayed at 2/8 and score was worse in both high-cap cases.
  • Guided todo_write increased candidate cost by about 18.8% in the default active-prune slice, so it should stay experimental until a longer-task confirmatory slice proves value.
  • Any follow-up evaluation should assume active prune stays on by default and keep the main comparison as baseline tools off vs the selected task/todo treatment on. Variant-vs-variant runs can explain mechanism, but they should not replace the baseline acceptance question.

Review focus:

  • MAKA_CONTEXT_TASK_TOOLS=on should be the only task-tool entrypoint.
  • When enabled, the extra tool surface should be exactly todo_write.
  • Per-cell taskToolSummary should stay minimal and store only todoWriteCalls; detailed tool-name counts remain in the existing toolSummary.actualToolCallCounts contract.
  • A/B task-tool activation should be derived from todoWriteCalls > 0, while enabled-but-zero-call and budget-exhausted cells still count in the observed attempt denominator.
  • Task replay/tool-result text should stay bounded and rendered through the safe ledger renderer.

Risk / rollback:

  • Risk is limited to explicit experiment runs, but enabled runs can increase model cost.
  • Individual commits are split by revert reason. If the experiment is not useful, remove or revert the default-off task-ledger experiment and Harbor wiring without changing normal headless/Harbor behavior.

Checklist

  • Scope matches the PR title and excludes unrelated changes.
  • Verification lists commands/results.
  • User-facing impact, docs, changelog, migrations, and breaking changes are noted.
  • Risk, rollback, or review focus is called out for non-trivial changes.
  • UI changes are not applicable.

@Astro-Han
Astro-Hanforce-pushed the feat/317-crud-lite-task-tools branch from d8749f1 to b88cf00CompareJuly 6, 2026 06:50
@Astro-Han
Astro-Han merged commit 3dca793 into mainJul 6, 2026
2 checks passed
@Astro-Han
Astro-Han deleted the feat/317-crud-lite-task-tools branch July 14, 2026 05:05
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han