Uh oh!
There was an error while loading. Please reload this page.
feat(headless): add experimental todo task-tool baseline - #561
Merged
Conversation
Astro-Hanforce-pushed
the
feat/317-crud-lite-task-tools
branch
from
July 6, 2026 06:50
d8749f1 to
b88cf00CompareUh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Default-off Harbor/headless task-tool experiment for #317, built on the task-system wiring from #537.
This PR ships one experimental baseline:
todo_writewith turn-tail guidance/replay. It records minimal task-tool usage in cell output/WAL/A/B reports and wiresMAKA_MAX_STEPSthrough the Harbor host ai-sdk path.The production default is unchanged: no task/todo tool is exposed unless the experiment env flag is set.
Why
Refs #317
We need a baseline for whether common task/todo tool surfaces help long-running benchmark tasks. Local A/B slices and trace review showed that CRUD-lite and bare
todo_writemostly add schema cost without reliably changing model behavior, whiletodo_writewith explicit turn-tail guidance makes the model externalize and update a task plan in most runs.The evidence is still not strong enough for production enablement, so this PR keeps the path experimental, default-off, and removable.
Scope
Changed:
todo_write.MAKA_CONTEXT_TASK_TOOLS=on, withMAKA_CONTEXT_TASK_REPLAY_MAX_CHARSfor replay/guidance bounds.taskToolSummarythrough Harbor cell output, fixed-prompt WAL events, and A/B summaries/reports. Per-cell summary stores onlytodoWriteCalls; activation and activated attempt ids are derived fromtodoWriteCalls > 0.MAKA_MAX_STEPSinto the Harbor ai-sdk backend path.todo_writeregistration, replay caps, safe rendering, Harbor env isolation, output/WAL/report summaries, runtime-policy env isolation, and max-steps parsing.Not included:
Verification
npm run -w @maka/headless testpassed on 2026-07-06: 726 tests, 725 pass, 0 fail, 1 skipped. The skipped test requires a clean execution checkout to prove prompt-repo safety ordering.git diff --checkpassed./tmp/maka-task-tools-final/task-tools-final-1783289721020/summary.md.runtimeEventsPathfunction-call sequences,model_stream_startedtool schema metadata, and task/todo call timing for off/CRUD/bare-todo/guided-todo arms.User-facing impact
None. This is a default-off headless/Harbor experiment. No UI, docs, changelog, migrations, or breaking changes.
Reviewer notes
Experiment evidence:
Primary evidence uses the default active-prune setting from #340. The treatment under test is still only task/todo tools off vs on. CRUD-lite and bare
todo_writeare historical shape-screening evidence and are not retained as production paths in this PR.crud-vs-off-prune-on-highcaptodo_writetodo-vs-off-prune-on-highcaptodo_write+ guidancetodo-guidance-vs-off-prune-on-highcapTrace findings:
2723chars to4442chars, but only activated in1/8attempts in both high-cap cases.todo_writealso mostly behaved like extra schema. It raised visible tool schema from about2723chars to3545chars, but activated in only2/8attempts and often appeared late.todo_writechanged the trace shape: activation reached7/8, the median first task-tool call position was1, and many long tasks calledtodo_writewithin about 2-3 seconds.mailmanandfinancial-document-processor, the model repeatedly rewrote the todo list fromin_progresstocompletedand used it as a process anchor through final verification.todo_writeimprovedlarge-scale-text-editingfrom budget-exhausted/fail to pass and reduced non-task calls slightly overall, butfinancial-document-processorandmteb-leaderboardbecame longer or more expensive.large-scale-text-editing. So the trace supports better long-task behavior shape, not a stable score-improvement claim.overfull-hboxmarked todo items complete but failed verification.Interpretation:
todo_writeplus turn-tail guidance as the experimental baseline.1/8in both high-cap cases.todo_writeis excluded because adoption stayed at2/8and score was worse in both high-cap cases.todo_writeincreased candidate cost by about18.8%in the default active-prune slice, so it should stay experimental until a longer-task confirmatory slice proves value.Review focus:
MAKA_CONTEXT_TASK_TOOLS=onshould be the only task-tool entrypoint.todo_write.taskToolSummaryshould stay minimal and store onlytodoWriteCalls; detailed tool-name counts remain in the existingtoolSummary.actualToolCallCountscontract.todoWriteCalls > 0, while enabled-but-zero-call and budget-exhausted cells still count in the observed attempt denominator.Risk / rollback:
Checklist