Uh oh!
There was an error while loading. Please reload this page.
feat(schema): add max_tokens field to AgentDef for per-agent override - #471
feat(schema): add max_tokens field to AgentDef for per-agent override#471nskun (nskun) wants to merge 6 commits into
Conversation
nskun (nskun)
commented
Aug 20, 2026
@microsoft-github-policy-service agree |
ea17315 to
cd9a25bCompareCodecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@## main #471 +/- ##
=======================================
Coverage ? 91.90% =======================================
Files ? 144 Lines ? 23272 Branches ? 0 =======================================
Hits ? 21389 Misses ? 1883 Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Jason Robert (jrob5756)
left a comment
There was a problem hiding this comment.
The schema work here is solid. The field sits in the right place next to max_agent_iterations and session_key, the bounds agree with both RuntimeConfig.max_tokens and the runtime assertion at claude.py:365, and using is not None instead of copying the truthiness check on the line above was the right call. Ruff, ty and the full test_config suite are all green.
My concerns are all about what happens once the schema has accepted the value.
agent_builder.py:307 is the only place in the repo that reads this field, and it is reachable from claude.py alone. copilot, which is the default provider, has no max_tokens reference anywhere. hermes:283 reads only the runtime value. aca's AcaAgentPayload is extra="forbid" and does not list the field, so it cannot cross the wire. On four of five providers this validates clean and then quietly does nothing. The two fields immediately above it in AgentDef are plumbed into all five (copilot:1296,1301, hermes:247,252, claude_agent_sdk:883,891, aca:763,764), so the surrounding convention sets a different expectation.
On the one provider that does read it, _coerce_for_thinking can raise the value without saying so. max_tokens=1 with reasoning.effort=max comes out as 64000, and where the clamp-down branch at agent_builder.py:244 logs at INFO, the up-bump at :240-242 logs nothing at any level. A small per-agent cap on an agent that also sets reasoning is the most likely way anyone reaches for this field, so in its most probable configuration it does the reverse of what it advertises.
There is also an asymmetry with claude-agent-sdk. factory.py:195 raises ProviderError for runtime.max_tokens, and the comment above it says silently dropping the value "would quietly violate user intent". The per-agent field never reaches that check, so moving one line two levels deeper in the same YAML file gets past the guard. That also makes docs/providers/comparison.md:170 inaccurate, since it currently states both settings are rejected at the factory.
A ProviderCapabilities flag plus a check in _check_agent_capabilities would settle the first and third points together. session_key, max_session_seconds, working_dir and skills all use that shape already, and it would catch claude-agent-sdk at validate time rather than partway into a run.
Smaller things: the restriction lists in docs/workflow-syntax.md (lines 936, 1022, 1162) enumerate every rejected field and now omit this one, and the same lists appear in plugins/conductor/skills/conductor/references/authoring.md and yaml-schema.md, which ship to agents as skill content. There is no CHANGELOG.md entry under Unreleased.
Nothing above is a criticism of the mechanics, which are clean. Happy to look again once the provider question is settled.
| max_agent_iterations: 200 instead of using the default limit. | ||
| """ | ||
| max_tokens: int | None = Field(None, ge=1, le=200000) |
There was a problem hiding this comment.
This validates clean on copilot, hermes, aca and claude-agent-sdk, and then has no effect on any of them. agent_builder.py:307 is the only read in the repo and it only runs under claude.
The two fields directly above this one are honoured by every provider (copilot:1296,1301, hermes:247,252, claude_agent_sdk:883,891, aca:763,764), so anyone reading AgentDef would expect the same treatment here.
Two ways to close it. Either wire up the remaining four providers, or declare the support explicitly and let validation refuse the rest. The second is what session_key and max_session_seconds already do:
# providers/capabilities.pymax_tokens: bool=False"""``True`` when the provider applies a per-agent ``max_tokens`` output cap.``False`` means the value would be silently ignored, so workflows that set itfail validation instead."""# config/validator.py, inside _check_agent_capabilitiesifagent.max_tokensisnotNoneandnotcaps.max_tokens:
errors.append(
f"Agent '{agent.name}' sets max_tokens={agent.max_tokens!r} but provider "f"'{provider_name}' does not apply per-agent output token caps "f"(capabilities.max_tokens=False). Remove it, use runtime.max_tokens where "f"the provider honours it, or override the agent to a provider that does."
)Then set max_tokens=True on ClaudeProvider and leave the default everywhere else. That also picks up claude-agent-sdk, which today refuses runtime.max_tokens at factory.py:195 but lets this one through.
| Overrides the workflow-level runtime.max_tokens for this agent. | ||
| Only applies to provider-backed agents (not script or human_gate). |
There was a problem hiding this comment.
human_gate accepts max_tokens as things stand. I checked against a valid gate fixture and it takes the field without complaint, even though it rejects reasoning and session_key. questions accepts it too, which is odder still given it rejects model with the reason "no provider is invoked".
max_session_seconds and max_agent_iterations have the same hole, so this is inherited rather than introduced. But this is the line where max_tokens's contract gets written down, and it currently names the one type the code does not cover.
| Overridestheworkflow-levelruntime.max_tokensforthisagent. | |
| Onlyappliestoprovider-backedagents (notscriptorhuman_gate). | |
| Overridestheworkflow-levelruntime.max_tokensforthisagent. Controls | |
| responselength, notthecontextwindow (thatbudgetiscontext.max_tokens). | |
| Rejectedonscript, workflow, wait, set, andterminatesteps. |
| if self.max_tokens is not None: | ||
| raise ValueError("script agents cannot have 'max_tokens'") |
There was a problem hiding this comment.
This block is repeated verbatim five times, and the copying is what let human_gate and questions slip through.
validate_agent_type already has a standalone-guard idiom for this, at lines 1929, 1944 and 1956. The comment on the stdin one states the reasoning outright: being a standalone guard rather than a per-branch check, it also covers the types that have no branch of their own.
The same shape replaces all five and closes the gap:
ifself.typenotin (None, "agent") andself.max_tokensisnotNone:
raiseValueError(
f"'{self.type}' agents cannot have 'max_tokens' ""(only provider-backed agents support this field)"
)It is a net reduction in lines, and nothing existing can break: extra="forbid" meant no workflow could carry the field at all before this PR, so the new rejection is strictly tighter than nothing. validator.py:1733 already defines _LLM_AGENT_TYPES = frozenset({None, "agent"}) if you would rather have one source of truth, and schema.py does not import validator.py, so there is no cycle.
Your five existing rejection tests still pass against this, since they only match on the field name.
| assert agent.max_session_seconds == 90.0 | ||
| class TestAgentDefMaxTokens: |
There was a problem hiding this comment.
All eleven tests here exercise AgentDef.__init__. None of them reach the line that consumes the field, which was dead code until this PR. Delete agent_builder.py:307-308 and this suite still goes green.
tests/test_integration/test_parameter_flow_verification.py was written for this exact worry (its module docstring names it) and already covers the workflow-level value. The cheapest addition is two synchronous tests in test_pydantic_ai_agent_builder.py::TestSamplingSettings:
deftest_agent_max_tokens_overrides_workflow_default(self) ->None:
"""A per-agent max_tokens must win over the workflow-level default."""agent_def=AgentDef(name="sampler", max_tokens=1000)
pydantic_agent=build_agent(
agent_def, system_prompt="", rendered_prompt="", default_max_tokens=4096
)
assertpydantic_agent.model_settings["max_tokens"] ==1000deftest_workflow_default_used_when_agent_max_tokens_unset(self) ->None:
"""With no per-agent override the workflow default still applies."""agent_def=AgentDef(name="sampler")
pydantic_agent=build_agent(
agent_def, system_prompt="", rendered_prompt="", default_max_tokens=4096
)
assertpydantic_agent.model_settings["max_tokens"] ==4096I ran both against this branch and they pass.
One gap in the range coverage too: 0, -100 and 200001 are all tested, but the accepted endpoints 1 and 200000 are not, so swapping ge/le for gt/lt would go unnoticed. TestAgentDefMaxSessionSeconds has test_minimum_boundary for the same reason.
Worth adding a case for reasoning as well. max_tokens=1000 with reasoning.effort=low currently produces model_settings["max_tokens"] == 6144, which may be correct for the Anthropic API but is worth pinning so it cannot drift unnoticed.
| """Test that script agents cannot have max_tokens.""" | ||
| with pytest.raises(ValidationError) as exc_info: | ||
| AgentDef(name="s", type="script", command="echo hi", max_tokens=8192) | ||
| assert "max_tokens" in str(exc_info.value) |
There was a problem hiding this comment.
This assertion cannot fail for the reason it looks like it is checking. Pydantic v2 echoes the input dict into the error message, so "max_tokens" in str(exc_info.value) is true for any ValidationError raised on this input. Drop the command kwarg and it still passes, on a completely unrelated "script agents require 'command'" error.
pytest.raises is carrying the test on its own here. The range tests further up (line 687) already match on real message text, so this is inconsistent within the same class rather than a house style:
| assert"max_tokens"instr(exc_info.value) | |
| assert"script agents cannot have 'max_tokens'"instr(exc_info.value) |
Same applies to the workflow, wait, set and terminate cases below.
nskun (nskun)
commented
Aug 25, 2026
Thank you so much for the incredibly thorough review — I really appreciate the time and care you put into it. |
nskun (nskun)
commented
Aug 27, 2026
Thank you again for the detailed review. I’ve pushed an initial update addressing the concrete issues you identified. My original intent was to support the stable-tier providers, Copilot and Claude. Claude can apply a per-agent output-token setting through Pydantic AI, but the current Copilot SDK does not expose an equivalent setting that Conductor can pass through. The initial update therefore followed the second approach you suggested: it enabled per-agent The current update includes:
The directly related test selection passes ( This is an interim update. I’m still working through the remaining changes and will post a follow-up when the PR is ready for another review. |
Summary
Add max_tokens to AgentDef, enabling per-agent output tokens limit configuration.
Background
agent_builder.py already has the fallback logic, but the field was missing from the schema:
Since AgentDef uses extra="forbid", specifying max_tokens in YAML caused a validation error, and getattr always returned None.
Changes
・Add max_tokens: int | None = Field(None, ge=1, le=200000) to AgentDef (matches RuntimeConfig.max_tokens constraints)
・Add forbidden-field checks in validate_agent_type for non-LLM agent types: script, workflow, wait, set, terminate
・Add TestAgentDefMaxTokens(11 tests)
Usage
When omitted inherits runtime.max_tokens as before. No logic change needed in agent_builder.py
Closes#470