Uh oh!
There was an error while loading. Please reload this page.
Fix output type loss in AgentOperator's human-in-the-loop review - #70132
Fix output type loss in AgentOperator's human-in-the-loop review#70132ColtenOuO wants to merge 3 commits into
Conversation
dec8555 to
5c61485CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
a632d04 to
2adcc5aCompareColtenOuO
commented
Aug 6, 2026
Thanks for reviewing, I have addressed all of the comments. If there still have anything need to improve or change, feel free to let me know >< |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
2adcc5a to
3489368Compare…L rehydrate rehydrate_pydantic_output() decided whether output_type had a usable pydantic schema by checking whether TypeAdapter(output_type) raised. A pydantic-ai output function does not raise there -- TypeAdapter builds a schema for its *arguments*, not its return value -- so validate_json() on the reviewer-approved string re-invoked the function with reviewer-controlled input, after the human had already approved the original output. The same check also left ToolOutput/ NativeOutput/PromptedOutput markers unrecognized, so a marker wrapping a single type fell back to a bare dict instead of validating into a real instance. Unwrap the three marker types first, then gate on isinstance(x, type) instead of "did TypeAdapter raise" -- the only case that check can't tell apart from a real, safe-to-validate type. Also logs a warning when dump_output_to_json()'s serialize side falls back to str(output), since that fallback used to be a hard failure and now degrades silently. Addresses review feedback on apache#70132.
ColtenOuO
commented
Aug 11, 2026
Thanks for catching this. I reproduced it myself and can confirm the output function re-invocation issue is real. The latest commit addresses everything from this round:
Ready for another pass whenever you have time, thanks again for the thorough review! |
…L rehydrate rehydrate_pydantic_output() decided whether output_type had a usable pydantic schema by checking whether TypeAdapter(output_type) raised. A pydantic-ai output function does not raise there -- TypeAdapter builds a schema for its *arguments*, not its return value -- so validate_json() on the reviewer-approved string re-invoked the function with reviewer-controlled input, after the human had already approved the original output. The same check also left ToolOutput/ NativeOutput/PromptedOutput markers unrecognized, so a marker wrapping a single type fell back to a bare dict instead of validating into a real instance. Unwrap the three marker types first, then gate on isinstance(x, type) instead of "did TypeAdapter raise" -- the only case that check can't tell apart from a real, safe-to-validate type. Also logs a warning when dump_output_to_json()'s serialize side falls back to str(output), since that fallback used to be a hard failure and now degrades silently. Addresses review feedback on apache#70132.
3489368 to
051a7c3CompareUh oh!
There was an error while loading. Please reload this page.
ColtenOuO
commented
Aug 18, 2026
Thanks @aaron-y-chen's review! I've addressed the problem in lastest commit, ready for the next round review! |
fc823eb to
080aafcCompareAdd dump_output_to_json() to utils/output_type.py, alongside rehydrate_pydantic_output(), so both directions of the review round-trip live in one module. Route all three serialization sites through it: HITLReviewMixin._to_string, AgentOperator.regenerate_with_feedback and LLMApprovalMixin.defer_for_approval. Serialize non-str, non-BaseModel output with TypeAdapter(...).dump_json() rather than str(output), which produced a Python repr instead of JSON, and fall back to str(output) when the value's type has no pydantic schema. Split schema build from validation in rehydrate_pydantic_output(). An output_type that TypeAdapter cannot build a schema for now falls back to json.loads() instead of raising PydanticSchemaGenerationError or AttributeError, neither of which the previous except clause caught. Use rehydrate_pydantic_output() in AgentOperator.execute()'s HITL branch in place of the bespoke json.loads()/except fallback. With the default output_type=str the approved string is no longer parsed as JSON. Add tests for both schema-build guards, for dump_output_to_json(), and for output_type=str with a JSON-parseable approved string, and fold the four TestToString cases into a single delegation test.
…L rehydrate rehydrate_pydantic_output() decided whether output_type had a usable pydantic schema by checking whether TypeAdapter(output_type) raised. A pydantic-ai output function does not raise there -- TypeAdapter builds a schema for its *arguments*, not its return value -- so validate_json() on the reviewer-approved string re-invoked the function with reviewer-controlled input, after the human had already approved the original output. The same check also left ToolOutput/ NativeOutput/PromptedOutput markers unrecognized, so a marker wrapping a single type fell back to a bare dict instead of validating into a real instance. Unwrap the three marker types first, then gate on isinstance(x, type) instead of "did TypeAdapter raise" -- the only case that check can't tell apart from a real, safe-to-validate type. Also logs a warning when dump_output_to_json()'s serialize side falls back to str(output), since that fallback used to be a hard failure and now degrades silently. Addresses review feedback on apache#70132.
rehydrate_pydantic_output gated the TypeAdapter path on isinstance(unwrapped, type), which is False for a generic alias like list[A]. That routed list[A] into the plain json.loads fallback meant for output functions and [A, B] union lists, silently turning a reviewer-approved list of A instances back into a list of plain dicts. Gate on typing.get_origin(...) is not None as well so generic aliases still reach TypeAdapter, while output functions (which are not generic aliases) keep falling back to JSON instead of being re-invoked with reviewer-controlled input. serialize_output's dump step also moves from an isinstance(rehydrated, BaseModel) check to adapter.dump_python(rehydrated, mode="python"), since a rehydrated list[A] is a list, not a BaseModel, and needs the same dict-shape dump its elements got before this fix.
080aafc to
d0c3550Compare
Summary
AgentOperator's human-in-the-loop review path (enable_hitl_review=True) loses the original output type for anyoutput_typethat is neitherstrnor a PydanticBaseModel(e.g.list[str],bool,dict).The output is round-tripped through a string while it's shown to a human reviewer:
HITLReviewMixin._to_stringserializes it before review, andAgentOperator.execute()deserializes it back intooutput_typeafter approval. The serialization side usedstr(output), which produces a Python repr (e.g."['tag-a', 'tag-b']"), not valid JSON.The deserialization side then tried
json.loads()on that repr, which always raised, and silently fell back to returning the raw repr string instead of the original list/bool/dict — a type change downstream tasks don't expect.regenerate_with_feedback()(used when a reviewer requests changes) had the identicalstr(output)bug.This is the same defect class already fixed for the
require_approval=Truepath in #70075, which switched toTypeAdapter(...).dump_json(...). That fix wasn't applied to the separateenable_hitl_review=Truepath, so it regressed here.Changes
utils/output_type.py: the existingrehydrate_pydantic_output()and a newdump_output_to_json().dump_output_to_json()replaces the three copies of the serialization blockthat had drifted apart —
HITLReviewMixin._to_string,AgentOperator.regenerate_with_feedbackandLLMApprovalMixin.defer_for_approval.That drift is how the original bug survived Preserve output_type through human approval in LLM operators #70075.
AgentOperator.execute()'s HITL-review branch reusesrehydrate_pydantic_outputinstead of a bespoke
json.loads/exceptfallback, so both review paths behaveconsistently.
Guarding an
output_typepydantic cannot build a schema forAgentOperatorpassesoutput_typestraight through toAgent(...), so pydantic-ai's[A, B]multi-output lists,ToolOutput/NativeOutput/PromptedOutputmarkers andoutput functions all reach these helpers.
TypeAdapterraisesPydanticSchemaGenerationError(aRuntimeError, not aValidationError) orAttributeErroron those, which neither side'sexceptclause caught:rehydrate_pydantic_outputnow separates schema build from validation and fallsback to plain
json.loads. Without this an already-approved output is lost, afterthe model call and after the reviewer has signed off.
dump_output_to_jsonfalls back tostr(output). This one fires before the sessionXCom push, so without it the task dies before the reviewer is ever shown a review
session. It is reachable through output functions, which only warn
("Falling back to unconstrained schema") for a non-schema-able return type and then
run fine.
Behaviour change
With the default
output_type=str, the approved string is no longer parsed as JSON:'42'42'42''{"text": "hi"}'{'text': 'hi'}'{"text": "hi"}''true'True'true''null'None'null'This matches the non-HITL path (which returns
result.outputunparsed) and therequire_approvalpath.stris the defaultoutput_typeandenable_hitl_reviewhas shipped since 0.1.0, so a Dag relying on the implicit parse needs an explicit
output_type(e.g.output_type=int) or its ownjson.loads()downstream.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines