Uh oh!
There was an error while loading. Please reload this page.
Version the worker-bound TaskInstance fields in the execution API schema - #68390
Conversation
The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed.
ashb
commented
Jun 11, 2026
The last two being required on the worker seems like a code smell. If it makes it to the worker it should just executed. That is the scheduler's job, not the workers |
Uh oh!
There was an error while loading. Please reload this page.
…he schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema.
kaxil
commented
Jun 11, 2026
Agreed and fixed in 78a6b82 -- the worker-facing schema now only gains |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
jason810496
left a comment
There was a problem hiding this comment.
Really nice catch, thanks. Do we need full tests before merge Btw?
Uh oh!
There was an error while loading. Please reload this page.
586b49e to
3d48d40CompareThe generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
Uh oh!
There was an error while loading. Please reload this page.
…ema (apache#68390) The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed. * Keep pool_slots and priority_weight executor-side; only queue joins the schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema. The generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
…ema (apache#68390) The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed. * Keep pool_slots and priority_weight executor-side; only queue joins the schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema. The generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
Why
Since #65958 the supervisor routes every task through
get_coordinator_manager().for_queue(ti.queue), so it depends onqueuebeing on the task instance it receives. That field (along withpool_slotsandpriority_weight) lived in aBaseTaskInstanceDTOduplicated between airflow-core and task-sdk (#67174), kept aligned by an AST-comparison prek hook, while theTaskInstanceschema in the versioned execution API spec lacked it.That spec is already the designated home for this vocabulary:
get_extra_schemas()injectsTaskInstance(andBundleInfo, imported fromairflow.executors.workloads) into the OpenAPI schema precisely so client SDKs generate these types from a versioned source. Adding the missing field there gives one source of truth instead of two hand-synced copies.What
queueto the execution APITaskInstanceschema, with anAddTaskInstanceQueueFieldCadwyn version change in the 2026-06-30 bundle, and regenerate the task-sdk datamodels.StartupDetails.ti,supervise_task()and the coordinator interfaces at the generatedTaskInstance.TaskInstanceDTOinto a subclass of the schema model that adds only the executor-side fields:pool_slotsandpriority_weight(read by the executor's priority sort and the edge executor's concurrency accounting, never by the worker), plusexternal_executor_idandexecutor_config(excluded from serialization).execution_time.workloadsmodule and thecheck-task-instance-dto-synchook. A round-trip test (executor DTO JSON validated by the SDK's generated model) covers the contract the hook used to enforce.parent_context_carrierfrom the DTO: it has no Python readers or writers anywhere in core, task-sdk or providers.Design notes
queueis worker-facing: the supervisor routes on it.pool_slotsandpriority_weightare executor concerns and stay on the executor DTO, still serialized in the workload (required, exactly the pre-PR wire shape) so older workers that deserialize the workload with the required-field model keep working.queuegets a default ("default") instead of being required. The executor always sends the real value (ExecuteTask.makevalidates from the ORM task instance, where the column is non-null); the default keeps hand-built instances such as tests anddry_runruns valid, and matches howmap_indexandhostnameare already modelled in this schema.VersionChangeentry is possible.plugins/www/openapi-genandgo-sdk/pkg/edgeapiare not regenerated here: their codegen is not wired into prek and already lags the spec on main (the Go client was last regenerated in Simple sending of return value from Go tasks to XCom #56481), and the affected fields are optional in both.