Uh oh!
There was an error while loading. Please reload this page.
Fix edge executor to support handling execute callback workload - #67679
Conversation
jscheffl
left a comment
There was a problem hiding this comment.
Looks good, let's have CI green and another CoPilot review
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Pull request overview
This PR extends the Edge3 provider’s EdgeExecutor/worker path to support Airflow 3.3+ ExecuteCallback workloads (in addition to ExecuteTask), including API schema updates and unit tests to validate serialization, queuing, and command parsing.
Changes:
- Add
ExecuteTypeBody/callback identification helpers and route/worker typing updates to accept callback workloads. - Extend EdgeExecutor queuing + worker API command parsing to recognize and deserialize
ExecuteCallback. - Add/extend unit tests and update EdgeExecutor documentation + worker API OpenAPI schema to reflect the new workload shape.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| providers/edge3/src/airflow/providers/edge3/executors/edge_executor.py | Queue ExecuteCallback workloads into EdgeJobModel alongside tasks. |
| providers/edge3/src/airflow/providers/edge3/cli/worker.py | Widen worker execution plumbing to accept a workload union (ExecuteTypeBody). |
| providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.py | Add callback-aware command parsing when fetching jobs. |
| providers/edge3/src/airflow/providers/edge3/worker_api/datamodels.py | Change EdgeJobFetched.command type to a workload union. |
| providers/edge3/src/airflow/providers/edge3/models/types.py | Introduce ExecuteTypeBody, callback tagging, and is_callback_execute(). |
| providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml | Update OpenAPI schema to include ExecuteCallback and discriminated union for command. |
| providers/edge3/docs/edge_executor.rst | Document callback workload support and identifier conventions. |
| providers/edge3/tests/unit/edge3/worker_api/routes/test_jobs.py | Add tests for callback-aware parse_command(). |
| providers/edge3/tests/unit/edge3/models/test_types.py | Add tests for ExecuteTypeBody validation and callback detection helper. |
| providers/edge3/tests/unit/edge3/executors/test_edge_executor.py | Add tests for queuing callback workloads into EdgeJobModel. |
| providers/edge3/tests/unit/edge3/cli/test_worker.py | Add serialization roundtrip coverage for callback jobs (EdgeJobFetched). |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
wjddn279
commented
Jun 1, 2026
Uh oh!
There was an error while loading. Please reload this page.
- amazon: move S3 transfer-operators fix (apache#67378, vincbeck) and EcsRunTaskOperator log-level fix (apache#67180, jscheffl) from Features to Bug Fixes; beautify the EksPodOperator entry (apache#65335). - google: beautify the idle/auto-stop TTL entry (apache#65653) — drop conventional-commit prefix. - openlineage: beautify the ProcessPoolExecutor self-heal entry (apache#67400). - edge3: reword the 3.8.0 note — the provider still supports Airflow 3.0+, only the execute-callback feature (apache#67679) needs 3.3+ (jscheffl); move the Swagger API docs entry (apache#67390) to Doc-only; beautify the team_name clarification (apache#66718). - apache/drill: move the flit.sdist housekeeping entry (apache#65861) to the excluded block to match the kafka convention (jscheffl).
* Prepare provider documentation 2026-06-02 * Address review feedback on provider changelogs - amazon: move S3 transfer-operators fix (#67378, vincbeck) and EcsRunTaskOperator log-level fix (#67180, jscheffl) from Features to Bug Fixes; beautify the EksPodOperator entry (#65335). - google: beautify the idle/auto-stop TTL entry (#65653) — drop conventional-commit prefix. - openlineage: beautify the ProcessPoolExecutor self-heal entry (#67400). - edge3: reword the 3.8.0 note — the provider still supports Airflow 3.0+, only the execute-callback feature (#67679) needs 3.3+ (jscheffl); move the Swagger API docs entry (#67390) to Doc-only; beautify the team_name clarification (#66718). - apache/drill: move the flit.sdist housekeeping entry (#65861) to the excluded block to match the kafka convention (jscheffl). * Make apache/drill 3.3.3 a doc-only release The only non-excluded drill change in this wave was the flit.sdist housekeeping entry, which jscheffl asked to exclude — leaving an empty changelog. Promote the DAG-to-Dag wording change (#66153) into a Doc-only section so drill 3.3.3 ships as a legitimate doc-only release instead of an empty one. * Fold post-prep provider commits into changelogs Incremental-update pass for commits that landed on main after this wave was prepared: - amazon: Propogate verify/botocore_config in redshift cluster triggers (#67876) → Features (mirrors the batch-triggers entry #67508). - databricks: Lock in workflow depends_on parent-key behavior (#66681) → Bug Fixes. - edge3: Fix Edge worker fork mode reporting supervisor failures as success (#67887) → Bug Fixes. - google: Migrate Stackdriver logging config to RemoteLogIO pattern (#66513) → Misc.
reopen: #63498
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.