Skip to content

Fix edge executor to support handling execute callback workload - #67679

Merged
jscheffl merged 3 commits into
apache:mainfrom
wjddn279:add-edge-executor-callback-workload
Jun 1, 2026
Merged

Fix edge executor to support handling execute callback workload#67679
jscheffl merged 3 commits into
apache:mainfrom
wjddn279:add-edge-executor-callback-workload

Conversation

@wjddn279

Copy link
Copy Markdown
Contributor

reopen: #63498


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, let's have CI green and another CoPilot review

Comment threadproviders/edge3/src/airflow/providers/edge3/models/types.py

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the Edge3 provider’s EdgeExecutor/worker path to support Airflow 3.3+ ExecuteCallback workloads (in addition to ExecuteTask), including API schema updates and unit tests to validate serialization, queuing, and command parsing.

Changes:

  • Add ExecuteTypeBody/callback identification helpers and route/worker typing updates to accept callback workloads.
  • Extend EdgeExecutor queuing + worker API command parsing to recognize and deserialize ExecuteCallback.
  • Add/extend unit tests and update EdgeExecutor documentation + worker API OpenAPI schema to reflect the new workload shape.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file
FileDescription
providers/edge3/src/airflow/providers/edge3/executors/edge_executor.pyQueue ExecuteCallback workloads into EdgeJobModel alongside tasks.
providers/edge3/src/airflow/providers/edge3/cli/worker.pyWiden worker execution plumbing to accept a workload union (ExecuteTypeBody).
providers/edge3/src/airflow/providers/edge3/worker_api/routes/jobs.pyAdd callback-aware command parsing when fetching jobs.
providers/edge3/src/airflow/providers/edge3/worker_api/datamodels.pyChange EdgeJobFetched.command type to a workload union.
providers/edge3/src/airflow/providers/edge3/models/types.pyIntroduce ExecuteTypeBody, callback tagging, and is_callback_execute().
providers/edge3/src/airflow/providers/edge3/worker_api/v2-edge-generated.yamlUpdate OpenAPI schema to include ExecuteCallback and discriminated union for command.
providers/edge3/docs/edge_executor.rstDocument callback workload support and identifier conventions.
providers/edge3/tests/unit/edge3/worker_api/routes/test_jobs.pyAdd tests for callback-aware parse_command().
providers/edge3/tests/unit/edge3/models/test_types.pyAdd tests for ExecuteTypeBody validation and callback detection helper.
providers/edge3/tests/unit/edge3/executors/test_edge_executor.pyAdd tests for queuing callback workloads into EdgeJobModel.
providers/edge3/tests/unit/edge3/cli/test_worker.pyAdd serialization roundtrip coverage for callback jobs (EdgeJobFetched).

Comment threadproviders/edge3/src/airflow/providers/edge3/cli/worker.py
Comment threadproviders/edge3/src/airflow/providers/edge3/cli/worker.py
Comment threadproviders/edge3/src/airflow/providers/edge3/cli/worker.py Outdated
Comment threadproviders/edge3/docs/edge_executor.rst Outdated
@wjddn279

Copy link
Copy Markdown
ContributorAuthor

@jscheffl

Hi jens! Thanks for review!
I've tested this pr with airflow-core (main) and it works well.
There is issue the callback state is stuck in QUEDED state but it will be fixed in #66141

@potiukpotiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 1, 2026

@jscheffljscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks great!

@jscheffl
jscheffl merged commit 8607e94 into apache:mainJun 1, 2026
97 checks passed
@ferruzziferruzzi mentioned this pull request Jun 2, 2026
2 tasks
potiuk added a commit to potiuk/airflow that referenced this pull request Jun 2, 2026
- amazon: move S3 transfer-operators fix (apache#67378, vincbeck) and EcsRunTaskOperator
log-level fix (apache#67180, jscheffl) from Features to Bug Fixes; beautify the
EksPodOperator entry (apache#65335).
- google: beautify the idle/auto-stop TTL entry (apache#65653) — drop conventional-commit prefix.
- openlineage: beautify the ProcessPoolExecutor self-heal entry (apache#67400).
- edge3: reword the 3.8.0 note — the provider still supports Airflow 3.0+, only the
execute-callback feature (apache#67679) needs 3.3+ (jscheffl); move the Swagger API docs
entry (apache#67390) to Doc-only; beautify the team_name clarification (apache#66718).
- apache/drill: move the flit.sdist housekeeping entry (apache#65861) to the excluded block
to match the kafka convention (jscheffl).
potiuk added a commit that referenced this pull request Jun 3, 2026
* Prepare provider documentation 2026-06-02
* Address review feedback on provider changelogs
- amazon: move S3 transfer-operators fix (#67378, vincbeck) and EcsRunTaskOperator
log-level fix (#67180, jscheffl) from Features to Bug Fixes; beautify the
EksPodOperator entry (#65335).
- google: beautify the idle/auto-stop TTL entry (#65653) — drop conventional-commit prefix.
- openlineage: beautify the ProcessPoolExecutor self-heal entry (#67400).
- edge3: reword the 3.8.0 note — the provider still supports Airflow 3.0+, only the
execute-callback feature (#67679) needs 3.3+ (jscheffl); move the Swagger API docs
entry (#67390) to Doc-only; beautify the team_name clarification (#66718).
- apache/drill: move the flit.sdist housekeeping entry (#65861) to the excluded block
to match the kafka convention (jscheffl).
* Make apache/drill 3.3.3 a doc-only release
The only non-excluded drill change in this wave was the flit.sdist housekeeping
entry, which jscheffl asked to exclude — leaving an empty changelog. Promote the
DAG-to-Dag wording change (#66153) into a Doc-only section so drill 3.3.3 ships as
a legitimate doc-only release instead of an empty one.
* Fold post-prep provider commits into changelogs
Incremental-update pass for commits that landed on main after this wave was prepared:
- amazon: Propogate verify/botocore_config in redshift cluster triggers (#67876) → Features
(mirrors the batch-triggers entry #67508).
- databricks: Lock in workflow depends_on parent-key behavior (#66681) → Bug Fixes.
- edge3: Fix Edge worker fork mode reporting supervisor failures as success (#67887) → Bug Fixes.
- google: Migrate Stackdriver logging config to RemoteLogIO pattern (#66513) → Misc.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providerskind:documentationprovider:edgeEdge Executor / Worker (AIP-69) / edge3ready for maintainer reviewSet after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@wjddn279@jscheffl@potiuk