Skip to content

Add Celery worker workload dispatch regression tests - #65849

Merged
shahar1 merged 9 commits into
apache:mainfrom
leeyspaul:tests/celery-executor-compatibility
May 2, 2026
Merged

Add Celery worker workload dispatch regression tests#65849
shahar1 merged 9 commits into
apache:mainfrom
leeyspaul:tests/celery-executor-compatibility

Conversation

@leeyspaul

@leeyspaulleeyspaul commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Addresses the comment here: #65752 with a focused intent on Celery provider workload dispatch path.

This adds focused Celery executor tests for the worker-side execute_workload path across the Airflow 3.3 compatibility boundary.

Specifically, the tests now cover:

  • Airflow 3.0-3.2: serialized ExecuteTask payloads are decoded and routed to supervise(...)
  • Airflow 3.3+: serialized ExecuteTask payloads are decoded and routed to BaseExecutor.run_workload(...)

A regression in this area was not caught because existing tests did not directly exercise the Celery worker-side deserialization and dispatch path for serialized workload payloads. These tests make that contract explicit for both sides of the version gate. This PR intentionally keeps the fix small and targeted. It does not add a broader compatibility framework or scheduler-side matrix tests. Those would be valuable follow-up coverage, but this change focuses on the worker-side regression path that caused the issue in #65752. I'll follow up with some further discussion on other tests we could add across executors.

Related: #65793


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@leeyspaulleeyspaul changed the title [WIP] tests: Compatibility test for CeleryExecutor[WIP] tests: Celery worker workload dispatch compatibilityApr 27, 2026
@leeyspaulleeyspaul changed the title [WIP] tests: Celery worker workload dispatch compatibility tests: Celery worker workload dispatch compatibilityApr 27, 2026
@leeyspaul
leeyspaul marked this pull request as ready for review April 27, 2026 23:31
@leeyspaul

Copy link
Copy Markdown
ContributorAuthor

@shahar1 a small scoped PR addressing issue #65752 (comment).

I'll follow up on #65793 about other potential tests!

@shahar1shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix static checks

@leeyspaul

Copy link
Copy Markdown
ContributorAuthor

@shahar1 re-ran the merge with main. All checks seem to be passing now!

@leeyspaul
leeyspaul requested a review from shahar1May 1, 2026 22:11
@shahar1shahar1 changed the title tests: Celery worker workload dispatch compatibilityAdd Celery worker workload dispatch regression testsMay 2, 2026
@shahar1
shahar1 merged commit ea34922 into apache:mainMay 2, 2026
89 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@leeyspaul@shahar1