Skip to content

[KubernetesExecutor] Don't skip succeeded status - #42624

Merged
RNHTTR merged 2 commits into
apache:mainfrom
vlieven:kubernetes_dont_skip_succeeded_events
Oct 11, 2024
Merged

[KubernetesExecutor] Don't skip succeeded status#42624
RNHTTR merged 2 commits into
apache:mainfrom
vlieven:kubernetes_dont_skip_succeeded_events

Conversation

@vlieven

@vlievenvlieven commented Oct 1, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

This PR implements the fix proposed in #41436 and guards against duplicated success messages by having _change_state() in the KubernetesExecutor only emit to the event_buffer if the task is in self.running,
as proposed by @jedcunningham in #40516.

closes: #40516, closes#41436

@boring-cyborgboring-cyborgBot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Oct 1, 2024
@boring-cyborg

boring-cyborgBot commented Oct 1, 2024

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@vlievenvlieven mentioned this pull request Oct 1, 2024
Closed
2 tasks
jedcunningham
jedcunningham reviewed Oct 4, 2024
View reviewed changes

@jedcunninghamjedcunningham left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vlieven.

Can you fix (or probably remove) the broken test, and add coverage for the early return?

@vlieven

vlieven commented Oct 7, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Sorry about that! I removed the broken test, and added a new one to check that no event is emitted when the task key is not present in running.

@RNHTTR

RNHTTR commented Oct 9, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

@vlieven can you please resolve the merge conflicts? I believe this is caused by providers being moved to the root of the project. We'll probably also want a backport PR.

@ashbvlieven
Split providers out of the main "airflow/" tree into a UV workspace p…
d5b6d44
…roject (#42505)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
@vlieven

vlieven commented Oct 10, 2024

Copy link
Copy Markdown
ContributorAuthor

Hi @RNHTTR, I rebased the changes, should be good again now.
Could you point me to what is expected as backport PR?

@RNHTTR

RNHTTR commented Oct 11, 2024

Copy link
Copy Markdown
Contributor

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

@eladkal

eladkal commented Oct 11, 2024

Copy link
Copy Markdown
Contributor

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

Why? This is provider only change. PR is against main branch and that is how it should be.

@RNHTTR

RNHTTR commented Oct 11, 2024

Copy link
Copy Markdown
Contributor

Hi @RNHTTR, I rebased the changes, should be good again now. Could you point me to what is expected as backport PR?

I believe you just need to create a new branch against v2-10-test (not main), run git cherry-pick <commit> (you can find the commit ID in the PR where it says “merge commit into main”), and creating a new PR with thie new branch.

Why? This is provider only change. PR is against main branch and that is how it should be.

Oh, duh, sorry about that.

@RNHTTR
RNHTTR merged commit a052d9e into apache:mainOct 11, 2024
@boring-cyborg

boring-cyborgBot commented Oct 11, 2024

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

kunaljubce pushed a commit to kunaljubce/airflow that referenced this pull request Oct 13, 2024
@vlieven@ashb
@RNHTTR@kunaljubce
Split providers out of the main "airflow/" tree into a UV workspace p…
97e14d1
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
pavansharma36 pushed a commit to pavansharma36/airflow that referenced this pull request Oct 14, 2024
@vlieven@ashb
@RNHTTR@pavansharma36
Split providers out of the main "airflow/" tree into a UV workspace p…
3bbfaba
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
R7L208 pushed a commit to R7L208/airflow that referenced this pull request Oct 17, 2024
@vlieven@ashb
@RNHTTRLorin
Split providers out of the main "airflow/" tree into a UV workspace p…
a9ac157
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
@vlieven@ashb
@RNHTTR@harjeevanmaan
Split providers out of the main "airflow/" tree into a UV workspace p…
01553fd
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
@vlieven@ashb
@RNHTTR@PaulKobow7536
Split providers out of the main "airflow/" tree into a UV workspace p…
df3d685
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
@vlieven@ashb
@RNHTTR@ellisms
Split providers out of the main "airflow/" tree into a UV workspace p…
c59b6f1
…roject (apache#42505) (apache#42624)
This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.
In addition to the straight file rename the other changes I had to make here
are:
- Some mypy/typing fixes.
Mypy can be fragile about what it picks up when, so maybe some of those
changes were caused by that. But the typing changes aren't large.
- Improve typing in common.sql type stub
Again, likely a mypy file oddity, but the types should be safe
- Removed the `check-providers-init-file-missing` check
This isn't needed now that airflow/providers shouldn't exist at all in the
main tree.
- Create a "dev.tests_common" package that contains helper files and common
pytest fixtures
Since the provider tests are no longer under tests/ they don't automatically
share the fixtures from the parent `tests/conftest.py` so they needed
extracted.
Ditto for `tests.test_utils` -- they can't be easily imported in provider
tests anymore, so they are moved to a more explicit shared location.
In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>
@vlieven
vlieven deleted the kubernetes_dont_skip_succeeded_events branch November 3, 2025 08:04
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RNHTTRRNHTTRRNHTTR approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala

Assignees

No one assigned

Labels

area:providersprovider:cncf-kubernetesKubernetes (k8s) provider related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Airflow scheduler misses success events of worker pods on kubernetes Sensors stuck in queued after being spot interrupted

5 participants

@vlieven@RNHTTR@eladkal@jedcunningham@ashb