Skip to content

Move all k8S classes to cncf.kubernetes provider - #32767

Merged
potiuk merged 2 commits into
apache:mainfrom
potiuk:move-kubernetes-classes-to-provider
Jul 26, 2023
Merged

Move all k8S classes to cncf.kubernetes provider#32767
potiuk merged 2 commits into
apache:mainfrom
potiuk:move-kubernetes-classes-to-provider

Conversation

@potiuk

@potiukpotiuk commented Jul 22, 2023•
edited
Loading

Copy link
Copy Markdown
Member

This is the big move of all Kubenetes classes to go to provider.

The changes that are implemented in this move:

  • replaced all imports from airflow.kubernetes to cncf.kubernetes
    Swith PEP-563 dynamic import rediretion and deprecation messages
    those messages now support overriding the "replacement" hints
    to make K8s deprecations more accurate
  • pre_7_4_0_compatibility package with classes used by past
    providerrs have been "frozen" and stored in the package with
    import redirections from airflow.kubernetes(with deprecation warnings)
  • kubernetes configuration is moved to kubernetes provider
  • mypy started complaining about conf and set used in configuration.
    so better solution to handle deprecations and hinting conf
    returning AirlfowConfigParsing was added.
  • example_kuberntes_executor uses configuration reading not in
    top level but in execute method
  • PodMutationHookException and PodReconciliationError have
    been moved to cncf.kubernetes provider and they are imported
    from there with fallback to an airflow.exception ones in case
    old provider is used in Airflow 2.7.0
  • k8s methods in task_instance have been deprecated and reolaced
    with functions in "cncf.kubernetes` template_rendering module
    the old way still works but raise deprecaton warnings.
  • added extras with versions for celery and k8s
  • raise AirflowOptionalProviderFeatureException in case there is
    attempt to use CeleryK8sExecutor and cncf.k8s is not installed.
  • added few "new" core utils to k8s (hashlib_wrapper etc)
  • both warnings and errors indicate minimum versions for both cncf.k8s
    and Celery providers.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiukpotiuk added the full tests needed We need to run full set of tests for this PR to merge label Jul 22, 2023
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 3 times, most recently from 0489b31 to 2b5f5c9CompareJuly 22, 2023 12:10
@potiukpotiuk mentioned this pull request Jul 22, 2023
Merged
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 2 times, most recently from d3a18e9 to 4e281b6CompareJuly 22, 2023 18:39
@potiuk
potiuk marked this pull request as ready for review July 22, 2023 18:39
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 2 times, most recently from 83e9b83 to dcc849dCompareJuly 22, 2023 19:44
@potiuk

potiuk commented Jul 22, 2023

Copy link
Copy Markdown
MemberAuthor

Based on #32775 so only last commit counts.

@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch from dcc849d to 5bff371CompareJuly 23, 2023 08:19
@potiukpotiuk changed the title Move kubernetes classes to providerMove all k8S classes to cncf.kubernetes providerJul 23, 2023
@potiuk

potiuk commented Jul 23, 2023

Copy link
Copy Markdown
MemberAuthor

Only last commit counts.

@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 2 times, most recently from 76afae1 to 4e56e6aCompareJuly 23, 2023 08:42
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 3 times, most recently from 17c3838 to e2ed30eCompareJuly 25, 2023 07:24
@potiuk
potiuk requested a review from uranusjr as a code ownerJuly 25, 2023 07:24
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch 3 times, most recently from 205c9a4 to d7d4bbbCompareJuly 25, 2023 14:58
@potiuk

potiuk commented Jul 25, 2023

Copy link
Copy Markdown
MemberAuthor
  • all is now green
  • added detailed explanation in the commit about the scope of changes
  • added pre-commits that are preventing accidental top-level imports of airflow.kubernetes from anywhere but itself (for back-compat) and airflow.providers.cncf.kubernetes from anywhere in the code except the "kubernetes" command
  • added explanation that you need cncf.kubernetes > 7.4.0 or celery > 3.3.0 when your imports using old 'airflow.kubernetes" or "airflow.executors" fail.

jedcunningham
jedcunningham reviewed Jul 25, 2023
View reviewed changes
Comment threadairflow/utils/deprecation_tools.py Outdated
Comment threadairflow/utils/deprecation_tools.py Outdated
Comment threadairflow/utils/deprecation_tools.py Outdated
Comment threadairflow/config_templates/__init__.py Outdated
Comment threadairflow/kubernetes/pre_7_4_0_compatibility/__init__.py Outdated
Comment threadairflow/providers/cncf/kubernetes/provider.yaml Outdated
Comment threaddocs/apache-airflow/core-concepts/executor/celery_kubernetes.rst Outdated
Comment threaddocs/apache-airflow/core-concepts/executor/kubernetes.rst Outdated
Comment threaddocs/apache-airflow/core-concepts/executor/local_kubernetes.rst Outdated
Comment threadtests/cli/commands/test_task_command.py Outdated
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch from d7d4bbb to c486137CompareJuly 25, 2023 19:22
@potiuk

potiuk commented Jul 25, 2023

Copy link
Copy Markdown
MemberAuthor

Applied this round of comments from @jedcunningham . Also added newsfragment

@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch from c486137 to eb6573aCompareJuly 25, 2023 19:48
@potiuk
potiuk requested a review from eladkal as a code ownerJuly 25, 2023 19:48
@potiuk
Move all k8S classes to cncf.kubernetes provider
b7af1c4
This is the big move of all Kubenetes classes to go to provider.
The changes that are implemented in this move:
* replaced all imports from airflow.kubernetes to cncf.kubernetes
Swith PEP-563 dynamic import rediretion and deprecation messages
those messages now support overriding the "replacement" hints
to make K8s deprecations more accurate
* pre_7_4_0_compatibility package with classes used by past
providerrs have been "frozen" and stored in the package with
import redirections from airflow.kubernetes(with deprecation warnings)
* kubernetes configuration is moved to kubernetes provider
* mypy started complaining about conf and set used in configuration.
so better solution to handle deprecations and hinting conf
returning AirlfowConfigParsing was added.
* example_kuberntes_executor uses configuration reading not in
top level but in execute method
* PodMutationHookException and PodReconciliationError have
been moved to cncf.kubernetes provider and they are imported
from there with fallback to an airflow.exception ones in case
old provider is used in Airflow 2.7.0
* k8s methods in task_instance have been deprecated and reolaced
with functions in "cncf.kubernetes` template_rendering module
the old way still works but raise deprecaton warnings.
* added extras with versions for celery and k8s
* raise AirflowOptionalProviderFeatureException in case there is
attempt to use CeleryK8sExecutor and cncf.k8s is not installed.
* added few "new" core utils to k8s (hashlib_wrapper etc)
* both warnings and errors indicate minimum versions for both cncf.k8s
and Celery providers.
@potiuk
potiukforce-pushed the move-kubernetes-classes-to-provider branch from eb6573a to b7af1c4CompareJuly 25, 2023 21:22
jedcunningham
jedcunningham approved these changes Jul 25, 2023
View reviewed changes
Comment threadnewsfragments/32767.significant.rst Outdated
@potiuk@jedcunningham
Update newsfragments/32767.significant.rst
decbead
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
@potiuk
potiuk merged commit e934603 into apache:mainJul 26, 2023
@potiuk
potiuk deleted the move-kubernetes-classes-to-provider branch July 26, 2023 06:25
@potiukpotiuk mentioned this pull request Jul 27, 2023
Merged
potiuk added a commit to potiuk/airflow that referenced this pull request Jul 27, 2023
@potiuk
Add missing deprecated imports after k8s move
425419b
The apache#32767 missed imports of imports for a few kubernetes modules.
This PR adds the missing ones.
jedcunningham pushed a commit that referenced this pull request Jul 27, 2023
@potiuk
Add missing deprecated imports after k8s move (#32891)
46d7125
The #32767 missed imports of imports for a few kubernetes modules.
This PR adds the missing ones.
@eladkaleladkal mentioned this pull request Jul 29, 2023
Closed
69 tasks
@ephraimbuddyephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Aug 2, 2023
@Lee-WLee-W mentioned this pull request Aug 9, 2023
Closed
@potiukpotiuk mentioned this pull request Sep 13, 2023
Merged
potiuk added a commit to potiuk/airflow that referenced this pull request Sep 13, 2023
@potiuk
Move definition of Pod*Exceptions to pod_generator
6c5a4d3
The apache#32767 has moved all k8s classes to cncf.kubernetes provider,
however there was a mistake with location of Pod*Exceptions - rather
than in pod_manager they remained defined in the kubernetes_executor
package - which has the side-effect that trying to import them
in Airflow Pre 2.7 raised the
"You should not use the provider's executors in this version of
Airflow." error.
This change moves the exceptions to the pod_generator package
to fix the problem.
potiuk added a commit that referenced this pull request Sep 13, 2023
@potiuk
Move definition of Pod*Exceptions to pod_generator (#34346)
de92a81
The #32767 has moved all k8s classes to cncf.kubernetes provider,
however there was a mistake with location of Pod*Exceptions - rather
than in pod_manager they remained defined in the kubernetes_executor
package - which has the side-effect that trying to import them
in Airflow Pre 2.7 raised the
"You should not use the provider's executors in this version of
Airflow." error.
This change moves the exceptions to the pod_generator package
to fix the problem.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jedcunninghamjedcunninghamjedcunningham approved these changes
@ashbashbAwaiting requested review from ashbashb is a code owner
@dstandishdstandishAwaiting requested review from dstandish
@kaxilkaxilAwaiting requested review from kaxil
@bolkedebruinbolkedebruinAwaiting requested review from bolkedebruin
@XD-DENGXD-DENGAwaiting requested review from XD-DENG
@o-nikolaso-nikolasAwaiting requested review from o-nikolas
@pierrejeambrunpierrejeambrunAwaiting requested review from pierrejeambrun
@ryanahamiltonryanahamiltonAwaiting requested review from ryanahamilton
@bbovenzibbovenziAwaiting requested review from bbovenzi
@uranusjruranusjrAwaiting requested review from uranusjr
@eladkaleladkalAwaiting requested review from eladkal

Assignees

No one assigned

Labels

area:CLIarea:dev-toolsarea:pluginsarea:Triggererarea:webserverWebserver related Issuesfull tests neededWe need to run full set of tests for this PR to mergeprovider:cncf-kubernetesKubernetes (k8s) provider related issuestype:misc/internalChangelog: Misc changes that should appear in change log

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@potiuk@jedcunningham@ephraimbuddy