New Kubernetes Kueue Operators

[moiseenkov]

• implement new operators: KubernetesInstallKueueOperator, KubernetesStartKueueJobOperator in cncf-kubernetes provider
• refactor GKE operators in google provider in order to reduce complexity and code duplication

[moiseenkov]

@eladkal , @potiuk,

This PR contains changes for both Google and cncf-kubernetes providers, and the Google provider depends on changes in cncf-kuberneets. For this reason I had to bump cncf-kubernetes version in Google provider's dependencies and it now breaks CI because it can't install cncf-kubernetes provider that wasn't released yet.
Therefore, should I split this PR, so we could release these changes gradually (cncf first, and google later) or we are OK with merging it all together now.

[potiuk]

@eladkal , @potiuk,

This PR contains changes for both Google and cncf-kubernetes providers, and the Google provider depends on changes in cncf-kuberneets. For this reason I had to bump cncf-kubernetes version in Google provider's dependencies and it now breaks CI because it can't install cncf-kubernetes provider that wasn't released yet. Therefore, should I split this PR, so we could release these changes gradually (cncf first, and google later) or we are OK with merging it all together now.

This is one option yes. There is another option. You can add the cncf.kubernetes provider to the list of "chicken-egg" providers in src/airflow_breeze/global_constants.py -> similarly to what I've done in #44714 (about to be merged).

[potiuk]

Also updated documentation about it here #44720

[eladkal]

@eladkal , @potiuk,
This PR contains changes for both Google and cncf-kubernetes providers, and the Google provider depends on changes in cncf-kuberneets. For this reason I had to bump cncf-kubernetes version in Google provider's dependencies and it now breaks CI because it can't install cncf-kubernetes provider that wasn't released yet. Therefore, should I split this PR, so we could release these changes gradually (cncf first, and google later) or we are OK with merging it all together now.

This is one option yes. There is another option. You can add the cncf.kubernetes provider to the list of "chicken-egg" providers in src/airflow_breeze/global_constants.py -> similarly to what I've done in #44714 (about to be merged).

Why not just using AirflowOptionalProviderFeatureException?

[moiseenkov]

This is one option yes. There is another option. You can add the cncf.kubernetes provider to the list of "chicken-egg" providers in src/airflow_breeze/global_constants.py -> similarly to what I've done in #44714 (about to be merged).

@potiuk , thank you for the suggestion! I followed this idea. But it seems to me that the new version of Kubernetes provider should be announced in provider.yaml anyway... I can update my PR once it happens.

Why not just using AirflowOptionalProviderFeatureException?

I've also considered this, but unfortunately provided changes aren't optional. For example the GKEStartKueueInsideClusterOperator now inherits from KubernetesInstallKueueOperator which was introduced in this PR. At the same time the GKEStartKueueInsideClusterOperator is a bit older and will be broken if the Kubernetes provider isn't upgraded. Thus I had to bump the cncf-kubernetes provider version in google provider's dependencies.

[potiuk]

Tests failing - breeze (easier to fix) and compatibility tests with airflow 2.8 - 2.9 (a bit more difficult). The way how to run compatibilty test is described in detail in https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#compatibility-provider-unit-tests-against-older-airflow-releases - with some examples how to deal with the tests.

[moiseenkov]

Hi @potiuk ,
The CI is finally green. Thank you very much for your guidance!
The PR is ready for review and merging hopefully.

[potiuk]

Cool. The good things that all deprecations have been removed already in the previous wave - so we will not have another bump in k8s provider MAJOR version, so this one is good to go.