Bump minimum Airflow version in providers to Airflow 2.6.0

[eladkal]

Note: suspended providers are not included in bump min airflow version (daskexecutor provider)

[potiuk]

Interesting error with providers. It looks like (Can't remember that) Airflow 2.6.0 had "apache-airflow-providers-hive" dependency instead of "apache-airflow-providers-apache-hive". But it seems that 2.6.1 has it fixed already - so just setting

--use-airflow-version 2.6.1

should fix the issue.

[potiuk]

Ok. Two fixes are needed:

1. rm -rf dist/apache_airflow_providers_cohere*.whl - added here:

   airflow/.github/workflows/ci.yml

   Line 823 in 41f4766

   rm -rf dist/apache_airflow_providers_common_io*.whl

With the comment that is should be removed when we bump to 2.7.1

2. add min_airflow_version = 2.7.1 for cohere provider - seems that importlib limitation in airflow (<5) is incompatible with cohere's (>=6) - and it was only removed in 2.7.1 (Fix dependencies for celery and opentelemetry for Python 3.8 #33579)

So looks ike Cohere provider is only actually installable for Airflow 2.7.1+ - we should add in their provider.yaml

[eladkal]

add min_airflow_version = 2.7.1 for cohere provider - seems that importlib limitation in airflow (<5) is incompatible with cohere's (>=6) - and it was only removed in 2.7.1 (#3357)

I think the PR you linked is wrong?
Why do you claim that cohere provider needs Airflow>=2.7.1?
I don't see anything binding that provider to newer core versions (unlike common.fs)

[eladkal]

The cohere provider can't work with Airflow 2.6 unless using Python 3.9 due to version of importlib

OK thanks to @potiuk we may have workaround for this

8ad4e9b

[potiuk]

Seems like we are havin issue with just released 0.23.1 of Pytest-asyncio, which was supposed to fix bugs in (also just released but 10 hours earlier). I opened an issue pytest-dev/pytest-asyncio#703 to ask what's going on.

[potiuk]

And PR here to temporarily limit asyncio until we know whats going on #36037

[eladkal]

rebased after #36061
lets see if CI passes

[potiuk]

rebased after #36061 lets see if CI passes

Close enough - you need to also remove 2.5.0 from the list :

https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#bump-min-airflow-version-for-providers

Update BASE_PROVIDERS_COMPATIBILITY_CHECKS in src/airflow_breeze/global_constants.py to remove the versions of Airflow that are not applicable any more.

[eladkal]

Close enough - you need to also remove 2.5.0 from the list

Thanks! I new i forgot something

[potiuk]

Looks good :)

[ephraimbuddy]

This should be in 2.8 right @eladkal ?

[eladkal]

This should be in 2.8 right @eladkal ?

If the breeze changes are relevant yes.

[ephraimbuddy]

This should be in 2.8 right @eladkal ?

If the breeze changes are relevant yes.

I think it popped up because of the generated/ folder. I will try cherry-picking it

[potiuk]

I think it's not strictly necessary for 2.8 - I will cherry-pick it eventually anyway (if @ephraimbuddy will not do it ) - but for 2.8.0 (i will eventually make it completely uneccessary when we move to more modern tooling for building airflow package ~ couple of months or so.