Add support for chicken-egg providers to dockerhub release process

[potiuk]

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Testing the workflow changes here: https://github.com/apache/airflow/actions/runs/7059328822

[potiuk]

Next attempt: https://github.com/apache/airflow/actions/runs/7059562344

[potiuk]

Third attempt (getting closer) : https://github.com/apache/airflow/actions/runs/7059678993

[potiuk]

4th attempt - https://github.com/apache/airflow/actions/runs/7060055811

[potiuk]

One step closer: https://github.com/apache/airflow/actions/runs/7060262963 - we need to also support case where airflow + other providers is installed from PyPI but the single provider from local docker-context-files (also added a documentation describing why we are doing it and how)

[potiuk]

OK. The workflow passed here (including image verification step) - (with some intermittent failure for 3.11 which I re-run)

https://github.com/apache/airflow/actions/runs/7060444121

And the 2.8.0b1 images now contain the common.io provider:

docker run -it apache/airflow:2.8.0.b1-python3.8 bash airflow@86cb1fa40068:/opt/airflow$ pip freeze | grep provider
apache-airflow-providers-amazon==8.12.0
apache-airflow-providers-celery==3.4.1
apache-airflow-providers-cncf-kubernetes==7.10.0
apache-airflow-providers-common-io @ file:///docker-context-files/apache_airflow_providers_common_io-1.1.0.dev0-py3-none-any.whl#sha256=26a8e1c51586d6e1d831b01327664943cb69ce9c4f6c589e04096a5090733a72
apache-airflow-providers-common-sql==1.8.1
apache-airflow-providers-docker==3.8.2
apache-airflow-providers-elasticsearch==5.2.0
apache-airflow-providers-ftp==3.6.1
apache-airflow-providers-google==10.12.0
apache-airflow-providers-grpc==3.3.0
apache-airflow-providers-hashicorp==3.5.0
apache-airflow-providers-http==4.7.0
apache-airflow-providers-imap==3.4.0
apache-airflow-providers-microsoft-azure==8.3.0
apache-airflow-providers-mysql==5.4.0
apache-airflow-providers-odbc==4.2.0
apache-airflow-providers-openlineage==1.2.1
apache-airflow-providers-postgres==5.8.0
apache-airflow-providers-redis==3.4.1
apache-airflow-providers-sendgrid==3.3.0
apache-airflow-providers-sftp==4.7.0
apache-airflow-providers-slack==8.4.0
apache-airflow-providers-snowflake==5.1.2
apache-airflow-providers-sqlite==3.5.0
apache-airflow-providers-ssh==3.8.1

[potiuk]

(note - you will need to pull the images again if you pulled them previously )

[ephraimbuddy]

I will work on getting a new beta. WDYT?

[potiuk]

I will work on getting a new beta. WDYT?

I think it makes sense only if this one is merged an there are other fixes cherry-picked - but yeah - why not :)

BTW. There is also a very small caveat / edge case.

Those images that use chicken-egg providers, do not use "constraints" mainly because - like this is in this case - they use pre-release version of the "chicken-egg providers" build from sources.

So they are generally getting latest matching versions of packages from PyPI, not the ones in the constraints. This is likely never going to cause us problems - because in 9X% of cases they will be the same as constraints, but there is a very small probability that someone released new versions of package between the moment where constraints were updated (when Package is released) and when the image has been built (which will usually happen 10-20 minutes later)

So it's not perfect solution but I consider it "good enough"