Pass extra index url with build secrets

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.

[gunziptarball]

Proposed approach to installing packages from a password-protected extra index URL without persisting in a built image. I could use help with building a test for this too (see TODO remarks). I'll try to add one when I have more time, but wanted to pitch the idea first. Thanks! 😄

Also many thanks and props to @potiuk for really cool work on the Dockerfile. I've learned a lot about the new features from this!

References

• Build --secret with buildkit docker/cli#1288
• https://pythonspeed.com/articles/docker-build-secrets/

related: #22492

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

Thenks for kudos. Sorry for not responding before.

I tnink however, that this is one is completely not needed. The image is multi-segment one. And pip install is deliberately only done in the "airlfow-build-image" stage. Then installed packages are copied (without requirements or secrets) into the final (main) segment and original "build" segment image is not stored anywhere but locally stored `build-image' state - which you can remove after the build is successful.

The final image has only installed package binaries in /root/.local - no other information is stored. You can check it by using dive command and examining the layers.

Provisionally closing that one unless there is something I do not understand.

# This is a multi-segmented image. It actually contains two images:
#
# airflow-build-image - there all airflow dependencies can be installed (and
# built - for those dependencies that require
# build essentials). Airflow is installed there with
# --user switch so that all the dependencies are
# installed to ${HOME}/.local
#
# main - this is the actual production image that is much
# smaller because it does not contain all the build
# essentials. Instead the ${HOME}/.local folder
# is copied from the build-image - this way we have
# only result of installation and we do not need
# all the build essentials. This makes the image
# much smaller.

[gunziptarball]

Thanks for the review! I had thought this would be a problem based on this article but I overlooked the bit that said a work around was to use multi stage builds. In this particular case I guess it is indeed unneeded.

[potiuk]

Yeah. The buildkit --secret is pretty cool but it does not solve the problem of building things locally. If you want to have remote cache (like we do) and use it extensively during the build then I found it rather useless. Multi-stage builds are much more cache-friendly in this case because when you rebuild the image locally with cache, you can pull most of the layers rather than rebuild them.