Final fix for image caching by potiuk · Pull Request #22618 · apache/airflow

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk · 2022-03-30T13:32:18Z

This is - I hope - final fix for our image caching. I fought with
it for quite a while and it turned out that I was fighting with
buidkit bug when preparing a multiplatform image.

I think I finally got it under control, the main problem was that
when we prepared multi-platform image, seems like the image that
was prepared "last" was overriding the cache that was prepared
by the other platform build. This cause numerous problems with
investigating it, because sometimes it looked like it worked,
when the "other" platform was faster to build and someties it
did not when it was slower. It almost drove me mad.

It looks like however, that the inline cache works better in
this case (and I got it repetetively working) so finally we might
go back to faster builds on CI and fast pull for ./breeze.

I also opened an issue at buildkit, hoping that they will be able
to fix it:

moby/buildkit#2758

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

This is - I hope - final fix for our image caching. I fought with it for quite a while and it turned out that I was fighting with buidkit bug when preparing a multiplatform image. I think I finally got it under control, the main problem was that when we prepared multi-platform image, seems like the image that was prepared "last" was overriding the cache that was prepared by the other platform build. This cause numerous problems with investigating it, because sometimes it looked like it worked, when the "other" platform was faster to build and someties it did not when it was slower. It almost drove me mad. It looks like however, that the inline cache works better in this case (and I got it repetetively working) so finally we might go back to faster builds on CI and fast pull for ./breeze. I also opened an issue at buildkit, hoping that they will be able to fix it: moby/buildkit#2758

potiuk · 2022-03-30T13:33:52Z

I think I have FINALLY got the caching working.

CC: @Bowrna - this might slightly modify the #22344

potiuk · 2022-03-30T13:35:20Z

BTW. @mik-laj - this MIGHT MEAN that we can get the --link back - but I want to make it one-thing-at-a-time.

github-actions · 2022-03-30T13:40:21Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

potiuk · 2022-04-04T11:24:23Z

FYI: The problem with caching multi-node builds has been confirmed and it looks like it's a buildx plugin bug,

I closed the buildkit one and opened docker/buildx#1044

potiuk requested review from ashb, jedcunningham and mik-laj as code owners March 30, 2022 13:32

boring-cyborgBot added the area:dev-tools label Mar 30, 2022

potiuk requested a review from ephraimbuddy March 30, 2022 13:32

mik-laj approved these changes Mar 30, 2022
View reviewed changes

github-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Mar 30, 2022

potiuk merged commit 9e66d14 into apache:mainMar 30, 2022

potiuk deleted the final-fix-for-caching-images branch March 30, 2022 13:41

Bowrna mentioned this pull request Mar 31, 2022
ci prepare-build-cache #22344
Merged

potiuk restored the final-fix-for-caching-images branch April 26, 2022 20:53

potiuk deleted the final-fix-for-caching-images branch July 29, 2022 20:13