Uh oh!
There was an error while loading. Please reload this page.
ARROW-12112: [CI] Reduce footprint of conda-integration image - #9891
ARROW-12112: [CI] Reduce footprint of conda-integration image#9891pitrou wants to merge 1 commit into
Conversation
Uh oh!
There was an error while loading. Please reload this page.
da2d253 to
72b9f03CompareThere was a problem hiding this comment.
Does this really help? Docker layers are on top of each other. This thus then only "hide" the files but does not remove them from disk.
There was a problem hiding this comment.
Ah... That's a good question. But I assume image layers are compressed? At least this addresses the uncompressed container size.
There was a problem hiding this comment.
Once you install something in a layer, it gets added to the total image size. This is why we should prefer install and cleanup up steps in the same layer.
There was a problem hiding this comment.
In this case, the install was done in conda-cpp...
There was a problem hiding this comment.
We should check whether we need the clangdev and llvmdev packages. I would guess that llvm would be sufficient as we link dynamically. Not sure what part of clang(dev) we need.
There was a problem hiding this comment.
Instead of focusing on shrinking the docker images, I'd rather consider to free up more space on the github actions hosted agents (if we can easily), see the space usage here: https://github.com/apache/arrow/pull/9814/checks?check_run_id=2205470543#step:4:83
The GHA hosted agents are provisioned using packer, here is the configuration for ubuntu 20.04: https://github.com/actions/virtual-environments/blob/main/images/linux/toolsets/toolset-2004.json
Presumably the android SDK consumes a lot of space, but there are other toolsets we don't use either.
There was a problem hiding this comment.
Well, we can do both. The size of docker images and container also affects local development.
There was a problem hiding this comment.
I meant deleting is probably easier and quicker, reduced image size would be better of course.
There was a problem hiding this comment.
Hmm, the problem is that the delete step can take 1 minute if I remove the Android toolkit...
ba11008 to
5ca8677CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
* Reduce conda packages footprint * Make Rust install more minimal * Free up more space on Github Actions
5ca8677 to
24e592dComparepitrou
commented
Apr 6, 2021
I think I addressed your review comments @kszucs |
kszucs
commented
Apr 7, 2021
The build failures are unrelated, merging. |
This seems to save about 2GB in the image's root directory.