Cache fuchsia checkout in a docker layer

[erickt]

This allows the Fuchsia checkout to be cached and shared across builds, which avoids any rate limits with checking out Fuchsia. Note though that the Fuchsia checkout is about 27GiB as of checkout, which might cause separate problems.

try-job: x86_64-fuchsia

[rustbot]

r? @marcoieni

rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[erickt]

@bors try

[bors]

⌛ Trying commit 4b63766 with merge 191a16c...

[bors]

💔 Test failed - checks-actions

[klensy]

Fuchsia checkout is about 27GiB as of checkout

git sparse checkout?

[Kobzol]

Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.

[erickt]

Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.

@Kobzol - I'm not too familiar with how the docker run.sh scripts work, but if I'm reading src/ci/docker/host-x86_64/x86_64-fuchsia/Dockerfile, the ENV SCRIPT ... at the send suggests to me we're running the build-fuchsia.sh script outside of the cache. So factoring out the checkout might avoid all that IO.

Fuchsia checkout is about 27GiB as of checkout

git sparse checkout?

@klensy: we're working on a way to use sparse checkouts, but that's not actually the majority of the issue. Turns out most of our artifacts are prebuilt binaries, like our Go and Dart toolchains, which we really don't need to run our limited tests for Rust. I don't think we have an easy way to filter out these artifacts from being download, but I might be able to just delete them from the docker directory to remove unnecessary things though. I'll experiment with this.

[Kobzol]

Our Docker builds are automatically cached, and unless their inputs change, they are not rebuilt on CI. So I'm not sure how much more caching we could do, tbh.

@Kobzol - I'm not too familiar with how the docker run.sh scripts work, but if I'm reading src/ci/docker/host-x86_64/x86_64-fuchsia/Dockerfile, the ENV SCRIPT ... at the send suggests to me we're running the build-fuchsia.sh script outside of the cache. So factoring out the checkout might avoid all that IO.

Fuchsia checkout is about 27GiB as of checkout

git sparse checkout?

@klensy: we're working on a way to use sparse checkouts, but that's not actually the majority of the issue. Turns out most of our artifacts are prebuilt binaries, like our Go and Dart toolchains, which we really don't need to run our limited tests for Rust. I don't think we have an easy way to filter out these artifacts from being download, but I might be able to just delete them from the docker directory to remove unnecessary things though. I'll experiment with this.

Oh, I see, I didn't notice that. Well, that actually seems quite wasteful! I suppose that we could move the download to a Docker layer (so run it during docker build, which is cached), and then only do bash scripts/rust/build_fuchsia_from_rust_ci.sh when the container runs (so run it during docker run? I don't know if the recent errors were in the download or build phase though.

[bors]

☔ The latest upstream changes (presumably #142003) made this pull request unmergeable. Please resolve the merge conflicts.

[Dylan-DPC]

@erickt any updates on this? thanks

[Dylan-DPC]

Closing this as inactive. Feel free to reöpen this pr or create a new pr if you get the time to work on this. Thanks