Skip to content

ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches#8821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jorgecarleitao wants to merge 5 commits into from
Closed

ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches #8821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
189362f
Simplify CI build from rust.
jorgecarleitao Dec 1, 2020
1c392cf
Removed no longer needed files.
jorgecarleitao Dec 3, 2020
1584c76
Improved README about how to compile the project in docker.
jorgecarleitao Dec 5, 2020
2e3de55
Improved comment.
jorgecarleitao Dec 5, 2020
774e704
Fixed integration.
jorgecarleitao Dec 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
401 changes: 294 additions & 107 deletions .github/workflows/rust.yml
View file
Open in desktop

Large diffs are not rendered by default.

79 changes: 0 additions & 79 deletions .github/workflows/rust_cron.yml
View file
Open in desktop

This file was deleted.

78 changes: 0 additions & 78 deletions ci/docker/debian-10-rust.dockerfile
View file
Open in desktop

This file was deleted.

41 changes: 0 additions & 41 deletions ci/scripts/rust_coverage.sh
View file
Open in desktop

This file was deleted.

30 changes: 0 additions & 30 deletions ci/scripts/rust_lint.sh
View file
Open in desktop

This file was deleted.

42 changes: 0 additions & 42 deletions ci/scripts/rust_pyarrow_integration.sh
View file
Open in desktop

This file was deleted.

55 changes: 0 additions & 55 deletions ci/scripts/rust_test.sh
View file
Open in desktop

This file was deleted.

30 changes: 0 additions & 30 deletions docker-compose.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -102,7 +102,6 @@ x-hierarchy:
- debian-java:
- debian-java-jni
- debian-js
- debian-rust
- fedora-cpp:
- fedora-python
- ubuntu-cpp:
Expand DownExpand Up@@ -903,35 +902,6 @@ services:
/bin/bash -c "
/arrow/ci/scripts/r_sanitize.sh /arrow"

################################ Rust #######################################

debian-rust:

@kszucskszucsDec 4, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we plan to use plain GHA jobs without docker-compose it'd be nice to keep it around to let contributors execute rust builds without installing the toolchain locally.

@jorgecarleitaojorgecarleitaoDec 4, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The arguments I used to remove it:

  • this compose and associated dockerfile are no longer tested under the CI, and their primary purpose was to be used in the CI
  • the dockerfile is not needed, as the default rust image is sufficient
  • the compose contains non-trivial aspects, such as:
    • volume mounting with delegate
    • using the no-longer needed rust_test.sh
    • non-default CARGO_* env variables
  • docker-compose run debian-rust is IMO less transparent to a user about what is happening, and the person needs to go to a +1k LOC file to understand.

In opposition, this whole project can now be built using

docker run --rm -v $(pwd)/rust:/rust -it rust /bin/bash -c "cd /rust && cargo build"

which IMO is really simple to understand and reproduce. Wouldn't this command on the README be sufficient for contributors?

In the end, I evaluated that having these around would be more confusing than just having the user to follow standard rust practices of building a rust project (including using the official docker image), and therefore there was no need for the maintenance burden of having these around.

@kszucskszucsDec 4, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, sounds good to me - lucky us that we have cargo around.

Please document how to reproduce the builds locally with docker.

@jorgecarleitaojorgecarleitaoDec 5, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a section on the README on how to compile, which includes both pure cargo and docker variants.

# Usage:
# docker-compose build debian-rust
# docker-compose run debian-rust
image: ${REPO}:${ARCH}-debian-10-rust-${RUST}
build:
context: .
dockerfile: ci/docker/debian-10-rust.dockerfile
cache_from:
- ${REPO}:${ARCH}-debian-10-rust-${RUST}
args:
arch: ${ARCH}
rust: ${RUST}
shm_size: *shm-size
environment:
CARGO_HOME: /rust/cargo
CARGO_TARGET_DIR: /build/rust
volumes: &rust-volumes
- .:/arrow:delegated
- ${DOCKER_VOLUME_DIRECTORY:-.docker}/${ARCH}-debian-${DEBIAN}-rust:/build:delegated
command: &rust-command >
/bin/bash -c "
echo ${RUST} > /arrow/rust/rust-toolchain &&
if [ ! -d "/build/rust" ]; then mv /rust/target /build/rust; fi &&
/arrow/ci/scripts/rust_build.sh /arrow /build &&
/arrow/ci/scripts/rust_test.sh /arrow /build"

################################# Go ########################################

debian-go:
Expand Down
Loading