Generate XZ-compressed tarballs by ranma42 · Pull Request #41600 · rust-lang/rust

ranma42 · 2017-04-28T07:02:04Z

Integrate the new rust-installer and extend manifests with keys for xz-compressed tarballs.

One of the steps required for #21724

instead of combining the component and target into a URL.

The same unavailable target value is used in two different places. Abstracting it makes it easier to update it and recognise its purpose.

rust-highfive · 2017-04-28T07:02:15Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

ranma42 · 2017-04-28T07:03:28Z

I tried to add the required packages to the CI infrastructure, but I am not very confident that everything is in place. I trust the CI itself to warn about such issues.
Should I ALLOW_PR=1 to do that?

ishitatsuyuki · 2017-04-28T11:20:30Z

You can use ALLOW_PR to get rapid feedback; doing it on one of the builders (preferably x86_64-linux-gnu) doesn't jam the queue so much. Make sure you add some sort of WIP tag.

nikomatsakis · 2017-04-28T18:34:01Z

Who should review this? Not, I think, me.

nikomatsakis · 2017-04-28T18:34:24Z

r? @alexcrichton -- feel free to pick someone else, I figured you'd know the right person tho

alexcrichton

Looks excellent to me, thakns @ranma42!

Right now if this gets past bors then we unfortunately don't have many guarantees. It should exercise all xz-producing code but doesn't exercise the manifest generator, that'll happen when the next nightly is built.

You can also run everything locally (at least container-wise) with ./src/ci/docker/run.sh arm-android (etc)

alexcrichton · 2017-04-28T19:40:09Z

This doesn't currently validate that the xz file actually exists, could that be done as an extra sanity check?

self.digests should only contain digests for existing files. Do you expect the file to be deleted between the digest computation and the construction of the manifest?

Oh good point yeah, but that means that the url could be specified and the digest could be none, right? Could we statically prevent a situation such as that?

xz_url: xz_digest.as_ref().map(|_| self.url(&xz_filename)) makes sure that the url is only specified if the digest is Something (and that its contents are based on the filename, not on the digest value).

gah sorry, yes of course!

alexcrichton · 2017-04-28T19:41:08Z

A bunch of containers don't actually produce tarballs, only the dist-* ones do I believe. Could this package be left out of all the containers (like this one) not producing tarballs?

Uhm... Actually I might have removed one too many: does the cross image also produce tarballs? Even if it is not dist-*, it is flagged DEPLOY=1.

Oops yes sorry DEPLOY=1 is the marker to look for.

alexcrichton · 2017-04-28T19:42:45Z

cc @brson

mattico · 2017-04-28T20:31:43Z

Out of curiosity: why are you using p7zip rather than xz-utils (which is already being installed)?

ranma42 · 2017-04-28T20:45:28Z

@mattico 7z is available on AppVeyor (where it is somewhat harder to install software). See rust-lang/rust-installer#57 (comment)

alexcrichton · 2017-04-29T17:17:16Z

Ok everything looks great to me @ranma42! Want to squash and remove '[WIP]' and r+?

ranma42 · 2017-04-29T19:44:57Z

Squashed and renamed :)

alexcrichton · 2017-04-30T04:29:00Z

@bors: r+

bors · 2017-04-30T04:29:01Z

📌 Commit 36b9ab7 has been approved by alexcrichton

bors · 2017-04-30T07:46:55Z

⌛ Testing commit 36b9ab7 with merge 5616298...

bors · 2017-04-30T07:58:07Z

💔 Test failed - status-travis

ranma42 · 2017-04-30T08:24:15Z

Uh, of course distcheck also requires 7z. I amended the last commit adding it (and mentioning in the commit message the affected docker images).

alexcrichton · 2017-04-30T09:12:20Z

@bors: r+

bors · 2017-04-30T09:12:20Z

📌 Commit bf4082a has been approved by alexcrichton

bors · 2017-05-03T05:52:34Z

⌛ Testing commit 6bf71f0 with merge d5948bd...

bors · 2017-05-03T06:04:09Z

💔 Test failed - status-travis

TimNN · 2017-05-03T06:05:45Z

Looks legitimate:

[00:03:47] make-tarballs: error: need 7z

(https://travis-ci.org/rust-lang/rust/jobs/228215768)

@bors r-

MacOSX does not ship `7z` nor `xz`. Let's use `xz`, just like on the other *nix systems.

ranma42 · 2017-05-03T06:11:27Z

I must have messed up when squashing the update of the rust-installer submodule, sorry. It should be fixed by my latest push.

alexcrichton · 2017-05-03T14:02:57Z

@bors: r+

bors · 2017-05-03T14:02:58Z

📌 Commit 5e522d7 has been approved by alexcrichton

Generate XZ-compressed tarballs Integrate the new `rust-installer` and extend manifests with keys for xz-compressed tarballs. One of the steps required for rust-lang#21724

bors · 2017-05-03T19:18:33Z

⌛ Testing commit 5e522d7 with merge 234472d...

aidanhs · 2017-05-03T19:39:09Z

Think this caused a rollup failure in #41725 on the osx xcode 7 builders

The command "brew update && brew install xz" failed and exited with 1 during

Can't decipher why from the logs, but both of the osx xcode 7 builders failed, so I suspect it'll fail here as well.

alexcrichton · 2017-05-03T19:55:50Z

@aidanhs oddly enough all builders have gotten past the xz installation stage here ...

alexcrichton · 2017-05-03T19:58:16Z

@bors: r-

er I believe @frewsxcv is correct

@ranma42 can you ensure that there are retry loops around the installations on OSX?

alexcrichton · 2017-05-03T19:58:31Z

@bors: retry

Wrap the installation on macOS with `travis_retry`.

ranma42 · 2017-05-03T20:42:50Z

I added travis_retry to the update and install commands, but it is not clear to me why the log was mangled in that way (as if two commands were running concurrently).

alexcrichton · 2017-05-03T22:32:21Z

@bors: r+

Yeah I also found that log... interestingly ordered

bors · 2017-05-03T22:32:22Z

📌 Commit 98dd82c has been approved by alexcrichton

Generate XZ-compressed tarballs Integrate the new `rust-installer` and extend manifests with keys for xz-compressed tarballs. One of the steps required for rust-lang#21724

Rollup of 6 pull requests - Successful merges: #41543, #41600, #41715, #41720, #41721, #41730 - Failed merges:

bors · 2017-05-04T01:40:12Z

⌛ Testing commit 98dd82c with merge b16c7a2...

As of rust-lang/rust#41600 we now need `xz` to build Rust from HEAD.

ranma42 added 2 commits April 25, 2017 16:08

Accept arbitrary filenames into URLs in build-manifest
ac8892b
instead of combining the component and target into a URL.

Define the unavailable target just once and re-use it
5439c9b
The same unavailable target value is used in two different places. Abstracting it makes it easier to update it and recognise its purpose.

rust-highfive assigned nikomatsakis Apr 28, 2017

ranma42 changed the title ~~Generate XZ-compressed tarballs~~[WIP] Generate XZ-compressed tarballsApr 28, 2017

shepmaster added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Apr 28, 2017

rust-highfive assigned alexcrichton and unassigned nikomatsakis Apr 28, 2017

alexcrichton reviewed Apr 28, 2017
View reviewed changes

alexcrichton added the T-tools label Apr 28, 2017

ranma42 force-pushed the xz-dist branch from 0fcb111 to 36b9ab7Compare April 29, 2017 19:44

ranma42 changed the title ~~[WIP] Generate XZ-compressed tarballs~~Generate XZ-compressed tarballsApr 29, 2017

ranma42 force-pushed the xz-dist branch from 36b9ab7 to bf4082aCompare April 30, 2017 08:23

ranma42 added 2 commits May 3, 2017 08:09

Emit information about XZ packages in manifest
5dfa08e

Fix MacOSX build
5e522d7
MacOSX does not ship `7z` nor `xz`. Let's use `xz`, just like on the other *nix systems.

ranma42 force-pushed the xz-dist branch from 6bf71f0 to 5e522d7Compare May 3, 2017 06:11

frewsxcv mentioned this pull request May 3, 2017
Rollup of 4 pull requests #41725
Closed

Retry brew commands upon failure
98dd82c
Wrap the installation on macOS with `travis_retry`.

frewsxcv mentioned this pull request May 3, 2017
Rollup of 6 pull requests #41735
Merged

bors added a commit that referenced this pull request May 3, 2017
Auto merge of#41735- frewsxcv:rollup, r=frewsxcv
b16c7a2
Rollup of 6 pull requests - Successful merges: #41543, #41600, #41715, #41720, #41721, #41730 - Failed merges:

bors merged commit 98dd82c into rust-lang:masterMay 4, 2017

alexcrichton mentioned this pull request May 5, 2017
rustup nightly install failing rust-lang/rustup#1092
Closed

brandt added a commit to brandt/homebrew-core that referenced this pull request May 6, 2017
rust: fix missing build requirement xz
ceac172
As of rust-lang/rust#41600 we now need `xz` to build Rust from HEAD.

Uh oh!

Uh oh!

Conversation

ranma42 commented Apr 28, 2017

Uh oh!

rust-highfive commented Apr 28, 2017

Uh oh!

ranma42 commented Apr 28, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishitatsuyuki commented Apr 28, 2017

Uh oh!

nikomatsakis commented Apr 28, 2017

Uh oh!

nikomatsakis commented Apr 28, 2017

Uh oh!

alexcrichton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ranma42Apr 28, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexcrichton commented Apr 28, 2017

Uh oh!

mattico commented Apr 28, 2017

Uh oh!

ranma42 commented Apr 28, 2017

Uh oh!

alexcrichton commented Apr 29, 2017

Uh oh!

ranma42 commented Apr 29, 2017

Uh oh!

alexcrichton commented Apr 30, 2017

Uh oh!

bors commented Apr 30, 2017

Uh oh!

bors commented Apr 30, 2017

Uh oh!

bors commented Apr 30, 2017

Uh oh!

ranma42 commented Apr 30, 2017

Uh oh!

alexcrichton commented Apr 30, 2017

Uh oh!

bors commented Apr 30, 2017

Uh oh!

bors commented May 3, 2017

Uh oh!

bors commented May 3, 2017

Uh oh!

TimNN commented May 3, 2017

Uh oh!

ranma42 commented May 3, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexcrichton commented May 3, 2017

Uh oh!

bors commented May 3, 2017

Uh oh!

bors commented May 3, 2017

Uh oh!

aidanhs commented May 3, 2017

Uh oh!

ranma42 commented Apr 28, 2017•
edited
Loading

ranma42Apr 28, 2017•
edited
Loading

ranma42 commented May 3, 2017•
edited
Loading