Implement synchronization scheme for incr. comp. directory by michaelwoerister · Pull Request #35718 · rust-lang/rust

michaelwoerister · 2016-08-16T15:48:11Z

This PR implements a copy-on-write-based synchronization scheme for the incremental compilation cache directory. For technical details, see the documentation at the beginning of rustc_incremental/persist/fs.rs.

The PR contains unit tests for some functions but for testing whether the scheme properly handles races, a more elaborate test setup would be needed. It would probably involve a small tool that allows to manipulate the incremental compilation directory in a controlled way and then letting a compiler instance run against directories in different states. I don't know if it's worth the trouble of adding another test category to compiletest, but I'd be happy to do so.

Fixes#32754
Fixes#34957

rust-highfive · 2016-08-16T15:48:26Z

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

michaelwoerister · 2016-08-16T15:48:50Z

cc @nikomatsakis @alexcrichton @brson @rust-lang/compiler

Mark-Simulacrum · 2016-08-16T16:20:32Z

nit: Both this bug! and the one before it should be "Trying to initialize.."

nrc · 2016-08-17T04:27:53Z

r? @nikomatsakis

michaelwoerister · 2016-08-17T20:43:01Z

OK, I've implemented the read-write-lock approach. That seems like a clear improvement. I did not, however, move the lock file out of the directory. It made cleanup code more complicated in a few places and having it in the directory should be just as correct. I think, I also addressed all the nits.

nikomatsakis · 2016-08-18T18:36:38Z

is this just copied from compiletest?

It's moved here from rustc_incremental::persist::util which doesn't exist anymore.

nikomatsakis · 2016-08-18T18:59:00Z

ok, r=me modulo nits

michaelwoerister · 2016-08-19T16:39:50Z

@bors r=nikomatsakis

bors · 2016-08-19T16:39:51Z

📌 Commit d1d8258 has been approved by nikomatsakis

michaelwoerister · 2016-08-26T21:09:41Z

@alexcrichton @eddyb This seems to be a restriction purely in the Windows API (i.e. DeleteFileW) and not in the underlying file system. Would it be too much of a performance hit to always prepend that magic prefix?

eddyb · 2016-08-26T21:17:08Z

@michaelwoerister You could do it conditionally, since you already know the length.

michaelwoerister · 2016-08-26T21:18:49Z

@eddyb yep.

Manishearth · 2016-08-27T04:00:17Z

@bors r-

Tidy fails. Feel free to re-r+ when fixed.

/build/src/librustc_incremental/persist/fs.rs:415: trailing whitespace
/build/src/librustc_incremental/persist/fs.rs:1008: trailing whitespace
thread 'main' panicked at 'some tidy checks failed', /build/src/tools/tidy/src/main.rs:53

bors · 2016-08-28T20:21:44Z

☔ The latest upstream changes (presumably #35984) made this pull request unmergeable. Please resolve the merge conflicts.

…on cache.

… on some 32bit systems.

By using "s-" instead of "sess-" as a prefix and encoding numbers as base36 instead of base16.

…longer paths

michaelwoerister · 2016-08-29T19:15:27Z

@alexcrichton Care to take a look at those last two commits and r+ if they look good to you?

alexcrichton · 2016-08-29T19:22:59Z

@bors: r+

bors · 2016-08-29T19:23:00Z

📌 Commit bcd2f90 has been approved by alexcrichton

bors · 2016-08-30T19:20:43Z

⌛ Testing commit bcd2f90 with merge 5a8ca4b...

alexcrichton · 2016-08-30T19:57:43Z

@bors: retry force clean

restarted buildbot

bors · 2016-08-31T19:56:16Z

⌛ Testing commit bcd2f90 with merge 2c01bb8...

…crichton Implement synchronization scheme for incr. comp. directory This PR implements a copy-on-write-based synchronization scheme for the incremental compilation cache directory. For technical details, see the documentation at the beginning of `rustc_incremental/persist/fs.rs`. The PR contains unit tests for some functions but for testing whether the scheme properly handles races, a more elaborate test setup would be needed. It would probably involve a small tool that allows to manipulate the incremental compilation directory in a controlled way and then letting a compiler instance run against directories in different states. I don't know if it's worth the trouble of adding another test category to `compiletest`, but I'd be happy to do so. Fixes#32754Fixes#34957

bors · 2016-09-01T00:40:37Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing 2c01bb8 to master...

rust-highfive assigned nrc Aug 16, 2016

Mark-Simulacrum reviewed Aug 16, 2016
View reviewed changes

rust-highfive assigned nikomatsakis and unassigned nrc Aug 17, 2016

michaelwoerister force-pushed the incr-comp-dir-locking branch from 15f3787 to d86ad10Compare August 17, 2016 20:40

nikomatsakis reviewed Aug 18, 2016
View reviewed changes

michaelwoerister force-pushed the incr-comp-dir-locking branch 2 times, most recently from 14b9dc8 to d1d8258Compare August 19, 2016 16:39

michaelwoerister added 11 commits August 29, 2016 14:27

Move flock.rs from librustdoc to librustc_data_structures.
6ef8198

Add some features to flock.
206e7b6

Implement copy-on-write scheme for managing the incremental compilati…
3e9bed9
…on cache.

incr.comp.: Move lock files out of directory being locked
794fd31

Fix rustbuild
004a7eb

incr. comp.: Fix test_timestamp_serialization so it does not overflow…
b67f57a
… on some 32bit systems.

incr.comp.: Make path's of session directories slightly shorter.
a3dc5f9
By using "s-" instead of "sess-" as a prefix and encoding numbers as base36 instead of base16.

incr.comp.: Make compiletest generate shorter cache directory names.
9d11068

Fix tidy-errors
68d2275

compiletest: Canonicalize paths when remove incr.comp. dir, enabling …
50b008a
…longer paths

incr.comp.: Canonicalize path to session directory before deleteing it.
bcd2f90

michaelwoerister force-pushed the incr-comp-dir-locking branch from bf29d72 to bcd2f90Compare August 29, 2016 18:59

bors merged commit bcd2f90 into rust-lang:masterSep 1, 2016

This was referenced Sep 1, 2016
incr. comp.: Take spans into account for ICH #36025
Merged
Use workspaces and switch to a single Cargo.lock. #36032
Closed
Replace try! with ?. #36041
Merged

Uh oh!

Uh oh!

Conversation

michaelwoerister commented Aug 16, 2016

Uh oh!

rust-highfive commented Aug 16, 2016

Uh oh!

michaelwoerister commented Aug 16, 2016

Uh oh!

Mark-SimulacrumAug 16, 2016

Choose a reason for hiding this comment

Uh oh!

michaelwoeristerAug 16, 2016

Choose a reason for hiding this comment

Uh oh!

nrc commented Aug 17, 2016

Uh oh!

michaelwoerister commented Aug 17, 2016

Uh oh!

nikomatsakisAug 18, 2016

Choose a reason for hiding this comment

Uh oh!

michaelwoeristerAug 19, 2016

Choose a reason for hiding this comment

Uh oh!

nikomatsakis commented Aug 18, 2016

Uh oh!

michaelwoerister commented Aug 19, 2016

Uh oh!

bors commented Aug 19, 2016

Uh oh!

michaelwoerister commented Aug 26, 2016

Uh oh!

eddyb commented Aug 26, 2016

Uh oh!

michaelwoerister commented Aug 26, 2016

Uh oh!

Manishearth commented Aug 27, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bors commented Aug 28, 2016

Uh oh!

michaelwoerister commented Aug 29, 2016

Uh oh!

alexcrichton commented Aug 29, 2016

Uh oh!

bors commented Aug 29, 2016

Uh oh!

bors commented Aug 30, 2016

Uh oh!

alexcrichton commented Aug 30, 2016

Uh oh!

bors commented Aug 31, 2016

Uh oh!

bors commented Sep 1, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Manishearth commented Aug 27, 2016•
edited
Loading