Detect truncated DepGraph files

[saethlin]

I suspect that the following issues are caused by truncated incr comp files:

• rustc panics due to stale incremental compilation info #120582
• Compiler panicked when running tests on Windows 11 WSL #121499
• panic on compile of basic Serenity example #122210
• using mold results into panic #123073

We fail with an allocation failure or capacity overflow in this case because we assume that the ending bytes of an DepGraph file are the lengths of arrays. If the file has somehow been truncated then the ending bytes are probably some of our varint encoding, which tries to eliminate zero bytes, so interpreting a random 8 bytes as an array length has a very high chance of producing a byte capacity over isize::MAX.

Now theoretically since #119510 merged I have fixed the out-of-disk issues and yet in #120894 (comment) I still see some decoding failures that look like out-of-disk ICEs, for example https://crater-reports.s3.amazonaws.com/beta-1.77-1/beta-2024-02-10/gh/scottfones.aoc_2022/log.txt

So this PR should ensure that we get an ICE that clearly identifies if the file in question is truncated.

[rustbot]

r? @lcnr

rustbot has assigned @lcnr.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[cjgillot]

Should this be an ICE or a fatal error? Do we suggest a cargo clean?

[amyipdev]

My personal suggestion (granted, as a non-contributor, so worth a grain of salt) would be to suggest a cargo clean with an ICE if possible. cargo clean does appear to unstick and allow later compilation to continue normally, and a fatal error doesn't seem appropriate/the right severity.

[saethlin]

If the assertion I'm trying to add in this PR ever fails, we have a new defect somewhere. We should ICE.

If we want to encourage people to paper over the problem with cargo clean we should just remove the file for them.

[Noratrieb]

Would it be simpler to just append some magic bytes like b"the-end-of-rust" or something like that? Then all the metadata calls wouldn't be needed.

[saethlin]

I'd rather report the expected and actual numbers so that we can tell if the dep graph is truncated, unexpectedly large, and about how big it is. Some systems have file size limits that it's plausible we run into, and having a number would let us know about that.

[saethlin]

Turns out this is the hard part:

rust/compiler/rustc_incremental/src/persist/save.rs

Lines 169 to 181 in 8401645

	file_format::write_file_header(&mut encoder, sess);
	// First encode the commandline arguments hash
	sess.opts.dep_tracking_hash(false).encode(&mut encoder);
	Some(DepGraph::new(
	&sess.prof,
	prev_graph,
	prev_work_products,
	encoder,
	sess.opts.unstable_opts.query_dep_graph,
	sess.opts.unstable_opts.incremental_info,
	))

the DepGraph that's encoded here doesn't go through save_in, which is truly awful.

[lcnr]

r? @cjgillot feel free to reassign

[lcnr]

r? compiler

[petrochenkov]

I think it's fine to land this as an ICE first and then think what to do next when some actual examples of that ICE are found in practice.
@bors r+

[bors]

📌 Commit 11f8866 has been approved by petrochenkov

It is now in the queue for this repository.