Add expansion info to crate metadata

[ibabushkin]

This implements the serialization and deserialization of hygiene info in crate metadata, preserving macro expansion information across crates, addressing some of the points mentioned in #40847. There is some work left to do, however: Apparently there is code that hides macro backtraces from external crates and some of the decoding logic might need to be expanded or moved to accomodate for all current use cases I might have missed.

r? @jseyfried

[ibabushkin]

I couldn't find any info on how multiple declarations of thread_local!s behave, so this might need to be moved for the expected behaviour.

[ibabushkin]

Okay, a lot of symptoms point to having two thread locals being the issue. However, replacing them with one leads to absolutely uncontrolled memory consumption.

[ibabushkin]

So currently, the on-demand loading mechanism for hygiene data is still incorrect (though I have a local fix for that). Moreover, some tests still fail. My current theory is that encoding the hygiene data of a crate doesn't force all of it's dependencies' hygiene data to be decoded first.

One could fix this either by forcing a complete decoding procedure for each dependency upon encoding of by simply making decoding of that particular metadata strict (which I'd like to avoid).

[bors]

☔ The latest upstream changes (presumably #43710) made this pull request unmergeable. Please resolve the merge conflicts.

[ibabushkin]

For reference, to reproduce the crash on the backtrace tests in a more comfortable environment:

RUST_LOG=debug ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc -g --crate-type=lib src/test/run-pass/backtrace.rs

I'm starting my search from there.

[ibabushkin]

Okay, the implementation now doesn't cause any crashes and is okay resource-usage-wise. What remains to do now is to remove leftover cruft, and to clean up functionality in other places. For instance, cross-crate macro backtraces would probably benefit from these changes, but the integration remains to be implemented. Finally, some tests might be in order. Given all these points, I think now is a good time to review what we've got so far.

[jseyfried]

Started reviewing, looks good so far. I think we should be able to test using nested macro_rules! (at least to observe that this PR behaves differently from master) and/or nested macros (with #![feature(decl_macro)]).

[ibabushkin]

Alright, I'll fiddle something like that together.

[ibabushkin]

Quick update: in the meantime, some recent changes are breaking the functionality again - I'll try to find out what happens.

[bors]

☔ The latest upstream changes (presumably #43968) made this pull request unmergeable. Please resolve the merge conflicts.

[arielb1]

What's the status of this PR? Are you still interested in it @ibabushkin?

[ibabushkin]

On 2017-09-05, Ariel Ben-Yehuda wrote: What's the status of this PR? Are you still interested in it @ibabushkin? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #43847 (comment)

Yes, absolutely. I'm sorry for the stall - some bugs (re?)appeared and some other things got in the way :/

[aidanhs]

Hi @ibabushkin, just a friendly ping to make sure this PR doesn't get lost! :)

[arielb1]

What's the status of this PR? Are you still interested in it @ibabushkin?

[ibabushkin]

Yes, absolutely. I'm sorry for the delays piling up - I fear I'll only be able to get back to this early next month :/. Would you prefer me to close the PR in the meantime and reopen it when it is ready?

[carols10cents]

Would you prefer me to close the PR in the meantime and reopen it when it is ready?

That's up to you @ibabushkin! We're happy to hold off on the reminder pings until October if you want to leave this open :)

[aidanhs]

Hi @ibabushkin, you mentioned early October, so just a friendly ping to make sure you're still aware of this PR!

[ibabushkin]

I'm taking up the work again! :) Thank you all for your patience.

[kennytm]

This test failed.

[00:56:33] ---- [compile-fail] compile-fail-fulldeps/macro-spans.rs stdout ----
[00:56:33] [00:56:33] error: /checkout/src/test/compile-fail-fulldeps/macro-spans.rs:16: expected error not found: recursive type `Name::Name` has infinite size
[00:56:33] [00:56:33] error: 0 unexpected errors found, 1 expected errors not found
[00:56:33] status: exit code: 101
[00:56:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail-fulldeps/macro-spans.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused"
[00:56:33] not found errors (from test file): [
[00:56:33] Error {
[00:56:33] line_num: 16,
[00:56:33] kind: Some(
[00:56:33] Error
[00:56:33] ),
[00:56:33] msg: "recursive type `Name::Name` has infinite size"
[00:56:33] }
[00:56:33] ]
[00:56:33] [00:56:33] thread '[compile-fail] compile-fail-fulldeps/macro-spans.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:1084:12
[00:56:33] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:56:33] [00:56:33] [00:56:33] failures:
[00:56:33] [compile-fail] compile-fail-fulldeps/macro-spans.rs
[00:56:33] [00:56:33] test result: FAILED. 52 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

[ibabushkin]

I am aware :) - fixing it is not quite trivial, working on it.

…

On 2017-10-14, kennytm wrote: kennytm commented on this pull request. > +// Copyright 2017 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or +// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license +// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. + +// aux-build:macro_spans_lib.rs + +#[macro_use] +extern crate macro_spans_lib; + +abc!(Name); //~ ERROR recursive type `Name::Name` has infinite size This test failed. ``` [00:56:33] ---- [compile-fail] compile-fail-fulldeps/macro-spans.rs stdout ---- [00:56:33] [00:56:33] error: /checkout/src/test/compile-fail-fulldeps/macro-spans.rs:16: expected error not found: recursive type `Name::Name` has infinite size [00:56:33] [00:56:33] error: 0 unexpected errors found, 1 expected errors not found [00:56:33] status: exit code: 101 [00:56:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail-fulldeps/macro-spans.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused" [00:56:33] not found errors (from test file): [ [00:56:33] Error { [00:56:33] line_num: 16, [00:56:33] kind: Some( [00:56:33] Error [00:56:33] ), [00:56:33] msg: "recursive type `Name::Name` has infinite size" [00:56:33] } [00:56:33] ] [00:56:33] [00:56:33] thread '[compile-fail] compile-fail-fulldeps/macro-spans.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:1084:12 [00:56:33] note: Run with `RUST_BACKTRACE=1` for a backtrace. [00:56:33] [00:56:33] [00:56:33] failures: [00:56:33] [compile-fail] compile-fail-fulldeps/macro-spans.rs [00:56:33] [00:56:33] test result: FAILED. 52 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out ``` -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #43847 (review)

[ibabushkin]

It seems that the error display logic broke in the meantime, which is rather odd, but hints at more serious problems. I am currently considering starting this feature from scratch.

[alexcrichton]

ping @ibabushkin, was curious how the progress is here!

[ibabushkin]

@alexcrichton To be honest, I am currently hung up debugging the test failures - and I suspect more systematic problems with this changeset. I asked in #rust-internals about approaches to a more comfortable debugging/logging setup, but so far I wasn't really successful.

[carols10cents]

@alexcrichton To be honest, I am currently hung up debugging the test failures - and I suspect more systematic problems with this changeset. I asked in #rust-internals about approaches to a more comfortable debugging/logging setup, but so far I wasn't really successful.

@jseyfried do you have any ideas on how @ibabushkin could debug the test failures here?

[bors]

☔ The latest upstream changes (presumably #45674) made this pull request unmergeable. Please resolve the merge conflicts.

[carols10cents]

Hi @ibabushkin, thank you for this PR! Due to inactivity, merge conflicts, and the possibility you raised that there might be systematic problems with this changeset, I'm going to close this.

Please do continue to reach out when you have time to work on this to get help (try pinging @jseyfried specifically on IRC?) and either reopen this PR or start a new PR when you're ready!