Record proc macro harness order for use during metadata deserialization by Aaron1011 · Pull Request #68814 · rust-lang/rust

Aaron1011 · 2020-02-03T23:49:05Z

When we generate the proc macro harness, we now explicitly recorder the
order in which we generate entries. We then use this ordering data to
deserialize the correct proc-macro-data from the crate metadata.

rust-highfive · 2020-02-03T23:49:09Z

r? @varkor

(rust_highfive has picked a reviewer for you, use r? to override)

rust-highfive · 2020-02-04T00:53:48Z

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-02-03T23:49:40.8007554Z ========================== Starting Command Output ===========================
2020-02-03T23:49:40.8010475Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/6ca53ed3-2685-45c9-843c-e0490d7c4ff6.sh
2020-02-03T23:49:40.8010583Z 2020-02-03T23:49:40.8013379Z ##[section]Finishing: Disable git automatic line ending conversion
2020-02-03T23:49:40.8019046Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68814/merge to s
2020-02-03T23:49:40.8020682Z Task : Get sources
2020-02-03T23:49:40.8020713Z Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-03T23:49:40.8020742Z Version : 1.0.0
2020-02-03T23:49:40.8020773Z Author : Microsoft
---
2020-02-03T23:49:42.5639196Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-02-03T23:49:42.5652631Z ##[command]git config gc.auto 0
2020-02-03T23:49:42.5658921Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-02-03T23:49:42.5663998Z ##[command]git config --get-all http.proxy
2020-02-03T23:49:42.5673820Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/68814/merge:refs/remotes/pull/68814/merge
---
2020-02-04T00:42:53.0410167Z .................................................................................................... 1700/9570
2020-02-04T00:42:57.9844809Z .................................................................................................... 1800/9570
2020-02-04T00:43:10.2843115Z ...........................i........................................................................ 1900/9570
2020-02-04T00:43:17.2024019Z .................................................................................................... 2000/9570
2020-02-04T00:43:31.1600219Z .................iiiii.............................................................................. 2100/9570
2020-02-04T00:43:40.9755568Z .................................................................................................... 2300/9570
2020-02-04T00:43:43.4243700Z .................................................................................................... 2400/9570
2020-02-04T00:43:48.2833764Z .................................................................................................... 2500/9570
2020-02-04T00:44:08.2998357Z .................................................................................................... 2600/9570
---
2020-02-04T00:46:41.1377851Z ............................................................i...............i....................... 4900/9570
2020-02-04T00:46:48.0886880Z .................................................................................................... 5000/9570
2020-02-04T00:46:55.5952197Z .................................................................................................... 5100/9570
2020-02-04T00:47:00.0215209Z ...i................................................................................................ 5200/9570
2020-02-04T00:47:10.3802521Z .............................................................................ii.ii........i...i..... 5300/9570
2020-02-04T00:47:18.6403440Z ...............i.................................................................................... 5500/9570
2020-02-04T00:47:27.8635074Z .................................................................................................... 5600/9570
2020-02-04T00:47:34.1423060Z ................................................................i................................... 5700/9570
2020-02-04T00:47:41.0006263Z .................................................................................................... 5800/9570
2020-02-04T00:47:41.0006263Z .................................................................................................... 5800/9570
2020-02-04T00:47:48.1333420Z .................................................................................................... 5900/9570
2020-02-04T00:47:56.5268035Z .......................................................ii...i..ii...........i....................... 6000/9570
2020-02-04T00:48:16.7734392Z .................................................................................................... 6200/9570
2020-02-04T00:48:23.9472035Z .................................................................................................... 6300/9570
2020-02-04T00:48:23.9472035Z .................................................................................................... 6300/9570
2020-02-04T00:48:31.5553221Z ...................................................................................i..ii............ 6400/9570
2020-02-04T00:48:54.7621439Z .................................................................................................... 6600/9570
2020-02-04T00:49:02.8633191Z ...........................................................i........................................ 6700/9570
2020-02-04T00:49:04.9381000Z .................................................................................................... 6800/9570
2020-02-04T00:49:07.1978004Z .............................................................i...................................... 6900/9570
---
2020-02-04T00:50:43.9614813Z .................................................................................................... 7600/9570
2020-02-04T00:50:48.9378489Z .................................................................................................... 7700/9570
2020-02-04T00:50:55.4238653Z .................................................................................................... 7800/9570
2020-02-04T00:51:05.5594662Z .................................................................................................... 7900/9570
2020-02-04T00:51:11.5219864Z ......................iiiiiii.i..................................................................... 8000/9570
2020-02-04T00:51:25.3581059Z .................................................................................................... 8200/9570
2020-02-04T00:51:33.6019208Z .................................................................................................... 8300/9570
2020-02-04T00:51:47.3339296Z .................................................................................................... 8400/9570
2020-02-04T00:51:54.5320249Z .................................................................................................... 8500/9570
---
2020-02-04T00:53:43.0374233Z 2020-02-04T00:53:43.0375136Z ---- [ui] ui/ast-json/ast-json-output.rs stdout ----
2020-02-04T00:53:43.0375540Z diff of stdout:
2020-02-04T00:53:43.0375864Z 2020-02-04T00:53:43.0377266Z - {"module":{"inner":{"lo":0,"hi":0},"items":[{"attrs":[],"id":0,"span":{"lo":0,"hi":0},"vis":{"node":"Inherited","span":{"lo":0,"hi":0}},"ident":{"name":"core","span":{"lo":0,"hi":0}},"kind":{"variant":"ExternCrate","fields":[null]},"tokens":{"_field0":[[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["extern",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["crate",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["core",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":"Semi","span":{"lo":0,"hi":0}}]},"NonJoint"]]}}],"inline":true},"attrs":[],"span":{"lo":0,"hi":0}}
2020-02-04T00:53:43.0377790Z + {"module":{"inner":{"lo":0,"hi":0},"items":[{"attrs":[],"id":0,"span":{"lo":0,"hi":0},"vis":{"node":"Inherited","span":{"lo":0,"hi":0}},"ident":{"name":"core","span":{"lo":0,"hi":0}},"kind":{"variant":"ExternCrate","fields":[null]},"tokens":{"_field0":[[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["extern",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["crate",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["core",false]},"span":{"lo":0,"hi":0}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":"Semi","span":{"lo":0,"hi":0}}]},"NonJoint"]]}}],"inline":true},"attrs":[],"span":{"lo":0,"hi":0},"proc_macros":[]}
2020-02-04T00:53:43.0378386Z 2020-02-04T00:53:43.0378603Z 2020-02-04T00:53:43.0378844Z The actual stdout differed from the expected stdout.
2020-02-04T00:53:43.0379667Z Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/ast-json/ast-json-output/ast-json-output.stdout
2020-02-04T00:53:43.0379667Z Actual stdout saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/ast-json/ast-json-output/ast-json-output.stdout
2020-02-04T00:53:43.0380404Z To update references, rerun the tests and pass the `--bless` flag
2020-02-04T00:53:43.0381262Z To only update this specific test, also pass `--test-args ast-json/ast-json-output.rs`
2020-02-04T00:53:43.0381839Z error: 1 errors occurred comparing output.
2020-02-04T00:53:43.0381839Z error: 1 errors occurred comparing output.
2020-02-04T00:53:43.0382236Z failed to decode compiler output as json: line: {"module":{"inner":{"lo":218,"hi":236},"items":[{"attrs":[],"id":4294967040,"span":{"lo":218,"hi":236},"vis":{"node":"Inherited","span":{"lo":218,"hi":218}},"ident":{"name":"core","span":{"lo":231,"hi":235}},"kind":{"variant":"ExternCrate","fields":[null]},"tokens":{"_field0":[[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["extern",false]},"span":{"lo":218,"hi":224}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["crate",false]},"span":{"lo":225,"hi":230}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["core",false]},"span":{"lo":231,"hi":235}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":"Semi","span":{"lo":235,"hi":236}}]},"NonJoint"]]}}],"inline":true},"attrs":[],"span":{"lo":218,"hi":236},"proc_macros":[]}
2020-02-04T00:53:43.0382725Z output: {"module":{"inner":{"lo":218,"hi":236},"items":[{"attrs":[],"id":4294967040,"span":{"lo":218,"hi":236},"vis":{"node":"Inherited","span":{"lo":218,"hi":218}},"ident":{"name":"core","span":{"lo":231,"hi":235}},"kind":{"variant":"ExternCrate","fields":[null]},"tokens":{"_field0":[[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["extern",false]},"span":{"lo":218,"hi":224}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["crate",false]},"span":{"lo":225,"hi":230}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":{"variant":"Ident","fields":["core",false]},"span":{"lo":231,"hi":235}}]},"NonJoint"],[{"variant":"Token","fields":[{"kind":"Semi","span":{"lo":235,"hi":236}}]},"NonJoint"]]}}],"inline":true},"attrs":[],"span":{"lo":218,"hi":236},"proc_macros":[]}
2020-02-04T00:53:43.0383770Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-02-04T00:53:43.0384003Z 2020-02-04T00:53:43.0384242Z 2020-02-04T00:53:43.0384477Z failures:
2020-02-04T00:53:43.0384477Z failures:
2020-02-04T00:53:43.0384984Z [ui] ui/ast-json/ast-json-output.rs
2020-02-04T00:53:43.0386111Z 2020-02-04T00:53:43.0389122Z test result: FAILED. 9517 passed; 1 failed; 52 ignored; 0 measured; 0 filtered out
2020-02-04T00:53:43.0389380Z 2020-02-04T00:53:43.0392748Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:348:22
2020-02-04T00:53:43.0402221Z 2020-02-04T00:53:43.0402534Z 2020-02-04T00:53:43.0404794Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-7/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "7.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2020-02-04T00:53:43.0405470Z 2020-02-04T00:53:43.0405631Z 2020-02-04T00:53:43.0437283Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2020-02-04T00:53:43.0437962Z Build completed unsuccessfully in 0:58:37
2020-02-04T00:53:43.0437962Z Build completed unsuccessfully in 0:58:37
2020-02-04T00:53:43.0469102Z == clock drift check ==
2020-02-04T00:53:43.0486815Z local time: Tue Feb 4 00:53:43 UTC 2020
2020-02-04T00:53:43.6027043Z network time: Tue, 04 Feb 2020 00:53:43 GMT
2020-02-04T00:53:43.6032531Z == end clock drift check ==
2020-02-04T00:53:44.0076980Z 2020-02-04T00:53:44.0182827Z ##[error]Bash exited with code '1'.
2020-02-04T00:53:44.0193319Z ##[section]Finishing: Run build
2020-02-04T00:53:44.0217958Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/68814/merge to s
2020-02-04T00:53:44.0219948Z Task : Get sources
2020-02-04T00:53:44.0220008Z Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-02-04T00:53:44.0220064Z Version : 1.0.0
2020-02-04T00:53:44.0220102Z Author : Microsoft
2020-02-04T00:53:44.0220102Z Author : Microsoft
2020-02-04T00:53:44.0220141Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-02-04T00:53:44.0220184Z ==============================================================================
2020-02-04T00:53:44.4163471Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-02-04T00:53:44.4204372Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/68814/merge to s
2020-02-04T00:53:44.4308673Z Cleaning up task key
2020-02-04T00:53:44.4309369Z Start cleaning up orphan processes.
2020-02-04T00:53:44.4404453Z Terminate orphan process: pid (5195) (python)
2020-02-04T00:53:44.4610297Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

petrochenkov · 2020-02-04T06:30:36Z

r? @petrochenkov
There's must be some other issue here, this shouldn't be necessary.
I'll look what happens a bit later.

Aaron1011 · 2020-02-04T16:54:32Z

this shouldn't be necessary.

There's probably some way to fix the ordering mismatch, but I really don't think it's the right approach. We shouldn't have to care about the precise order in which we generate the harness - we've already been wrong twice.

Explicitly defining the order ensures that we'll never have to worry about this again. If we try to patch the visitation orders to match, I'm almost certain that we'll hit this kind of issue again. We might miss another weird corner case, or someone could change the AST/HIR such that the itertion order is different.

petrochenkov · 2020-02-09T13:50:20Z

(This will probably has to wait until the next weekend.)

petrochenkov · 2020-02-15T19:48:08Z

~~Ahaha, it turns out the order of items in HIR is not related to the order of items in AST at all, any similarity is purely coincidental.~~

Me and alexcrichton used some seriously outdated knowledge in #64528.
This PR is doing the right thing, but needs some more comments.

See #68814 (comment).

petrochenkov · 2020-02-15T20:03:45Z

r=me with comments added to AST as well.

Fixesrust-lang#68690 When we generate the proc macro harness, we now explicitly recorder the order in which we generate entries. We then use this ordering data to deserialize the correct proc-macro-data from the crate metadata.

Aaron1011 · 2020-02-15T20:52:22Z

@petrochenkov: I've added an additional comment.

petrochenkov · 2020-02-15T20:56:53Z

@bors r+

bors · 2020-02-15T20:56:54Z

📌 Commit 5164598 has been approved by petrochenkov

Mark-Simulacrum · 2020-02-15T22:21:28Z

@bors rollup=never

bors · 2020-02-16T01:46:09Z

⌛ Testing commit 5164598 with merge 2a0d1cb...

…nkov Record proc macro harness order for use during metadata deserialization Fixes#68690 When we generate the proc macro harness, we now explicitly recorder the order in which we generate entries. We then use this ordering data to deserialize the correct proc-macro-data from the crate metadata.

bors · 2020-02-16T05:08:02Z

☀️ Test successful - checks-azure
Approved by: petrochenkov
Pushing 2a0d1cb to master...

petrochenkov · 2020-02-16T12:08:24Z

I won't revert this PR since it had already landed, but the error was actually in the encoder.

tcx.hir().krate().items
=>
tcx.hir().krate().module.item_ids

Item order in HIR does repeat the item order in AST, Crate::items was just a wrong list to walk (it accumulates all items from all modules in the crate in semi-arbitrary order).

rust-highfive assigned varkor Feb 3, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 3, 2020

Aaron1011 force-pushed the fix/proc-macro-order-two branch from 591a367 to 5eb06a0Compare February 4, 2020 02:36

rust-highfive assigned petrochenkov and unassigned varkor Feb 4, 2020

petrochenkov reviewed Feb 15, 2020
View reviewed changes

Comment threadsrc/libsyntax/ast.rs Outdated

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 15, 2020

Aaron1011 added 2 commits February 15, 2020 15:48

Add additional comment
5164598

Aaron1011 force-pushed the fix/proc-macro-order-two branch from 5eb06a0 to 5164598Compare February 15, 2020 20:52

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 15, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Feb 16, 2020

bors merged commit 5164598 into rust-lang:masterFeb 16, 2020

petrochenkov mentioned this pull request Feb 16, 2020
O(log n) lookup of associated items by name #69072
Merged

Uh oh!

Uh oh!

Conversation

Aaron1011 commented Feb 3, 2020

Uh oh!

rust-highfive commented Feb 3, 2020

Uh oh!

rust-highfive commented Feb 4, 2020

Uh oh!

petrochenkov commented Feb 4, 2020

Uh oh!

Aaron1011 commented Feb 4, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrochenkov commented Feb 9, 2020

Uh oh!

petrochenkov commented Feb 15, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Feb 15, 2020

Uh oh!

Aaron1011 commented Feb 15, 2020

Uh oh!

petrochenkov commented Feb 15, 2020

Uh oh!

bors commented Feb 15, 2020

Uh oh!

Mark-Simulacrum commented Feb 15, 2020

Uh oh!

bors commented Feb 16, 2020

Uh oh!

bors commented Feb 16, 2020

Uh oh!

petrochenkov commented Feb 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Aaron1011 commented Feb 4, 2020•
edited
Loading

petrochenkov commented Feb 15, 2020•
edited
Loading