Do not remove closures during ReplaceBodyWithLoop pass

[marmeladema]

Fixes#71820

[rust-highfive]

r? @ecstatic-morse

(rust_highfive has picked a reviewer for you, use r? to override)

[marmeladema]

cc @petrochenkov

[ecstatic-morse]

What happens if I have code like the following:

fnfoo(){let x = |()| {};}

I think the closure still gets removed because it's in a StmtKind::Local. Is this what we want? If not, we should map these to bare StmtKind::Exprs instead.

[marmeladema]

What happens if I have code like the following:

fnfoo(){let x = |()| {};}

I think the closure still gets removed because it's in a StmtKind::Local. Is this what we want? If not, we should map these to bare StmtKind::Exprs instead.

Yes the closure would probably gets removed, but i guess its the intended behavior of the simplification pass? The problem in my case is a closure with an exported macro inside. I tried various other code layout (useless blocks, nested closures, etc) but was not able to reproduce the bug when applying this change.

I think the real question is whether or not this simplification pass is needed at all? It looks like a weird hack and can imagine a lot of different bugs due to it.

[marmeladema]

What happens if I have code like the following:

fnfoo(){let x = |()| {};}

So i just tested your example with a macro inside, and indeed, you are right, i can reproduce the bug.

I think the closure still gets removed because it's in a StmtKind::Local. Is this what we want? If not, we should map these to bare StmtKind::Exprs instead.

Sorry, I am not sure i fully understand why you mean. Could you explain a bit more? Or maybe provide pseudo-code / example?

[ecstatic-morse]

My point is that this won't fix #71820 if you have the following, since you're not looking at all places an expression could appear in a Stmt.

fnfoo(){let x = |()| {#[macro_export]macro_rules! foo {}};}

Yeah, so I would add an expr() -> Option<&Expr> method to StmtKind as I mentioned above. You'll also look for the Expr in the Local variant of StmtKind. Then change the filter into a filter_map with a body like stmt.expr().map(|expr| Stmt::new(StmtKind::Expr(expr)) (this is just pseudo-code, you'll need to map items to themselves as well).

[ecstatic-morse]

I think the real question is whether or not this simplification pass is needed at all? It looks like a weird hack and can imagine a lot of different bugs due to it.

I can't speak to how rustdoc is using this. Perhaps someone else knows? I'd be surprised if leaving in closures caused anything to break, so I think this approach is fine. That said, you should mention why you're doing this in a comment, since this is very subtle.

edit:

Now that I think of it, there are probably more expressions that will introduce a new HirId just as closures do. For example, async blocks probably work like this?

[marmeladema]

Now that I think of it, there are probably more expressions that will introduce a new HirId just as closures do. For example, async blocks probably work like this?

Just tried it, and yes, the bug also happens with async block. Basically, any AST node that can contain a macro and that is going to get removed (ie: not an Item node) would trigger the issue.

[ecstatic-morse]

So basically we need to make the AST visitor more robust. Specifically, it should only remove AST nodes that contain no nested items. Does this seem feasible? Unfortunately, ReplaceBodyWithLoop is getting further away from doing what it promises. I think we will need to port rustdoc to a hypothetical libsyntax2 to remove it, however. See #43348 for context on why it is needed for now.

[marmeladema]

I've improved the implem based on your feedbacks @ecstatic-morse ! Thanks a lot.
Should we create a ticket for the proper fix and merge this PR in the mean time?

[ecstatic-morse]

I'm good with that. I'll take another look when CI is green.

[petrochenkov]

I'm still not sure why this issue happened only with macros but not other items.

[ecstatic-morse]

@petrochenkov I have very little intuition for how rustdoc works, but perhaps it is because we run this pass after macro expansion but before AST lowering? Maybe inner items that are not macro definitions are never seen by rustdoc but macro definitions are collected somewhere that is eventually processed. I would have assumed that rustdoc had tests for docs on nested items, however.

[rust-highfive]

The job x86_64-gnu-llvm-8 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

##[section]Starting: Linux x86_64-gnu-llvm-8
##[section]Starting: Initialize job
Agent name: 'Azure Pipelines 3'
Agent machine name: 'fv-az578'
Current agent version: '2.168.2'
##[group]Operating System
16.04.6
LTS
LTS
##[endgroup]
##[group]Virtual Environment
Environment: ubuntu-16.04
Version: 20200430.2
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu16/20200430.2/images/linux/Ubuntu1604-README.md
##[endgroup]
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Download all required tasks.
Downloading task: Bash (3.163.1)
Checking job knob settings.
Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} Start tracking orphan processes.
##[section]Finishing: Initialize job
##[section]Starting: Configure Job Name
==============================================================================
---
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /home/vsts/work/_temp/16f085c6-138e-43de-a0d4-cee5ea3971de.sh
##[section]Finishing: Disable git automatic line ending conversion
##[section]Starting: Checkout rust-lang/rust@refs/pull/72088/merge to s
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
---
##[command]git remote add origin https://github.com/rust-lang/rust
##[command]git config gc.auto 0
##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
##[command]git config --get-all http.proxy
##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/72088/merge:refs/remotes/pull/72088/merge
---
---> cb2676f08729
Step 5/8 : ENV RUST_CONFIGURE_ARGS --build=x86_64-unknown-linux-gnu --llvm-root=/usr/lib/llvm-8 --enable-llvm-link-shared --set rust.thin-lto-import-instr-limit=10
---> Using cache
---> df25ce111862
Step 6/8 : ENV SCRIPT python2.7 ../x.py test --exclude src/tools/tidy && python2.7 ../x.py test src/test/mir-opt --pass=build --target=armv5te-unknown-linux-gnueabi && python2.7 ../x.py test src/tools/tidy
---> 599b9ac96b27
Step 7/8 : ENV NO_DEBUG_ASSERTIONS=1
---> Using cache
---> 091087e35a36
---
Compiling fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
Compiling chalk-rust-ir v0.10.0
Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
Compiling chalk-solve v0.10.0
Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
Compiling fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
Compiling chalk-rust-ir v0.10.0
Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
Compiling chalk-solve v0.10.0
Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
.....................................................i.............................................. 1800/10160
.................................................................................................... 1900/10160
.......................................................................i..i......................... 2000/10160
.................................................................................................... 2100/10160
.............................................................iiiii.................................. 2200/10160
.................................................................................................... 2400/10160
.................................................................................................... 2500/10160
.................................................................................................... 2600/10160
.................................................................................................... 2700/10160
---
.................................................................................................... 5200/10160
.................................................................................................... 5300/10160
........................i........................................................................... 5400/10160
.................i.................................................................................. 5500/10160
........................ii.ii........i...i.......................................................... 5600/10160
.........................................................................i.......................... 5800/10160
.................................................................................................... 5900/10160
....................ii.....................................i........................................ 6000/10160
.................................................................................................... 6100/10160
.................................................................................................... 6100/10160
.................................................................................................... 6200/10160
.................................................................................ii...i..ii......... 6300/10160
.................................................................................................... 6500/10160
.................................................................................................... 6600/10160
.................................................................................................... 6700/10160
.................................................................................................... 6700/10160
..............i..ii................................................................................. 6800/10160
.................................................................................................... 7000/10160
....................................................................i............................... 7100/10160
.................................................................................................... 7200/10160
.................................................................................................... 7300/10160
---
.................................................................................................... 8100/10160
.................................................................................................... 8200/10160
...................................................................................i................ 8300/10160
.................................................................................................... 8400/10160
.....................................iiiiii.iiiii.i................................................. 8500/10160
.................................................................................................... 8700/10160
.................................................................................................... 8800/10160
.................................................................................................... 8900/10160
.................................................................................................... 9000/10160
---
Suite("src/test/codegen") not skipped for "bootstrap::test::Codegen" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
running 188 tests
iiii......i.............ii.i..........i...............................i..i..................i....i.. 100/188
..........i.i.i...iii..iiiiiiiiiiiiiiii........................iii..............ii......
finished in 5.565
Suite("src/test/codegen-units") not skipped for "bootstrap::test::CodegenUnits" -- not in ["src/tools/tidy"]
Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
running 9 tests
iiiiiiiii
finished in 0.149
Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
running 115 tests
iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i..i.......ii.i.ii.. 100/115
...iiii.....ii.
finished in 15.473
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
---
Doc-tests alloc
error[E0425]: cannot find function `pred` in this scope
--> /checkout/src/liballoc/collections/btree/set.rs:1135:52
|
1135 | let mut mapped_pred = |k: &T, _v: &mut ()| pred(k);
| ^^^^ help: you might have meant to use the available field: `self.pred`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0425`.
error: test failed, to rerun pass '--doc'
---
local time: Mon May 11 23:59:40 UTC 2020
network time: Mon, 11 May 2020 23:59:40 GMT
== end clock drift check ==
##[error]Bash exited with code '1'.
##[section]Finishing: Run build
##[section]Starting: Checkout rust-lang/rust@refs/pull/72088/merge to s
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: rust-lang/rust (GitHub)
##[section]Finishing: Checkout rust-lang/rust@refs/pull/72088/merge to s
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (4398) (python)
##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

[marmeladema]

@petrochenkov@ecstatic-morse My understanding is that the bug is triggered because we try to access the HirId of a parent DefId. This operation does not seem that common(ie: specifically getting the parent DefId then getting its HirId). Moreover its triggered by macros because during lowering, for the ones that are exported, they are registered in a specific "exported macros" vector which is processed later on for visibility checks.

[marmeladema]

So keeping the closures, even in the StmtKind::Local case might not the right solution after all because rustdoc now fails in liballoc:

let pred = &mut self.pred;
let mut mapped_pred = |k: &T, _v: &mut ()| pred(k);

First let statement is removed, but second is kept and fails to compile because pred does not exist anymore.

Another idea i had, was to monkey patch the parent DefId of exported macros during lowering. I don't really know the implications of such an approach.

[ecstatic-morse]

So basically we need to make the AST visitor more robust. Specifically, it should only remove AST nodes that contain no nested items.

@marmeladema This is what I was hinting at with the comment above. The AST visitor approach will only work if you remove all expressions that don't have nested items. Currently, the AST visitor is only handling block expressions, which means you're missing stuff like the single-expression closure above.

Monkey patching DefIds does seem more robust than stripping AST nodes, but I also don't know how you would go about it. My position is that ReplaceBodyWithLoop is already a hack, so we should keep any fixes internal and avoid changing other parts of the compiler just to make it work.

[crlf0710]

@marmeladema Ping from triage. The CI failure needs to be fixed.

[marmeladema]

@crlf0710 its broken but i don't know what is the best way to fix yet. Should I move the PR to draft for the time being?

[ecstatic-morse]

@marmeladema Do you mind if I take this over?

[marmeladema]

Be my guest 👍 For the record, there are some important refactor work going on right now like #72284 that might help fixing this.

[ecstatic-morse]

For the record, there are some important refactor work going on right now like #72284 that might help fixing this.

Sweet. I'll wait for that to land.

[petrochenkov]

#72284 seems unrelated to this PR.

[ecstatic-morse]

@marmeladema Seems like that PR landed. Is this still desirable? Or is there a different approach in the works?

[marmeladema]

So #72284 was indeed unrelated, i got confused with another macro def map that exists during lowering .. sorry for the confusion. I haven't time to try any other approach, feel free to pick that up @ecstatic-morse you will probably be more efficient that me at solving that issue :)

[jyn514]

Let me know if I can help with this, #71820 is blocking #73101.

[ecstatic-morse]

Superseded by #73103.