Overhaul `syntax::fold::Folder`. by nnethercote · Pull Request #58061 · rust-lang/rust

nnethercote · 2019-02-01T21:43:39Z

This PR changes syntax::fold::Folder from a functional style
(where most methods take a T and produce a new T) to a more
imperative style (where most methods take and modify a &mut T), and
renames it syntax::mut_visit::MutVisitor.

This makes the code faster and more concise.

nnethercote · 2019-02-01T22:22:54Z

@bors try

bors · 2019-02-01T22:23:07Z

⌛ Trying commit b4a6b98fb425e66926941efa55dce39aea400914 with merge 4d27e368a336fc53353c891ee0dc425399c612c0...

petrochenkov · 2019-02-01T22:40:57Z

@nnethercote
Hmm, have you seen #57662?

bors · 2019-02-02T00:33:51Z

☀️ Test successful - checks-travis
State: approved= try=True

nnethercote · 2019-02-02T00:36:12Z

I hadn't seen #57662. It's definitely overlapping with this PR. Notable differences, judging from a quick skim:

This PR changes Folder rather than introducing a new kind of AST walker. Therefore it has changes for all the existing Folder impls.
This PR has clear perf improvements (assuming the CI measurements match my local measurements).

nnethercote · 2019-02-02T00:38:52Z

The reason the new approach is faster is that an existing noop fold overwrites all the nodes with the same values. There are no reallocations due to the use of P::map and move_map, move_flat_map, but quite a lot of unnecessary writes. The new approach only overwrites data that needs overwriting, which for a typical pass is only a subset.

nnethercote · 2019-02-02T02:02:08Z

@rust-timer build 4d27e368a336fc53353c891ee0dc425399c612c0

rust-timer · 2019-02-02T02:02:09Z

Success: Queued 4d27e368a336fc53353c891ee0dc425399c612c0 with parent 23d8d0c, comparison URL.

rust-timer · 2019-02-02T05:15:27Z

Finished benchmarking try commit 4d27e368a336fc53353c891ee0dc425399c612c0

nnethercote · 2019-02-02T05:38:17Z

The perf results are similar to what I saw locally. Widespread improvements, mostly on "clean incremental" runs, with a best result of -3.4%.

nnethercote · 2019-02-02T08:03:58Z

Comparing this PR with #57662 some more, the handling of the T -> T cases is very similar. For example, noop_mod_expr in this PR is very similar to walk_expr in #57662 (though the former's use of destructuring avoids the need for all those ref mut patterns).

The handling of the other cases is where the differences are bigger. This PR keeps the existing structure for ones like fold_opt_expr (P<Expr> -> Option<P<Expr>>) and fold_stmt (Stmt -> SmallVec<[Stmt; 1]>); this minimizes change and means move_flat_map can still be used. (I did some ad hoc profiling with eprintln! and found that these non-T -> T cases are not hot, so keeping the existing structure seems ok.)

In contrast, #57662 introduces the Action type and a bunch of related machinery to handle these cases. It also uses the new walk_list/walk_list_mut macros instead of loops and/or move_flat_map to handle sequences.

nnethercote · 2019-02-02T08:28:19Z

A final summary!

This PR:

Modifies the existing AST visitor, Folder.
Modifies the handling of all the easy cases (T -> T) in Folder.
Leaves the tricky cases (non-T -> T) unchanged.
Uses loops and the existing move_flat_map to handle sequences.
Updates all the existing folders, and therefore reaps the full performance benefits.
Overall, the number of lines of code is barely changed.

#57662:

Adds a new AST visitor, MutVisitor.
Handles the easy cases in much the same way as in this PR.
Introduces a new way of handling the tricky case (via the Action type).
Doesn't handle some cases yet (e.g. visit_interpolated).
Uses the new walk_list/walk_list_mut macros to handle sequences.
Updates two of the existing folders, and therefore doesn't yet get the full performance benefits.
Adds a lot of new lines of code. Eventually, Folder should be removable, but the other added machinery (Action, etc.) will likely still result in an increase in code size.

Overall, I think this PR is more complete while involving less changes. But I'm biased :) Let me know if I've gotten any of the details wrong.

nnethercote · 2019-02-04T08:14:26Z

I have updated the code. It's now complete except for one thing: as the "njn:" comment in the code says, I think the syntax::fold::Folder name should change, though I'm not entirely sure what to. Suggestions are welcome.

r? @petrochenkov: the code is ready to review, but as well as making the main change, I have made a bunch of minor clean-ups. If you like I could pull out those clean-ups into separate commits preceding the main commit.

petrochenkov · 2019-02-04T11:37:07Z

Great work, thanks.
I see both this PR and #57662 as intermediate points to the end goal, but going through different trajectories.

I think I prefer this PRs trajectory. Why:

We can measure performance effects from the "noop folder vs noop visitor" and "tweaks to 1 -> many transformations" separately.
This PR is more "mechanical" and is based on the current Folder rather than Visitor, so it is more likely to preserve the exact behavior (bug compatibility, if you wish), e.g. around nonterminal folding.
I'm not entirely happy about some parts of Folder and after this PR they'll still be here and I'll actually be able to fix them, rather than have a "devil we don't know" after Add a MutVisitor for the AST #57662.

petrochenkov · 2019-02-04T11:40:00Z

Regarding naming.
For 1 -> 1 transformations I think we should take the terminology from #57662 - MutVisitor/visit_thing (especially considering #57662 (comment)).
For 1 -> many I think we should keep exactly what we have now (fold_thing/move_map/...) and don't rename anything, since it's an intermediate step anyway.

petrochenkov · 2019-02-04T11:49:55Z

@nnethercote

If you like I could pull out those clean-ups into separate commits preceding the main commit.

If that's not too annoying, could you split the PR into two commits

The part containing only mechanical changes to definitions fn fold_thing(&mut self, thing: Thing) -> Thing => fn visit_thing(&mut self, thing: &mut Thing) and minimal changes to their uses required for things to compile again.
Everything else (cleanups, moving code around, moving comments around, ...).

Specifically: - Remove dead methods: fold_usize, fold_meta_items, fold_opt_bounds. - Remove useless methods: fold_global_asm, fold_range_end. - Inline and remove unnecessary methods: fold_item_simple, fold_foreign_item_simple.

It doesn't need to return an `Option`.

It's simpler that way.

By eliminating some unnecessary methods, and moving/renaming some functions that look like they might be methods but aren't.

nnethercote · 2019-02-05T04:31:25Z

@petrochenkov: the new code is up.

I split out 8 clean-up patches, because that's what I would have wanted if I was the reviewer. Should make your job easier.
I switched to mut_visit::MutVisitor. (I feel like it should be visit_mut::VisitorMut to match IterMut, but there are already a couple of existing MutVisitor occurrences in the compiler.)
I kept the flat_map_t and filter_map_t names, because I think they make the code easier to understand than fold_t names. Also, you say this is an intermediate step, but intermediate steps sometimes hang around longer than people expect...
git unfortunately didn't treat the renaming of fold.rs as mut_visit.rs as a move. I guess the two files are too dissimilar. Sorry, I don't know how to make that better.

petrochenkov · 2019-02-05T10:06:43Z

@nnethercote

git unfortunately didn't treat the renaming of fold.rs as mut_visit.rs as a move.

The usual solution is to move the file renaming into a separate commit.
(Note that the Rust module can still be named mut_visit even if the file is named fold.rs.)

petrochenkov · 2019-02-05T10:07:33Z

r=me with the file renaming moved into a separate commit or without it.

This commit changes `syntax::fold::Folder` from a functional style (where most methods take a `T` and produce a new `T`) to a more imperative style (where most methods take and modify a `&mut T`), and renames it `syntax::mut_visit::MutVisitor`. The first benefit is speed. The functional style does not require any reallocations, due to the use of `P::map` and `MoveMap::move_{,flat_}map`. However, every field in the AST must be overwritten; even those fields that are unchanged are overwritten with the same value. This causes a lot of unnecessary memory writes. The imperative style reduces instruction counts by 1--3% across a wide range of workloads, particularly incremental workloads. The second benefit is conciseness; the imperative style is usually more concise. E.g. compare the old functional style: ``` fn fold_abc(&mut self, abc: ABC) { ABC { a: fold_a(abc.a), b: fold_b(abc.b), c: abc.c, } } ``` with the imperative style: ``` fn visit_abc(&mut self, ABC { a, b, c: _ }: &mut ABC) { visit_a(a); visit_b(b); } ``` (The reductions get larger in more complex examples.) Overall, the patch removes over 200 lines of code -- even though the new code has more comments -- and a lot of the remaining lines have fewer characters. Some notes: - The old style used methods called `fold_*`. The new style mostly uses methods called `visit_*`, but there are a few methods that map a `T` to something other than a `T`, which are called `flat_map_*` (`T` maps to multiple `T`s) or `filter_map_*` (`T` maps to 0 or 1 `T`s). - `move_map.rs`/`MoveMap`/`move_map`/`move_flat_map` are renamed `map_in_place.rs`/`MapInPlace`/`map_in_place`/`flat_map_in_place` to reflect their slightly changed signatures. - Although this commit renames the `fold` module as `mut_visit`, it keeps it in the `fold.rs` file, so as not to confuse git. The next commit will rename the file.

nnethercote · 2019-02-05T23:31:34Z

The usual solution is to move the file renaming into a separate commit.

Thank you for the suggestion, I did that.

@bors r=petrochenkov

bors · 2019-02-05T23:31:35Z

📌 Commit bfcbd23 has been approved by petrochenkov

bors · 2019-02-06T06:01:44Z

⌛ Testing commit bfcbd23 with merge 2596bc1...

…nkov Overhaul `syntax::fold::Folder`. This PR changes `syntax::fold::Folder` from a functional style (where most methods take a `T` and produce a new `T`) to a more imperative style (where most methods take and modify a `&mut T`), and renames it `syntax::mut_visit::MutVisitor`. This makes the code faster and more concise.

bors · 2019-02-06T08:42:41Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: petrochenkov
Pushing 2596bc1 to master...

Make `visit_clobber`'s impl safe This was originally introduced in rust-lang#58061 but I didn't see any perf discussion about it, so let's see what perf says. r? `@nnethercote`

petrochenkov self-assigned this Feb 1, 2019

petrochenkov mentioned this pull request Feb 2, 2019
Add a MutVisitor for the AST #57662
Closed

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 3, 2019

nnethercote force-pushed the overhaul-syntax-Folder branch from b4a6b98 to d05b178Compare February 4, 2019 08:11

petrochenkov removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Feb 4, 2019

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 4, 2019

nnethercote force-pushed the overhaul-syntax-Folder branch from d05b178 to f9d6e3eCompare February 5, 2019 04:05

nnethercote added 4 commits February 5, 2019 15:09

Remove some unnecessary ast:: and fold:: qualifiers.
b968641

Streamline Folder.
faa82eb
Specifically: - Remove dead methods: fold_usize, fold_meta_items, fold_opt_bounds. - Remove useless methods: fold_global_asm, fold_range_end. - Inline and remove unnecessary methods: fold_item_simple, fold_foreign_item_simple.

Fold some overlooked spans.
eea2dfe

Simplify fold_attribute.
f97e896
It doesn't need to return an `Option`.

nnethercote added 4 commits February 5, 2019 15:11

Change fold_qpath to fold_qself.
8909f70
It's simpler that way.

Neaten up fold_crate.
4730953

Streamline Folder some more.
372fe84
By eliminating some unnecessary methods, and moving/renaming some functions that look like they might be methods but aren't.

Various improvements in Folder impls.
970b5d1

nnethercote force-pushed the overhaul-syntax-Folder branch from f9d6e3e to 3263f02Compare February 5, 2019 04:24

nnethercote added 2 commits February 6, 2019 09:06

Rename fold.rs as mut_visit.rs.
bfcbd23

nnethercote force-pushed the overhaul-syntax-Folder branch from 3263f02 to bfcbd23Compare February 5, 2019 23:27

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 5, 2019

bors merged commit bfcbd23 into rust-lang:masterFeb 6, 2019

This was referenced Feb 6, 2019
The Genesis of Generic Germination #53645
Merged
Make the rustc driver and interface demand driven #56732
Merged
Support defining C compatible variadic functions #57760
Merged
libsyntax => 2018 #58125
Merged
Add const generics to the AST #58191
Merged

nnethercote deleted the overhaul-syntax-Folder branch February 6, 2019 21:36

matklad reviewed Feb 20, 2019
View reviewed changes

Comment threadsrc/libsyntax/mut_visit.rs

nnethercote mentioned this pull request Jul 25, 2019
"Aborted (core dumped)" with asm! after "error: expected expression, found <eof>" #62894
Closed

oli-obk mentioned this pull request Jul 10, 2024
Make visit_clobber's impl safe #127560
Merged

Uh oh!

Uh oh!

Conversation

nnethercote commented Feb 1, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nnethercote commented Feb 1, 2019

Uh oh!

bors commented Feb 1, 2019

Uh oh!

petrochenkov commented Feb 1, 2019

Uh oh!

bors commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

rust-timer commented Feb 2, 2019

Uh oh!

rust-timer commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 2, 2019

Uh oh!

nnethercote commented Feb 4, 2019

Uh oh!

petrochenkov commented Feb 4, 2019

Uh oh!

petrochenkov commented Feb 4, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrochenkov commented Feb 4, 2019

Uh oh!

nnethercote commented Feb 5, 2019

Uh oh!

petrochenkov commented Feb 5, 2019

Uh oh!

petrochenkov commented Feb 5, 2019

Uh oh!

nnethercote commented Feb 5, 2019

Uh oh!

bors commented Feb 5, 2019

Uh oh!

bors commented Feb 6, 2019

Uh oh!

bors commented Feb 6, 2019

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nnethercote commented Feb 1, 2019•
edited
Loading

petrochenkov commented Feb 4, 2019•
edited
Loading