Implement 1581 (FusedIterator) by Stebalien · Pull Request #35656 · rust-lang/rust

Stebalien · 2016-08-13T22:20:01Z

Implement on patterns. See Tracking issue for string patterns #27721 (comment).
Handle OS Iterators. A bunch of iterators (Args, Env, etc.) in libstd wrap platform specific iterators. The current ones all appear to be well-behaved but can we assume that future ones will be?
Does someone want to audit this? On first glance, all of the iterators on which I implemented FusedIterator appear to be well-behaved but there are a lot of them so a second pair of eyes would be nice.
I haven't touched rustc internal iterators (or the internal rand) because rustc doesn't actually call fuse().
FusedIterator can't be implemented on std::io::{Bytes, Chars}.

Closes: #35602 (Tracking Issue)
Implements: rust-lang/rfcs#1581

rust-highfive · 2016-08-13T22:20:08Z

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

Stebalien · 2016-08-13T22:21:58Z

Please wait for the Travis run to complete before handing off to bors. I haven't done a full bootstrap build yet.

Stebalien · 2016-08-13T22:23:12Z

r? @alexcrichton

alexcrichton · 2016-08-14T07:06:21Z

Thanks for the PR @Stebalien! I think it's fine to start conservatively so we don't necessarily need to tag all the iterators today.

I wonder though if this could use a similar trick to zip specialization to get rid of the done bool storage in the structure as well? In theory an already fused iterator doesn't need that boolean.

Stebalien · 2016-08-14T14:59:28Z

I wonder though if this could use a similar trick to zip specialization to get rid of the done bool storage in the structure as well? In theory an already fused iterator doesn't need that boolean.

You can but it makes the implementation a lot messier. See #32999 (comment). However, after thinking about it a bit, it does make the documentation simpler (we can claim "no overhead" without qualifying what we mean by overhead). I've added a commit that does this based on @apasel422's comment (#32999 (comment)).

I think it's fine to start conservatively so we don't necessarily need to tag all the iterators today.

I'm worried that FusedIterator is so invisible that that nobody will even think to implement it. That is, people add implementations of ExactSizeIterator etc. all the time because they're needed to satisfy bounds but, by design, FusedIterator is practically invisible.

apasel422 · 2016-08-14T15:06:41Z

My original suggestion of using
structRealFuse<I>{iter:Option<I>,}
can save even more space if the underlying iterator contains a NonZero that lets the Option space optimization kick in, but it does have the side effect of changing the time at which the underlying iterator's destructor is called.

but it does have the side effect of changing the time at which the underlying iterator's destructor is called
That's why I switched to this version. The real problem is that the two specializations of Fuse would behave very differently in an unexpected manner.
For example, say you have an iterator iterator holds a lock but you always use it under a Fuse wrapper. Now, someone implements FusedIterator on the underlying iterator. Suddenly, the inner iterator wouldn't be eagerly dropped and your code could deadlock.

Makes sense!

Stebalien · 2016-08-15T19:34:12Z

I've addressed your feedback. I can fixup the commits into a logical set of if you want but left them "historically accurate" for review purposes.

alexcrichton · 2016-08-16T00:26:44Z

Stylistically we tend to put the { here on the next line like:
fnfoo()where// ...{// ...}

alexcrichton · 2016-08-16T00:27:19Z

Looks like the compile failure on travis may be legitimate?

Stebalien · 2016-08-16T15:51:26Z

@alexcrichton done (I don't know if you're notified of commits...).

alexcrichton · 2016-08-16T18:35:21Z

Ok, look good to me! Can you add a few tests just as a smoke test as well that .fuse().fuse() doesn't cause an ICE or an assertion to get tripped anywhere?

Other than that with a squash this looks good to me.

apasel422 · 2016-08-16T18:40:27Z

I suspect that this is going to change Fuse<I> from covariant in I to invariant in I: https://is.gd/btpbYx

That's not good. Is there any way to fix that?

Ah. I see how to fix this... Actually, it might make the code cleaner.

I'll be curious to see your approach, as I earlier reported another instance of this issue here: #30642 (comment)

Oh dear (#35727).
Well, I thought that I could do the same thing as the Zip specialization and put the flag in a separate static data field (keeping the iterator in the main struct) but apparently that doesn't appease rust:
structFuse<I>{iter:I,fused: <IasFuseImpl>::Data,}
There must be some way to do this...

bors · 2016-08-17T20:13:00Z

☔ The latest upstream changes (presumably #35747) made this pull request unmergeable. Please resolve the merge conflicts.

This trait can be used to avoid the overhead of a fuse wrapper when an iterator is already well-behaved. Conforming to: RFC 1581 Closes: rust-lang#35602

Stebalien · 2016-08-18T16:17:51Z

So, I've thrown away the associated type specialization (breaking change), squashed, and rebased.

Stebalien · 2016-08-18T16:18:30Z

Note. I've yet to add sanity tests. I'll do that soon (hopefully).

alexcrichton · 2016-08-18T21:02:07Z

Hm I would personally be very wary of making the methods of the iterator trait default just yet. Is there any other alternative?

Stebalien · 2016-08-18T21:40:29Z

This makes the methods of the implementation of Iterator on Fuse default. Given that these are both defined in core, coherence (should) disallow further specializations of Iterator on Fuse outside of core.

alexcrichton · 2016-08-19T04:55:26Z

Oh whoops! Sorry about that I did indeed misread. @aturon just to confirm, but that's right in that no one else external to the standard library can specialize further here?

alexcrichton · 2016-08-23T05:45:35Z

Discussed in @rust-lang/libs triage today and the conclusion was that this is good to go. Thanks @Stebalien!

@bors: r+

bors · 2016-08-23T05:45:36Z

📌 Commit de91872 has been approved by alexcrichton

Stebalien · 2016-08-23T14:25:30Z

@alexcrichton (FYI, I still haven't added the sanity tests yet but thesis is due on 8 days so I'd rather not work on rust-proper until after that deadline).

bors · 2016-08-23T14:46:53Z

⌛ Testing commit de91872 with merge 0bd99f9...

Implement 1581 (FusedIterator) * [ ] Implement on patterns. See #27721 (comment). * [ ] Handle OS Iterators. A bunch of iterators (`Args`, `Env`, etc.) in libstd wrap platform specific iterators. The current ones all appear to be well-behaved but can we assume that future ones will be? * [ ] Does someone want to audit this? On first glance, all of the iterators on which I implemented `FusedIterator` appear to be well-behaved but there are a *lot* of them so a second pair of eyes would be nice. * I haven't touched rustc internal iterators (or the internal rand) because rustc doesn't actually call `fuse()`. * `FusedIterator` can't be implemented on `std::io::{Bytes, Chars}`. Closes: #35602 (Tracking Issue) Implements: rust-lang/rfcs#1581

bors · 2016-08-23T18:12:39Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-no-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-no-mir, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing 0bd99f9 to master...

Add a test to ensure Fuse stays covariant When rust-lang#70502 attempted to specialize the data types in `Fuse`, one of the problems we found was that it broke variance. This was also realized when `Fuse` was first added, rust-lang#35656 (diff), but now this PR adds a test so we don't forget again.

rust-highfive assigned aturon Aug 13, 2016

rust-highfive assigned alexcrichton and unassigned aturon Aug 13, 2016

Stebalien force-pushed the fused branch from 57b9ea3 to ace40e8Compare August 14, 2016 03:18

apasel422 reviewed Aug 14, 2016
View reviewed changes

alexcrichton mentioned this pull request Aug 15, 2016
Annotate all core iterators with their fusedness #34343
Closed

alexcrichton added the T-libs-api [DEPRECATED; DO NOT USE] label Aug 16, 2016

alexcrichton reviewed Aug 16, 2016
View reviewed changes

apasel422 reviewed Aug 16, 2016
View reviewed changes

Stebalien force-pushed the fused branch from 979ed71 to da39251Compare August 18, 2016 16:13

Add a FusedIterator trait.
de91872
This trait can be used to avoid the overhead of a fuse wrapper when an iterator is already well-behaved. Conforming to: RFC 1581 Closes: rust-lang#35602

Stebalien force-pushed the fused branch from da39251 to de91872Compare August 18, 2016 16:17

bors merged commit de91872 into rust-lang:masterAug 23, 2016

This was referenced Aug 23, 2016
Rollup of 15 pull requests #35944
Closed
Implement std::convert traits for char #35755
Merged

Stebalien mentioned this pull request Aug 28, 2016
Type inference ICE: assertion failed: !self.obligations_in_snapshot.get()#36053
Closed

Mark-Simulacrum mentioned this pull request Feb 19, 2019
Type inference fails even though all types are known #58517
Open

cuviper mentioned this pull request Apr 4, 2020
Specialize internal state of Fuse depending on underlying iterator #70502
Closed

cuviper mentioned this pull request Jun 2, 2020
Add a test to ensure Fuse stays covariant #72902
Merged

Uh oh!

Uh oh!

Conversation

Stebalien commented Aug 13, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-highfive commented Aug 13, 2016

Uh oh!

Stebalien commented Aug 13, 2016

Uh oh!

Stebalien commented Aug 13, 2016

Uh oh!

alexcrichton commented Aug 14, 2016

Uh oh!

Stebalien commented Aug 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StebalienAug 14, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Stebalien commented Aug 15, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexcrichton commented Aug 16, 2016

Uh oh!

Stebalien commented Aug 16, 2016

Uh oh!

alexcrichton commented Aug 16, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bors commented Aug 17, 2016

Uh oh!

Stebalien commented Aug 18, 2016

Uh oh!

Stebalien commented Aug 18, 2016

Uh oh!

alexcrichton commented Aug 18, 2016

Uh oh!

Stebalien commented Aug 18, 2016

Uh oh!

alexcrichton commented Aug 19, 2016

Uh oh!

alexcrichton commented Aug 23, 2016

Uh oh!

bors commented Aug 23, 2016

Uh oh!

Stebalien commented Aug 23, 2016

Uh oh!

bors commented Aug 23, 2016

Uh oh!

bors commented Aug 23, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Stebalien commented Aug 13, 2016•
edited
Loading

StebalienAug 14, 2016•
edited
Loading