add core::char::DecodeUtf8

[strake]

See issue

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[tbu-]

It would be more flexible to make the iterator's item type Option<char>, so the user of this API can decide what to do on bad characters: Erroring out, replacing it with the replacement character (.unwrap_or(replacement)), ...

[tbu-]

This can just be b as char, no need for the unsafe from_u32_unchecked.

[alexcrichton]

Idiomatically we tend to prefer to return a Result rather than an Option to indicate an error. This helps us give an entry point for an opaque struct to attach error information to.

[alexcrichton]

Thanks @strake! Could you also reexport this type from librustc_unicode? That way it'll make its way into std::char

[strake]

@alexcrichton All done. Not sure whether github notified you when i pushed...

[tbu-]

(@strake Github doesn't notify on pushes.)

[alexcrichton]

Could this be struct InvalidSequence(()) so we can backwards-compatibly add more variants? You may want to also derive PartialEq for it so assert_eq! can work.

[alexcrichton]

Thanks @strake!

[strake]

@alexcrichton Done. The check failed on Travis but passes on my machine; the failure seems to be in APT, not due to my modifications.

[alexcrichton]

Ah yeah travis is unfortunately having issues as of late, but otherwise looks good to me. This is tagged with T-libs to ensure it comes up during triage, and we'll discuss there!

[alexcrichton]

Thanks again for the PR @strake! The libs team got a chance to discuss this today and there were a few concerns, but the overall feeling was somewhat positive.

• Would it be possible to reuse code between std::str and here somehow to avoid open-coding decoding utf-8 twice?
• Do you have a particular use case in mind for this functionality beyond simply consistency with other APIs? (which is of course still good, however)

[strake]

Would it be possible to reuse code between std::str and here somehow to avoid open-coding decoding utf-8 twice?

All the methods i see in str take a &[u8] or a slice::Iter rather than an arbitrary Iterator, and i'd prefer to take an arbitrary Iterator.

Do you have a particular use case in mind for this functionality beyond simply consistency with other APIs? (which is of course still good, however)

Well, i'm writing a text editor which must deal with strings (e.g. file paths) which are UTF-8-coded by default but potentially invalid, in which case it ought to show the user what valid subsequences of the path it can. (I consider crashing on allocational failure ill-behavior in a text editor, so i am not using std.)

[alexcrichton]

Would it be possible to refactor a bit to share an implementation? It's not obvious unfortunately that this is correct as it seems some subtle logic is happening here, but it'd be nice to lean on the existing code.

Also just out of curiosity, but how come str::from_utf8 won't suffice for this use case? Presumably the bytes are held in memory somewhere?

[strake]

It's not obvious unfortunately that this is correct as it seems some subtle logic is happening here

I actually since found some helper functions in str we can use here. I think it clarifies the DecodeUtf8::next code. Sorry i missed this last time.

how come str::from_utf8 won't suffice for this use case?

If its input has any invalid sequences, str::from_utf8 fails completely. I need to decode the valid parts of a byte string or byte stream which potentially includes invalid sequences.

[bors]

☔ The latest upstream changes (presumably #34399) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

Hm yeah it's probably good to use at least some helper functions, but I believe we on the libs team were thinking more of wholesale using an implementation in str or something like that. It looks like there's still plenty of opportunity to go wrong in the DecodeUtf8 iterator as a significant amount of it's still open-coded.

It may not be the case that it's possible to share as str::from_utf8 is so specialized, but it was a passing thought of ours. In general we've avoided util modules, so if it comes to that it's probably not worth sharing if it's still open coded as much as it is still.

[strake]

It may not be the case that it's possible to share as str::from_utf8 is so specialized, but it was a passing thought of ours.

Yeah, i couldn't see how to so modify run_utf8_validation as it is so specialized to to validation rather than decoding, and next_code_point has a comment saying "Performance is sensitive to the exact formulation here" so i dare not modify it.

[tbu-]

The code does not decode UTF-8 according to the spec: https://encoding.spec.whatwg.org/#utf-8-decoder. It eats characters after an invalid byte sequence and does not check for overlong encodings. I suggest you to just implement the described algorithm, that also makes it easier to verify it's doing the right thing.

[strake]

I modified it to not eat characters and to check for too-long sequences. It now returns the reason why a sequence is invalid.

I looked at the algorithm on that page, and it seems more complicated, having more internal state, and is not of the same form as Iterator::next, potentially returning multiple tokens or "continue".

[alexcrichton]

Stylistically we tend to have multi-line closures always have braces around them, putting this if on the next line.

You could make this similarly intended by doing:

foo.map(|b| {if condition {returnOk(...)}
...
})

[alexcrichton]

@rust-lang/libs thoughts on having another utf-8 decoding loop by hand in libcore? It seems like our previous ones don't quite apply, and if it at least clearly follows a spec it won't exactly change that often so perhaps not the worst!

[tbu-]

I think it shouldn't return precise errors. This makes it hard to change the underlying algorithm and is probably not useful to the user of the API anyway.

[brson]

@alexcrichton I don't really care about having yet another UTF-8 decoder. The one test looks light - @strake does the test cover all the cases?

[strake]

• Braced closure
• No more precise errors
• More tests, which i think cover all the cases now

[BurntSushi]

Are any of these test cases checking for overlong sequences?

[strake]

Yes, [0xC1, 0x81] is an overlong sequence for 'A'.

[aturon]

@alexcrichton I don't object to having an additional hand-written loop. I expect us to grow this kind of micro-optimization over time in std.

[alexcrichton]

Ok, @strake could you also squash the commits down? After that I'll r+

[strake]

@alexcrichton yes, done

[alexcrichton]

@bors: r+ 837029f

Thanks!

[bors]

⌛ Testing commit 837029f with merge 6998018...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing 6998018 to master...