Chunked iterator API like `Utf8Chunks`

[thaliaarchi]

I've wanted chunked UTF-8 decoding twice recently for different escaping routines, and have used simdutf8::compat::from_utf8 in a loop to achieve that. I would really like to be able to use an API like Utf8Chunks from #[feature(utf8_lossy)] or bstr::Utf8Chunks, but with the faster validation of this crate. Utf8Chunks avoids the disconnect between the length of the valid prefix and the prefix as a string. Additionally, I suspect an API for this could avoid some overhead from decoding in a loop.

I ended up writing something close to this:

pub fn from_utf8_lossy(mut v: &[u8]) -> Cow<'_, str> {
 match simdutf8::compat::from_utf8(v) {
 Ok(s) => s.into(),
 Err(mut err) => {
 let mut cleaned = String::with_capacity(v.len());
 loop {
 cleaned.push_str(unsafe { str::from_utf8_unchecked(&v[..err.valid_up_to()]) });
 cleaned.push_str("\u{FFFD}");
 if let Some(error_len) = err.error_len() {
 v = &v[err.valid_up_to() + error_len..];
 match simdutf8::compat::from_utf8(v) {
 Ok(v) => cleaned.push_str(v),
 Err(err1) => {
 err = err1;
 continue;
 }
 }
 }
 break cleaned.into();
 }
 }
 }
}

Compare to the stdlib implementation of String::from_utf8_lossy, which avoids any direct offset fiddling and unchecked conversions:

pub fn from_utf8_lossy(v: &[u8]) -> Cow<'_, str> {
 let mut iter = Utf8Chunks::new(v);

 let first_valid = if let Some(chunk) = iter.next() {
 let valid = chunk.valid();
 if chunk.invalid().is_empty() {
 debug_assert_eq!(valid.len(), v.len());
 return Cow::Borrowed(valid);
 }
 valid
 } else {
 return Cow::Borrowed("");
 };

 const REPLACEMENT: &str = "\u{FFFD}";

 let mut res = String::with_capacity(v.len());
 res.push_str(first_valid);
 res.push_str(REPLACEMENT);

 for chunk in iter {
 res.push_str(chunk.valid());
 if !chunk.invalid().is_empty() {
 res.push_str(REPLACEMENT);
 }
 }

 Cow::Owned(res)
}