Uh oh!
There was an error while loading. Please reload this page.
Expose incremental utf8 decoding APIs - #2408
Conversation
| -- | ||
| module Streamly.Unicode.Stream | ||
| ( | ||
| DecodeState |
There was a problem hiding this comment.
Before exposing:
- Check the naming
- Check good documentation
- Check naming consistency
- Optional: Benchmark and Test
6a20721 to
5fa6d57CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
528ba6e to
4e308f1Compare4e308f1 to
6c754f4Compare| -- For multi-byte characters, the decoding state indicates the number of bytes | ||
| -- remaining to complete the character. It is usually initialized to a non-zero | ||
| -- value corresponding to the number of bytes in the multi-byte character, e.g | ||
| -- DecodeState will be 1 for 2-bytes char. |
There was a problem hiding this comment.
This documentation does not make much sense to the user of the library. Need to write which APIs use this, where does this come from, what values need to be supplied etc. The information is to be used to correctly understand how to use the APIs.
| -- Calculate the code point value: Depending on the type of the leading byte, | ||
| -- extract the significant bits from each byte of the sequence and combine them | ||
| -- to form the complete code point value. The specific bit manipulations will | ||
| -- differ based on the number of bytes used. |
There was a problem hiding this comment.
We should not define a codepoint here. Need to say what it means in the context of the APIs and to be able to use the APIs.
d8470ed to
3692740Compare| -- ** Resumable UTF-8 Decoding | ||
| , DecodeError(..) | ||
| , DecodeState | ||
| , CodePoint |
There was a problem hiding this comment.
We can return a more intelligent decode error:
data DecodeUTF8Error = DecodeUTF8Incomplete Word32 | DecodeUTF8NonStarter Word8 | DecodeUTF8Invalid
This should be enough to build a resumable decoder. When resuming we should supply the Word32 from DecodeUTF8Incomplete.
No description provided.