Detect Windows-1252 mojibake of U+2000-block symbols (fixes #233) - #235
Open
sebollin wants to merge 1 commit into
Open
Detect Windows-1252 mojibake of U+2000-block symbols (fixes #233)#235sebollin wants to merge 1 commit into
sebollin wants to merge 1 commit into
Conversation
fix_encoding("â…“") returned the input unchanged instead of "⅓". The bytes
are E2 85 93, the UTF-8 encoding of U+2153, read as Windows-1252. The
badness heuristic misses it because the second byte lands in the "common"
category, which is deliberately not treated as evidence on its own.
The obvious general rule -- [accented] [common] [punctuation] -- cannot be
used: it also matches "Charlotte Brontë…”", which the test suite already
guards against with a named negative case, and would turn it into a Korean
syllable. The difference is the lead byte: 'â' is 0xE2, which opens the
punctuation and symbol blocks, while 'ë' is 0xEB, which opens Hangul.
So the new alternative is restricted to 'â', in the same spirit as the
existing rule that is restricted to [ÂÃÎÐ] rather than to a whole category.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes#233.
fix_encoding("â…“")returns the input unchanged instead of⅓. The bytes areE2 85 93, which is the UTF-8 encoding of U+2153 read as Windows-1252. The heuristic misses it because the second byte lands incommon, andcommonis deliberately not treated as evidence on its own.The obvious general rule doesn't work. I started with
and it fails the negative case that's already in the suite for exactly this shape:
ë…”andâ…“are the same shape. What separates them is the lead byte:âis0xE2, which opens the punctuation and symbol blocks, whileëis0xEB, which opens Hangul.So I restricted the new alternative to
â, in the same spirit as the existing rule that is restricted to[ÂÃÎÐ]rather than to a whole category. Brontë is left alone,â…“andâ…›come out right, and the suite goes from 351 to 353 passing with the same 10 xfails.