Skip to content

Anchor the utf-8-variants regexes at the true end of the buffer (fixes #236) - #237

Open
sebollin wants to merge 1 commit into
rspeer:mainfrom
sebollin:eol-anchors-236
Open

Anchor the utf-8-variants regexes at the true end of the buffer (fixes #236)#237
sebollin wants to merge 1 commit into
rspeer:mainfrom
sebollin:eol-anchors-236

Conversation

@sebollin

Copy link
Copy Markdown

Fixes#236.

CESU8_EXPR, SURROGATE_EXPR and NULL_EXPR use $ where they mean "the buffer ends here", but in Python $ also matches just before a trailing newline. A \n at the end of the input therefore gets consumed as part of the match:

>>>b"\xc0\n".decode("utf-8-variants") # before'\x00'>>>b"\xed\xa0\xbd\xed\xb8\n".decode("utf-8-variants")
'😊'

The second one is the bad case: _buffer_decode_surrogates builds the codepoint from input[5] & 0x3F, so the newline byte becomes the low six bits and an invalid five-byte sequence decodes to U+1F60A instead of raising.

This replaces the seven $ anchors with \Z, which only matches at the very end of the string. The truncated-input behaviour the $ alternatives exist for is unchanged, since a truncated buffer ends where \Z matches.

After the change both examples raise UnicodeDecodeError, and ftfy.fix_encoding("Hi guys í ½í¸\n") keeps the newline instead of returning 'Hi guys 😊'.

Testing. The existing suite passes exactly as it does on main (336 passed, 10 xfailed under tests/; the same 7 environment-related failures appear with and without the patch on my machine — the CLI entry point and one wcwidth doctest).

I also ran a differential battery of 65,268 byte sequences — the full sweep of ED xx yy, C0 followed by every byte, six-byte CESU-8 candidates, truncations, overlong forms, valid UTF-8 mixed with CESU-8, and ~30k noise strings weighted towards 0xED/0xC0 — comparing the codec before and after. Exactly four sequences change, and all four are the bug:

b"\xc0\n" '\x00' -> UnicodeDecodeError
b"\x0a\xc0\n" '\n\x00' -> UnicodeDecodeError
b"\xc3\xa9\xc0\n" 'é\x00' -> UnicodeDecodeError
b"\xed\xa0\xbd\xed\xb8\n" '😊' -> UnicodeDecodeError

Sequences that should still decode are untouched, including a newline after a complete sequence (b"\xed\xa0\xbd\xed\xb8\x8d\n" and b"\xc0\x80\n"), and feeding the incremental decoder one byte at a time gives the same result as decoding the whole buffer.

Found while porting the codec to R for lupa, which needs CESU-8 decoding without a Python dependency.

Python's $ also matches before a trailing newline, so a \n at the end of
the input was consumed as part of a CESU-8 sequence or a Java null. Use \Z,
which only matches at the very end, keeping the intent for truncated input
in the streaming decoder.
Fixesrspeer#236
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

utf-8-variants: the $ anchors swallow a trailing newline

1 participant

@sebollin