Add utilities to "clean" surrogate code points from strings

[ncoghlan]

BPO	18814
Nosy	@malemburg, @ncoghlan, @pitrou, @vstinner, @ezio-melotti, @stevendaprano, @bitdancer, @yaseppochi, @vadmium, @serhiy-storchaka
Dependencies	bpo-22286: Allow backslashreplace error handler to be used on input bpo-23676: Add support of UnicodeTranslateError in standard error handlers
Files	convert_surrogates.py codecs_convert_escapes.patch codecs_convert_escapes_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-23.04:02:31.968>
labels = ['3.8', 'type-feature', 'library']
title = 'Add utilities to "clean" surrogate code points from strings'
updated_at = <Date 2018-03-30.06:52:39.866>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2018-03-30.06:52:39.866>
actor = 'ncoghlan'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-08-23.04:02:31.968>
creator = 'ncoghlan'
dependencies = ['22286', '23676']
files = ['36700', '38506', '38520']
hgrepos = []
issue_num = 18814
keywords = ['patch']
message_count = 55.0
messages = ['195937', '195959', '195960', '196047', '196049', '196054', '196063', '225791', '225793', '225800', '225801', '225806', '225809', '225811', '225817', '225818', '225822', '225858', '225869', '225875', '225889', '225972', '227334', '227339', '227340', '227341', '227342', '227343', '227344', '227345', '227346', '227362', '227364', '227365', '227367', '227368', '227369', '238182', '238193', '238277', '238285', '238286', '238309', '242799', '242810', '242939', '251682', '251693', '251694', '251701', '251708', '251740', '251836', '251854', '314682']
nosy_count = 11.0
nosy_names = ['lemburg', 'ncoghlan', 'pitrou', 'vstinner', 'ezio.melotti', 'Arfrever', 'steven.daprano', 'r.david.murray', 'sjt', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18814'
versions = ['Python 3.8']

Linked PRs

• gh-63014: Add utilites to transform surrogate codes and astral characters #121219

[ncoghlan]

Prompted by bpo-18713 and http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/, here are some possible utilities we could add to the codecs module to help deal with/debug issues related to surrogate escaped strings:

 def has_escaped_bytes(s):
 """Returns true if string contains surrogate escaped bytes"""
 ...

 def replace_escaped_bytes(s):
 """Replaces each surrogate escaped byte with a valid code point"""
 ...

 def decode_escaped_bytes(s, nominal_encoding, actual_encoding):
 """Reinterprets incorrectly decoded text using a new encoding"""
 return s.encode(nominal_encoding, 'surrogateescape').decode(actual_encoding)

[bitdancer]

The email package needs has_escaped_bytes. Currently it tries to encode to ascii to find out if there are any, which we proved by microbenchmark is the fastest way to do it as things stand.

What does replace do? Replace it with the unknown character code point? Email also needs that.

[serhiy-storchaka]

These functions are trivial.

 def has_escaped_bytes(s):
 try:
 s.encode()
 return False
 except UnicodeEncodeError:
 return True

 def replace_escaped_bytes(s):
 return s.encode('utf-8', 'replace').decode()

[pitrou]

Can you sum the use cases for these?
(don't want to read a blog post, sorry :-))

[vstinner]

"The email package needs has_escaped_bytes. Currently it tries to encode to ascii to find out if there are any, which we proved by microbenchmark is the fastest way to do it as things stand."

In which function you need to check this? What do you do if there are surrogate characters?

[bitdancer]

The email package uses surrogateescape to store unknown bytes in unicode strings, just as with the handle-bad-data-from-os API surrogateescape was introduced for. (For the same reason: the source data may have improperly encoded bytes that we must nevertheless preserve). What action certain parts of the code takes differs depending on whether or not there are such encoded bytes in the string. So the code needs to check and branch based on that.

The utility function is email.utils._has_surrogates. You can grep the email source to see where it is used, if you like.

[ncoghlan]

The use case is to take data from a surrogate escaped interface and either
filter it out entirely or convert it to a valid Unicode string at the point
of *input*, before letting it make its way into the rest of the
application. For example, this approach permits the Python equivalent of
the Linux console behaviour when it attempts to display the improperly
encoded file name.

This approach is likely to be easier to implement in cross-platform code
than dropping down to the bytes interfaces on POSIX systems and using the
Unicode APIs on Windows.

[ncoghlan]

Based on the latest round of bytes handling discussions on python-dev, I came up with this updated proposal:

 # Constant in the string module (akin to string.ascii_letters et al)
 escaped_surrogates = bytes(range(128, 256)).decode('ascii', errors='surrogateescape')

 # Helper to ensure a string contains no escaped surrogates
 # This allows it to be safely encoded without surrogateescape
 _match_surrogates = re.compile('[{}]'.format(escaped_surrogates))
 def clean(s, repl='\ufffd'):
 return _match_surrogates.sub(repl, s)

 # Helper to redecode a string that was decoded incorrectly
 # For example, WSGI strings are passed from the server to the
 # framework as latin-1 by default and may need to be redecoded
 def redecode(s, encoding, errors='strict', old_encoding='latin-1', old_errors='strict'):
 return s.encode(old_encoding, old_errors).decode(encoding, errors)

In addition to the concrete use cases David describes, I think these will also serve a useful documentation purpose, in highlighting the two main mechanisms for "smuggling" raw binary data through text APIs (i.e. surrogate escapes and latin-1 decoding).

[ncoghlan]

Note: I dropped the "has_escaped_bytes" idea, as "s != string.clean(s)" is functionally equivalent, albeit a bit more expensive if you don't actually want the cleaned string)

[ezio-melotti]

I think similar functions should be added in the unicodedata module rather than the string module or as str methods. If I'm not mistaken this was already proposed in another issue.
In C we already added macros like IS_{HIGH|LOW|}_SURROGATE and possibly others to help dealing with surrogates but AFAIK there's no Python equivalent yet.
As for the specific constants/functions/methods you propose, IMHO the name escaped_surrogates is not too clear. If it's a string of lone surrogates I would just call it unicodedata.surrogates (and .high_surrogates/.low_surrogates). These can also be used to build oneliner to check if a string contains surrogates and/or to remove them.
clean has a very generic name with no hints about surrogates, and its purpose is quite specific.
I'm also not a big fan of redecode. The equivalent calls to encode/decode are not much longer and more explicit. Also having to redecode often indicates that there's a bug before that should be fixed instead (if possible).

[ncoghlan]

The purpose of these changes it to provide tools specifically for working with surrogate escaped data, not for working with arbitrary lone Unicode surrogates.

"escaped_surrogates" is not defined by the Unicode spec, it's defined by the behaviour of the surrogateescape error handler that lets us tunnel arbitrary bytes through str objects and reproduce them faithfully at the far end. On reflection, I think codecs would be a better home than string (as that's where the error handler is defined), but it doesn't belong in unicodedata.

I'd be OK with changing the name of the clean function to "clean_escaped_surrogates".

Needing redecode is not a bug: it's baked into the WSGI spec in PEP-3333. I would be OK with providing it in wsgiref rather than the codecs or string modules, but I think we should provide it somewhere.

[serhiy-storchaka]

I agree with Ezio in all points. escaped_surrogates is inefficient for any purposes and incomplete. _match_surrogates can be created in more efficient way. clean() has too general and misleading name. redecode() looks just ridiculous, this cumbersome 4-argument function just combine two methods.

[ncoghlan]

Guys, you're Python 3 unicode experts, already thoroughly familiar with how surrogateescape works. These features are not for you.

The exact implementations don't matter. These need to exist, and they need to be documented with detailed explanations. People don't yet understand what the surrogateescape error handler is, and what it's for, or how to work with it.

People consider surrogate escaping this weird mysterious thing. It's not - it's a really simple mapping of the 128 bytes with the high order bit set to a different part of the Unicode code space. These functions make it obvious.

wsgiref.redecode is a natural consequence of the design of PEP-3333 - it makes sense to offer it there, as a hint that frameworks are likely to need it.

We have a serious, serious, learnability problem around binary data handling in Python 3. It's nowhere near as bad as the learnability problem for text handling in Python 2, but it still needs more scaffolding to help people grow in understanding.

[ezio-melotti]

That's why I think a function like redecode is a bad idea.
With Python 2 I've seen lot of people blindingly trying .decode when .encode failed (and the other way around) whenever they were getting an UnicodeError (and the fact that decoding Unicode results in an encode error didn't help).
I'm afraid that confused developers will try to (mis)use redecode as a workaround to attempt to fix something that shouldn't be broken in the first place, without actually understanding what the real problem is.
There's really no excuse for developers not to know Unicode nowadays (Joel Spolsky said the same thing more than 10 years ago).
IME people also tend to overestimate how difficult it is to understand Unicode. While it is true that you can go pretty deep down the rabbit hole, the basic concepts necessary for everyday use are pretty simple and straightforward.