email/charset.py convert() patch

[tkikuchi]

BPO	1443875
Nosy	@loewis, @warsaw, @vstinner, @bitdancer
Files	charset.py.patch: charset.py.patch ja-nonlegit.py: non-legitimate ja characters example

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2006-03-06.04:41:30.000>
labels = ['type-feature', 'library', 'expert-email']
title = 'email/charset.py convert() patch'
updated_at = <Date 2019-07-29.12:00:52.206>
user = 'https://bugs.python.org/tkikuchi'

bugs.python.org fields:

activity = <Date 2019-07-29.12:00:52.206>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2006-03-06.04:41:30.000>
creator = 'tkikuchi'
dependencies = []
files = ['7052', '7053']
hgrepos = []
issue_num = 1443875
keywords = ['patch']
message_count = 6.0
messages = ['49660', '49661', '49662', '49663', '49664', '348645']
nosy_count = 5.0
nosy_names = ['loewis', 'barry', 'tkikuchi', 'vstinner', 'r.david.murray']
pr_nums = []
priority = 'low'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1443875'
versions = ['Python 3.3']

[tkikuchi]

email/charset.py has convert() function to incorporate
using different charset for input and output for a
language like Japanese. Because the email user agent
in Japan frequently use characters outside the range of
legitimate JIS charset, these messages are shunted in
mailman delivery. Here is a patch to handle this case
by replacing the characters by '?' using the 'errors'
parameter in unicode/encode functions.

This patch includes the unicode string conversion patch
in bpo-1368247.

[warsaw]

Logged In: YES
user_id=12800

Do you have a test case for this? bpo-1368247 includes one just
for the encoding of unicode strings, but not for the error
argument. If you can include a simple test case, then I'll
add this patch.

[tkikuchi]

Logged In: YES
user_id=67709

Thanks! Here is an example program which fails on 4.0a
without the patch.

[loewis]

I'm -1 on this patch as-is. It changes the default error handling behavior from 'strict' to 'replace', for this method. This is a choice that the caller of these methods should make, so I think the default value for the error handling should be 'strict'. If then it is meaningful in mailman to set this to 'replace', this is mailman's choice. Python should never, on its own, create moji-bake.

If you revise this patch, please provide the test case as a patch to Lib/email/test somehow (probably patching both test_email and test_email_renamed).

[tkikuchi]

Ok, I worked a little on the code to propagate the choice of 'strict' or 'replace' but there are many we should fix. And we should fix the application, mailman, also. The problem is only related to Japanese and we'd rather find workarounds elsewhere. I put the priority of this patch lowest. Thanks for your comments.