Bugs and inconsistencies in unicodedata

[dscorbett]

BPO	36486
Nosy	@malemburg, @vstinner, @ezio-melotti, @dscorbett

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-30.14:41:13.171>
labels = ['3.8', 'type-bug', '3.7', 'expert-unicode', 'docs']
title = 'Bugs and inconsistencies in unicodedata'
updated_at = <Date 2019-04-05.18:28:55.383>
user = 'https://github.com/dscorbett'

bugs.python.org fields:

activity = <Date 2019-04-05.18:28:55.383>
actor = 'terry.reedy'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Unicode']
creation = <Date 2019-03-30.14:41:13.171>
creator = 'dscorbett'
dependencies = []
files = []
hgrepos = []
issue_num = 36486
keywords = []
message_count = 1.0
messages = ['339203']
nosy_count = 5.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'docs@python', 'dscorbett']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36486'
versions = ['Python 3.7', 'Python 3.8']

Linked PRs

• gh-80667: Fix Tangut ideographs names in unicodedata #101585
• gh-80667: fix case-sensitivity of some unicode literal escapes #107281
• [3.14] gh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-107281) #144753
• [3.13] gh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-107281) #144754
• gh-80667: Fix Tangut ideographs names in unicodedata #144789
• [3.14] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-144789) #144871
• [3.13] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-144789) (GH-144871) #144875

[dscorbett]

In unicodedata, the functions lookup and name have some bugs and inconsistencies.

lookup matches case-insensitively, except for the algorithmic names of Hangul syllables and CJK unified ideographs, which must be in all caps. The documentation does not explain how character names are fuzzily matched.

lookup accepts names like “CJK UNIFIED IDEOGRAPH-04E00”, where the code point has a leading zero.

lookup and name don’t implement rule NR2, defined in chapter 4 of Unicode, for Tangut ideographs’ names.

[cfbolz]

just hit the difference between CJK UNIFIED IDEOGAPH-*, HANGUL SYLLABLE * and other unicode names again:

>>> "\N{SNOWMAN}\N{SNOWmAn}\N{sNoWMAN}\N{snowman}"
'☃☃☃☃'
>>> # but:
>>> "\N{CJK UNIFIED IDEOGRAPH-3400}"
'㐀'
>>> "\N{cjk UNIFIED IDEOGRAPH-3400}"
 File "<stdin>", line 1
 "\N{cjk UNIFIED IDEOGRAPH-3400}"
 ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-29: unknown Unicode character name