gh-74902: Add Unicode Grapheme Cluster Break algorithm by serhiy-storchaka · Pull Request #143076 · python/cpython

serhiy-storchaka · 2025-12-22T16:34:03Z

Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex #29.

Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm.

Issue: Add unicode grapheme cluster break algorithm #74902

Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex python#29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

merwok · 2025-12-23T01:55:01Z

+ ``False`` otherwise.
+
+ .. versionadded:: next
+


The order of functions in this file doesn’t seem to be alphabetical or topical.
I think another ticket should be created to add a quick links table at the top.

Or we can split it on sections by type and order alphabetically inside a section.

merwok · 2025-12-23T01:56:02Z

 .. data:: ucd_3_2_0

- This is an object that has the same methods as the entire module, but uses the
+ This is an object that has most of the methods of the entire module, but uses the


This sentence is not fully right, but I can’t find the right suggestion with both «most of» and «same as».

merwok · 2025-12-23T01:58:48Z

These functions help compute width?

serhiy-storchaka · 2025-12-23T08:01:13Z

At least two implementations (in Perl's Unicode::GCString and builtin in C++) use graphemes. Naive implementation in C's wcwidth() does not work well with complex characters and Emoji.

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

merwok · 2025-12-23T14:59:33Z

Sorry if my question was not clear.
Are these functions building blocks that will be used to compute character widths, as needed by the other tickets about pyrepl, str justify, etc?
(maybe iter_graphemes?)

serhiy-storchaka · 2025-12-23T15:12:10Z

Yes, I think that _PyGraphemeBreak, _Py_InitGraphemeBreak() and _Py_NextGraphemeBreak() will be used in the implementation of unicodedata.width().

StanFromIreland · 2025-12-26T13:43:26Z

+ self.assertEqual(chunks.pop(), '', line)
+ input = ''.join(chunks)
+ with self.subTest(line):
+ result = list(unicodedata.iter_graphemes(input))


Did you mean to use the passed ucd argument?
Suggested change
result=list(unicodedata.iter_graphemes(input))
result=list(ucd.iter_graphemes(input))

StanFromIreland · 2025-12-26T13:43:43Z

+ self.assertEqual([x.start for x in result], breaks[:-1], comment)
+ self.assertEqual([x.end for x in result], breaks[1:], comment)
+ for i in range(1, len(breaks) - 1):
+ result = list(unicodedata.iter_graphemes(input, breaks[i]))


Suggested change
result=list(unicodedata.iter_graphemes(input, breaks[i]))
result=list(ucd.iter_graphemes(input, breaks[i]))
Continues above.

No, it is module-only function.

StanFromIreland · 2025-12-26T13:46:46Z

+}
+
+
+/* XXX Add doc strings. */


The above functions already have docstrings?

StanFromIreland · 2025-12-26T14:05:02Z

+ hdr = testfile.readline()
+ return unicodedata.unidata_version in hdr
+
+ @requires_resource('network')


Should it not be urlfetch resource?

Maybe. The other test (for normalization) uses the network resource).

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

…me_break_tests().

bedevere-bot · 2026-01-14T15:28:16Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Stable LTO 3.x (tier-3) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1654/builds/1940) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1654/builds/1940

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-s390x.lto/build/Lib/tempfile.py", line 484, in __del__
_warnings.warn(self.warn_message, ResourceWarning)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning: Implicitly cleaning up <HTTPError 403: 'Forbidden'>

bedevere-bot · 2026-01-14T15:58:09Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2750) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2750

Failed tests:

test_unicodedata

Test leaking resources:

test_unicodedata: memory blocks
test_unicodedata: references

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 24, done. remote: Counting objects: 6% (1/16) remote: Counting objects: 12% (2/16) remote: Counting objects: 18% (3/16) remote: Counting objects: 25% (4/16) remote: Counting objects: 31% (5/16) remote: Counting objects: 37% (6/16) remote: Counting objects: 43% (7/16) remote: Counting objects: 50% (8/16) remote: Counting objects: 56% (9/16) remote: Counting objects: 62% (10/16) remote: Counting objects: 68% (11/16) remote: Counting objects: 75% (12/16) remote: Counting objects: 81% (13/16) remote: Counting objects: 87% (14/16) remote: Counting objects: 93% (15/16) remote: Counting objects: 100% (16/16) remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 9% (1/11) remote: Compressing objects: 18% (2/11) remote: Compressing objects: 27% (3/11) remote: Compressing objects: 36% (4/11) remote: Compressing objects: 45% (5/11) remote: Compressing objects: 54% (6/11) remote: Compressing objects: 63% (7/11) remote: Compressing objects: 72% (8/11) remote: Compressing objects: 81% (9/11) remote: Compressing objects: 90% (10/11) remote: Compressing objects: 100% (11/11) remote: Compressing objects: 100% (11/11), done. remote: Total 24 (delta 5), reused 5 (delta 5), pack-reused 8 (from 2) 
From https://github.com/python/cpython
* branch main -> FETCH_HEAD
Note: switching to 'bab1d7a561ab015dd6bb97e255fd12a8ce367edf'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at bab1d7a561a gh-74902: Add Unicode Grapheme Cluster Break algorithm (GH-143076)
Switched to and reset branch 'main'
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal modulemake: *** [Makefile:2503: buildbottest] Error 2

bedevere-bot · 2026-01-14T16:11:44Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Refleaks 3.x (tier-3) has failed when building commit bab1d7a.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1613/builds/2681) and take a look at the build logs.
Check if the failure is related to this commit (bab1d7a) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1613/builds/2681

Failed tests:

test_unicodedata

Test leaking resources:

test_unicodedata: memory blocks
test_unicodedata: references

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 24, done. remote: Counting objects: 6% (1/16) remote: Counting objects: 12% (2/16) remote: Counting objects: 18% (3/16) remote: Counting objects: 25% (4/16) remote: Counting objects: 31% (5/16) remote: Counting objects: 37% (6/16) remote: Counting objects: 43% (7/16) remote: Counting objects: 50% (8/16) remote: Counting objects: 56% (9/16) remote: Counting objects: 62% (10/16) remote: Counting objects: 68% (11/16) remote: Counting objects: 75% (12/16) remote: Counting objects: 81% (13/16) remote: Counting objects: 87% (14/16) remote: Counting objects: 93% (15/16) remote: Counting objects: 100% (16/16) remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 9% (1/11) remote: Compressing objects: 18% (2/11) remote: Compressing objects: 27% (3/11) remote: Compressing objects: 36% (4/11) remote: Compressing objects: 45% (5/11) remote: Compressing objects: 54% (6/11) remote: Compressing objects: 63% (7/11) remote: Compressing objects: 72% (8/11) remote: Compressing objects: 81% (9/11) remote: Compressing objects: 90% (10/11) remote: Compressing objects: 100% (11/11) remote: Compressing objects: 100% (11/11), done. remote: Total 24 (delta 5), reused 5 (delta 5), pack-reused 8 (from 2) 
From https://github.com/python/cpython
* branch main -> FETCH_HEAD
Note: switching to 'bab1d7a561ab015dd6bb97e255fd12a8ce367edf'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at bab1d7a561a gh-74902: Add Unicode Grapheme Cluster Break algorithm (GH-143076)
Switched to and reset branch 'main'

…H-143076) Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex python#29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

…ythonGH-143076) Add the unicodedata.iter_graphemes() function to iterate over grapheme clusters according to rules defined in Unicode Standard Annex #29. Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break() and unicodedata.extended_pictographic() functions to get the properties of the character which are related to the above algorithm. (cherry picked from commit bab1d7a) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com> Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>

bedevere-app · 2026-04-08T12:24:12Z

GH-148247 is a backport of this pull request to the 3.14 branch.

serhiy-storchaka requested a review from AA-Turner as a code owner December 22, 2025 16:34

bedevere-appBot mentioned this pull request Dec 22, 2025
Add unicode grapheme cluster break algorithm #74902
Closed

bedevere-appBot added the awaiting core review label Dec 22, 2025

Try to fix rst.
b0585f9

AA-Turner mentioned this pull request Dec 22, 2025
gh-74902: add unicode grapheme cluster break algorithm #2673
Closed

AA-Turner reviewed Dec 22, 2025
View reviewed changes

Comment threadDoc/whatsnew/3.15.rst Outdated

grayjk mentioned this pull request Dec 22, 2025
What would wcwidth look like if it were built-in to Python? jquast/wcwidth#94
Closed

StanFromIreland reviewed Dec 22, 2025
View reviewed changes

Comment threadDoc/library/unicodedata.rst Outdated

merwok reviewed Dec 23, 2025
View reviewed changes

serhiy-storchakaand others added 2 commits December 23, 2025 11:37

iConvert the types to heap types.
37fa38f

Update Doc/library/unicodedata.rst
c46e9bd
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

StanFromIreland reviewed Dec 26, 2025
View reviewed changes

serhiy-storchakaand others added 6 commits December 26, 2025 22:06

Apply suggestions from code review
22cacf6
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Add tests for GB12 and GB13. Remove redundant parameter in run_graphe…
a95c3cb
…me_break_tests().

Rename Grapheme to Segment. Add __repr__(), remove __iter__().
ad50831

Merge branch 'main' into grapheme_cluster_break2
365d0c1

Merge branch 'main' into grapheme_cluster_break2
f96c8d0

Polishing.
3b3a50a

serhiy-storchaka enabled auto-merge (squash) January 14, 2026 14:16

serhiy-storchaka merged commit bab1d7a into python:mainJan 14, 2026
47 checks passed

bedevere-appBot removed the awaiting core review label Jan 14, 2026

serhiy-storchaka deleted the grapheme_cluster_break2 branch July 1, 2026 16:47

	result=list(unicodedata.iter_graphemes(input))
	result=list(ucd.iter_graphemes(input))

	result=list(unicodedata.iter_graphemes(input, breaks[i]))
	result=list(ucd.iter_graphemes(input, breaks[i]))

Uh oh!

Uh oh!

Conversation

serhiy-storchaka commented Dec 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merwok commented Dec 23, 2025

Uh oh!

serhiy-storchaka commented Dec 23, 2025

Uh oh!

merwok commented Dec 23, 2025

Uh oh!

serhiy-storchaka commented Dec 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Jan 14, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-appBot commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

serhiy-storchaka commented Dec 22, 2025•
edited
Loading