CJK support for textwrap

[fgallaire]

BPO	24665
Nosy	@birkenfeld, @terryjreedy, @vstinner, @ezio-melotti, @bitdancer, @methane, @fgallaire, @serhiy-storchaka, @yan12125, @JulienPalard, @xi
PRs	bpo-24665: double-width CJK chars support for textwrap #89 bpo-24665: Add CJK support in textwrap by default. #5649 bpo-24665: Add CJK support in textwrap by default. #5649 gh-56708: support custom len function in textwrap.wrap #28136
Files	CJK.patch CJK+fix.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-09.08:58:24.285>
created_at = <Date 2015-07-19.02:44:39.415>
labels = ['3.8', 'type-feature', 'library', 'expert-unicode']
title = 'CJK support for textwrap'
updated_at = <Date 2021-09-03.08:13:33.576>
user = 'https://github.com/fgallaire'

bugs.python.org fields:

activity = <Date 2021-09-03.08:13:33.576>
actor = 'xi2'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-09.08:58:24.285>
closer = 'methane'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2015-07-19.02:44:39.415>
creator = 'fgallaire'
dependencies = []
files = ['39949', '40451']
hgrepos = []
issue_num = 24665
keywords = ['patch']
message_count = 24.0
messages = ['246930', '246932', '247001', '247008', '247009', '247016', '247022', '247025', '276275', '284068', '287736', '287753', '287756', '287820', '287821', '287822', '288785', '297563', '312076', '321283', '321284', '321291', '321304', '323733']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'terry.reedy', 'vstinner', 'ezio.melotti', 'r.david.murray', 'methane', 'fgallaire', 'serhiy.storchaka', 'yan12125', 'mdk', 'xi2']
pr_nums = ['89', '5649', '5649', '28136']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24665'
versions = ['Python 3.8']

[serhiy-storchaka]

This is new feature and can be added only in 3.6.

bpo-12499 looks related. See also bpo-12568.

[fgallaire]

Bad wrapping of CJK chars is a bug.
I don't understand why Python2 should be broken forever!

[bitdancer]

Because to get proper unicode support, we wrote python3, and because handling anything other than single-character-width characters in textwrap is a new feature.

[fgallaire]

FUD about Python here is something I wasn't expecting.

Python 2 supports Unicode and is still used a lot by a lot of people.

CJK people are not subhumans, so don't support CJK is something called, wait... a bug ! And it's a shame that it was not fixed earlier.

Python 3 has this bug too, so it's not really what I would call a "proper unicode support".

[bitdancer]

The problem is (if I'm understanding this correctly, which I may not be, I'm not a unicode expert) is that how you compute and manipulate CJK characters in python2 differs depending on whether you are dealing with a wide build or a narrow build. And the fact that python3 doesn't handle it either is why this would be a new feature (see the referenced issues).

But I could be wrong. I leave it to the unicode experts.

[serhiy-storchaka]

What textwrap does not take into account the width of characters is not the only problem. It also does not take into account combining characters and control codes. Implementing all this will significantly complicate the code and possibly should lie outside the standard library. Such large change obviously is new feature. I believe that we should first provide a common interface to determine the width of the line (bpo-12499) and allow to determine the appropriate algorithm at the application level. Also provide helper functions like in bpo-12568.

[fgallaire]

If your unicode experts haven't fix this BUG still now, this will never be done (by this experts).

We can say they are not true unicode experts as they have forgotten since a so long time billions of CJK people !

[bitdancer]

Or perhaps we haven't had a CJK user interested enough in using textwrap to provide the needed enhancements. It seems like there is interest in solving the related problems recently, so perhaps some progress will be made now.

The fact that you view it as a bug does not mean that it is a bug from the point of view of the python project. Every enhancement could be considered a bug, but we have a strict backward compatibility policy for maintenance release (which python 2.7.x is) which follows the semantic versioning principle that the API does not change in a micro release. According to Serhiy, fixing this correctly will require API changes.

[fgallaire]

CJKwrap a little lib to fix this bug:
https://github.com/fgallaire/cjkwrap

[fgallaire]

Hello everybody,

This is a Python3 version of my lib CJKwrap:
https://github.com/fgallaire/cjkwrap3

It could be integrated as a new lib in Python 3.7.

People who are using textwrap will have no surprise, and people who wants CJK width support will be happy to use it.

Any remarks welcome.

Cheers

[fgallaire]

After discussion with Haypo, CJK support is now implemented as and option, disabled by default for backward compatibility reasons.

PR on GitHub:
#89