re.sub confusion between count and flags args

[mindauga]

BPO	11957
Nosy	@rhettinger, @terryjreedy, @vstinner, @ericvsmith, @jwilk, @ezio-melotti, @merwok, @serhiy-storchaka
Dependencies	bpo-23591: enum: Add Flags and IntFlags
Files	patch_11957 re_keyword_only.patch re_check_flags_type.patch re_deprecate_positional_count.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ezio-melotti'
closed_at = None
created_at = <Date 2011-04-29.18:27:10.198>
labels = ['expert-regex', 'type-feature', '3.7']
title = 're.sub confusion between count and flags args'
updated_at = <Date 2017-04-14.18:55:29.189>
user = 'https://bugs.python.org/mindauga'

bugs.python.org fields:

activity = <Date 2017-04-14.18:55:29.189>
actor = 'jwilk'
assignee = 'ezio.melotti'
closed = False
closed_date = None
closer = None
components = ['Regular Expressions']
creation = <Date 2011-04-29.18:27:10.198>
creator = 'mindauga'
dependencies = ['23591']
files = ['30821', '37067', '44825', '44826']
hgrepos = []
issue_num = 11957
keywords = ['patch']
message_count = 27.0
messages = ['134806', '134820', '134830', '135371', '135386', '135391', '136657', '143520', '186784', '186825', '186832', '186844', '186856', '192416', '230223', '230224', '230226', '230236', '230237', '230238', '230358', '230375', '277386', '277398', '291655', '291659', '291677']
nosy_count = 13.0
nosy_names = ['rhettinger', 'terry.reedy', 'vstinner', 'eric.smith', 'jwilk', 'ezio.melotti', 'eric.araujo', 'mrabarnett', 'python-dev', 'mindauga', 'serhiy.storchaka', 'mmilkin', 'umi']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11957'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

• gh-56166: Deprecate passing counts and maxsplit via position in re #107767
• gh-56166: Deprecate passing confusing positional arguments in re functions #107778

[mindauga]

re.sub don't substitute not ASCII characters:

Python 2.7.1 (r271:86832, Apr 15 2011, 12:11:58) Arch Linux

>>import re

>>a=u'aaa'
>>print re.search('(\w+)',a,re.U).groups()
(u'aaa')
>>print re.sub('(\w+)','x',a,re.U)
x

 BUT:

>>a=u'ąąą'
>>print re.search('(\w+)',a,re.U).groups()
(u'\u0105\u0105\u0105')
>>print re.sub('(\w+)','x',a,re.U)
ąąą

[ericvsmith]

The 4th parameter to re.sub() is a count, not flags.

[ezio-melotti]

Since this has been reported already several times (see e.g. bpo-11947), and it's a fairly common mistake, I think we should do something to avoid it.

A few possibilities are:

1. add a warning in the doc;
2. make count and flag keyword-only argument (raising a deprecation warning in 3.3 and actually change it later);
3. change the regex flags to some object that can be distinguished from ints and raise an error when a flag is passed to count;

[terryjreedy]

I like the idea of an internal REflag class with __new__, __or__, and __repr__==str. Str(re.A|re.L) might print as
"REflag: re.ASCII | re.IGNORE"
If it is *not* an int subclass, any attempt to use or mix with an int would raise. I checked and the doc only promises that flags can be or'ed. An __and__ method might be added if it were thought that people currently use & to check for flags set, though that is not currently promised.

[mrabarnett]

Something like "<re.Flag ASCII | IGNORE>" may be more Pythonic.

[terryjreedy]

Agreed, if we go that route.

[merwok]

I’d favor 1) or 2) over 3). Ints are short and very commonly used for flags.

[ezio-melotti]

See also bpo-12888 for an error in the stdlib caused by this.

[mmilkin]

I like option #2, and I was thinking of working on it today, poke me if anyone has a problem with this.

[mmilkin]

There is no sane way to issue a warning without changing the signature and we don't want to change the signature without issuing a deprecation warning for the function, so sadly option 3 is the only way for this to work, (Im going to not touch this till ENUMS are merged in.)