Argparse incorrectly handles '--' as argument to option

[mmaker]

BPO	14364
Nosy	@rhettinger, @merwok, @bitdancer, @mmaker, @vadmium, @shihai1991, @horazont
PRs	gh-58572: Fix behavior when '--' as argument to option in argparse #15714
Files	foo.py issue14364.test.patch dbldash.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-03-18.14:29:12.929>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = "Argparse incorrectly handles '--' as argument to option"
updated_at = <Date 2021-03-23.01:07:11.301>
user = 'https://github.com/mmaker'

bugs.python.org fields:

activity = <Date 2021-03-23.01:07:11.301>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2012-03-18.14:29:12.929>
creator = 'maker'
dependencies = []
files = ['24926', '29938', '45944']
hgrepos = []
issue_num = 14364
keywords = ['patch', 'needs review']
message_count = 17.0
messages = ['156254', '156256', '156275', '156293', '187283', '187344', '187346', '187349', '262545', '262558', '283501', '351230', '353865', '354001', '354049', '354066', '389361']
nosy_count = 9.0
nosy_names = ['rhettinger', 'bethard', 'eric.araujo', 'r.david.murray', 'maker', 'martin.panter', 'paul.j3', 'shihai1991', 'jssfr']
pr_nums = ['15714']
priority = 'high'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14364'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[mmaker]

http://docs.python.org/library/argparse.html#arguments-containing
The attached file shows different behaviours when using '--' immediately after an optional argument.

tumbolandia:cpython maker$ python foo.py --test=-- foo
[]
tumbolandia:cpython maker$ python foo.py --test -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)

The same is for single-dash arguments.

tumbolandia:cpython maker$ python foo.py -t -- foo
usage: foo.py [-h] [-t TEST] [yuri]
foo.py: error: argument -t/--test: expected 1 argument(s)
tumbolandia:cpython maker$ python foo.py -t-- foo
[]

Obviously argparse should return an error in both cases.
The bug is probably due to Lib/argparser.py:2211

[bitdancer]

It does look like there's anomalous behavior here of some sort, but I'd expect --test=-- to result in test="--", myself, rather than an error. My intuition is that '--' would need to be preceded by a space to function as the 'end of options' marker. Because of that, I've never tried the above scenario with a unix command, so maybe my intuition is wrong :)

Just for reference (the code paths may be different), there is another open issue about -- parsing, bpo-13922.

[bethard]

I just tried this with grep's "-e" and "--regexp":

$ cat > temp.txt
a--b
cdef
$ grep -e-- -v temp.txt
cdef
$ grep --regexp=-- -v temp.txt
cdef
$ grep -e -- -v temp.txt
cdef
$ grep --regexp -- -v temp.txt
cdef

And with diff's "-I" and "--ignore-matching-lines":

$ cat > temp2.txt
cdef
a--b
$ diff temp.txt temp2.txt
1d0
< a--b
2a2
> a--b
$ diff -I-- temp.txt temp2.txt 
$ diff -I -- temp.txt temp2.txt 
$ diff --ignore-matching-lines -- temp.txt temp2.txt 
$ diff --ignore-matching-lines=-- temp.txt temp2.txt 
$

Note though that for options that don't take an argument, the "--" is just removed:

$ grep -v -- a temp.txt 
cdef
$ diff -i -- temp.txt temp2.txt
1d0
< a--b
2a2
> a--b

So I guess the unix rule is: if an option that takes an argument is followed by "--", use that as the option's argument and continue parsing as usual. If an option that takes no argument is followed by "--", then delete the "--" and treat all following flags as positional arguments.

Argparse can't follow this directly, because then people who are using "--" to signal the end of an option with nargs="*" would start getting "--" included in that list. (And I know people have used "--" this way for a while.)

I guess my preference is what R. David Murray suggests: "--" when part of an argument (i.e. not separated by spaces) is treated like any other characters, and only a lone "--" signals the end of options (and is ignored otherwise). That would mean that both "--test=--" and "-t--" in your example would give you ["--"], and the other errors would stay as you saw them.

[mmaker]

+1 also for me.
I will try to work for a patch in the next days. :)

[BreamoreBoy]

@michele could you provide a patch for this?

[paulj3]

The patch that I recently submitted for http://bugs.python.org/issue13922
appears to solve this issue. It only removes the '--' that marked the end of options.

With:

parser = argparse.ArgumentParser()
parser.add_argument('-f','--foo')
print(parser.parse_args(['-f--']))
print(parser.parse_args(['--foo=--']))
print(parser.parse_args(['-f', '--']))

I get:

Namespace(foo='--')
Namespace(foo='--')
usage: foodash.py [-h] [-f FOO]
foodash.py: error: argument -f/--foo: expected one argument

[mmaker]

wow, I was just writing the unittests, thanks paul.
Shall I continue? I don't see any test case on tip.

[mmaker]

Yes, http://bugs.python.org/file29845/dbldash.patch seems to fix this.
Attaching the unittests, and noisying on your issue.

[vadmium]

In Python 3.5, this does not seem fixed. bpo-13922 is marked as resolved and fixed, but Pauls’s patch has not actually been committed.

>>> ap = ArgumentParser()
>>> ap.add_argument("-t", "--test", type=str, nargs=1)
_StoreAction(option_strings=['-t', '--test'], dest='test', nargs=1, const=None, default=None, type=<class 'str'>, choices=None, help=None, metavar=None)
>>> ap.add_argument('yuri', nargs='?')
_StoreAction(option_strings=[], dest='yuri', nargs='?', const=None, default=None, type=None, choices=None, help=None, metavar=None)
>>> ap.parse_args(["--test=--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["--test", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2
>>> ap.parse_args(["-t--", "foo"])
Namespace(test=[], yuri='foo')
>>> ap.parse_args(["-t", "--", "foo"])
usage: [-h] [-t TEST] [yuri]
: error: argument -t/--test: expected 1 argument
argparse.ArgumentError: argument -t/--test: expected 1 argument

During handling of the above exception, another exception occurred:

__main__.SystemExit: 2

[paulj3]

I made a mistake of trying to add to or refine a closed patch. Maybe I need to move the dbldash.patch over here for more formal consideration.

[paulj3]

I've copied 'dbldash.patch' over from http://bugs.python.org/issue13922. I mistakenly added it to a closed issue.

The patch is old, and should be revised. It is also missing tests for this '--opt=--' case, even though it solves it.

I also reference this patch in http://bugs.python.org/issue9571, which tries to refine dbldash handling for PARSER, REMAINDER nargs.

I'm raising the priority on this because the issue has a come up a couple of times on Stackoverflow:

http://stackoverflow.com/questions/40685320/python-argparse-with-as-the-value

[shihai1991]

some test cases which paul provided looks doesn't keep compatible.

In TestDoubleDashRemoval:
# output in my env is Namespace(cmd='cmd', foo=None, rest=['--', '--foo'])
('-- cmd -- -- --foo', NS(cmd='cmd', foo=None, rest=['--', '--', '--foo']))

# output in my env is Namespace(cmd='--', foo='1', rest=['1', '2'])
('-f1 -- -- 1 -- 2', NS(cmd='--', foo='1', rest=['1', '--', '2']))

# output in my env is Namespace(cmd='--foo', foo=None, rest=['--bar', '2'])
('-- --foo -- --bar 2', NS(cmd='--foo', foo=None, rest=['--', '--bar', '2']))

# output in my env is Namespace(cmd='cmd', foo=None, rest=['--foo', '-f2'])
('cmd -- --foo -- -f2', NS(cmd='cmd', foo=None, rest=['--foo', '--', '-f2']))

In TestDoubleDashRemoval1:
# output in my env is Namespace(cmd='cmd', foo='1', rest=['-f2', '3'])
('-f1 -- cmd -- -f2 3', NS(cmd='cmd', foo='1', rest=['--', '-f2', '3']))

my python's version is: 2.7.5, 3.6.8