argparse removing more "--" than it should

[jol]

BPO	37510
Nosy	@ericvsmith

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-07-05.20:45:30.760>
labels = ['3.7', 'type-bug', 'library']
title = 'argparse removing more "--" than it should'
updated_at = <Date 2019-07-07.19:16:53.792>
user = 'https://bugs.python.org/jol'

bugs.python.org fields:

activity = <Date 2019-07-07.19:16:53.792>
actor = 'paul.j3'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-07-05.20:45:30.760>
creator = 'jol'
dependencies = []
files = []
hgrepos = []
issue_num = 37510
keywords = []
message_count = 9.0
messages = ['347378', '347379', '347407', '347413', '347434', '347435', '347437', '347442', '347478']
nosy_count = 3.0
nosy_names = ['eric.smith', 'paul.j3', 'jol']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37510'
versions = ['Python 3.7']

Linked PRs

• gh-81691: Fix handling of multiple "--" (double dashes) in argparse #124233
• [3.13] gh-81691: Fix handling of multiple "--" (double dashes) in argparse (GH-124233) #124266
• [3.12] gh-81691: Fix handling of multiple "--" (double dashes) in argparse (GH-124233) #124267

[jol]

$ python -c '
import argparse
p = argparse.ArgumentParser()
p.add_argument("first_arg")
p.add_argument("args", nargs="*")
r = p.parse_args(["foo", "--", "bar", "--", "baz", "--", "zap"])
print(r.first_arg + " " + " ".join(r.args))
' 

returns:

foo bar baz -- zap

when I think it should return:

foo bar -- baz -- zap

[jol]

Sorry, I forgot to add details on my machine.

Python: 3.7.3
OS: Archlinux

[jol]

To be clear, my opinion is that a single call of parse_args() should only ever remove the first "--". Right now, it seems that it removes the first of each argument group, as determined by nargs, I guess.

[paulj3]

There are earlier bug/issues about the '--'.

Also look at the parser code itself. Keep in mind that parsing is done in two passes - once to identify flags versus arguments ('O/A') and then to allocate strings to arguments. I don't recall when '--' is being handled, possibly in both.

[jol]

There are earlier bug/issues about the '--'.

Yes, there are:

https://bugs.python.org/issue9571
https://bugs.python.org/issue22223
https://bugs.python.org/issue14364

But this one seems separate. Though they're related, they don't seem like duplicates, so that's why I thought I'd make this one.

Also look at the parser code itself. Keep in mind that parsing is done in two passes - once to identify flags versus arguments ('O/A') and then to allocate strings to arguments. I don't recall when '--' is being handled, possibly in both.

I'm not sure what your point is here. I did take a quick look at the code, yesterday. I think the identification part you mention is done right. It marks 'O/A' as you mention. Then, when it sees "--", it marks it "-", and the rest of the arguments are marked as "A".

I didn't look at the start of the code of the second pass or how it is concretely linked to the first pass. However, I did see that in the example I gave, _get_values() in argparse.py gets called twice (apparently once per argument group as determined by nargs, I guess) to remove the "--" present in arg_strings. The first time, arg_strings is ["foo", "--"] and the second time it's ["bar", "--", "baz", "--", "zap"].

So, that's what happens, and where part of the fix should probably be. I don't think the removal of "--" should happen in a function that gets called multiple times. Though I didn't spend the time to see where the code should be positioned, I can only imagine the correct behavior would be to remove the argument marked as "-" by the first pass mentioned.

I didn't mention this yesterday, because I figured there wouldn't be much value in sharing incomplete research like this, as opposed to a patch. I didn't want to influence the work of whoever chose to invest time in this for a proper fix.

[jol]

to remove the "--" present in arg_strings

*to remove the first "--" present...

[paulj3]

I looked at this issue way back, in 2013:

https://bugs.python.org/issue13922

I probably shouldn't have tacked this on to a closed issue.

[jol]

Maybe I can find the time to make a patch this weekend (either today or tomorrow). I hope I'm not underestimating this somehow, but I don't think this would take too long. The only issue I can foresee is in disagreement of what the correct behavior should be, which is why I gave my opinion that a single call of parse_args() should only ever remove a single "--".

If I don't submit a patch by Monday (PDT), everyone should assume I decided not to tackle this.

By the way, does this issue tracking platform support submitting to the issue thread by email? Maybe, I'll try that.

[paulj3]

https://bugs.python.org/file29845/dbldash.patch

while written against an earlier version of argparse, does what you want. I moved the '--' removal out of _get_values and into consume_positionals.

Note that a full patch should include test cases and documentation changes if any.

Also now github pull requests are the preferred patching route. But don't expect rapid response on this. The issue has been around for a long time without causing too many complaints.

Backward compatibility is always a concern when making changes to core functionality. We don't want to mess with someone's working code.
Though I doubt if there are many users who count on multiple '--' removals, one way or other. The argparse docs explicity calls this a 'pseudo-argument'.