argparse does not honor default argument for nargs=argparse.REMAINDER argument

[rgov]

BPO	35495
Nosy	@mblahay

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-12-14.15:59:40.287>
labels = ['type-bug', 'library', '3.9']
title = 'argparse does not honor default argument for nargs=argparse.REMAINDER argument'
updated_at = <Date 2019-07-18.21:03:41.175>
user = 'https://bugs.python.org/rgov'

bugs.python.org fields:

activity = <Date 2019-07-18.21:03:41.175>
actor = 'mblahay'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-12-14.15:59:40.287>
creator = 'rgov'
dependencies = []
files = []
hgrepos = []
issue_num = 35495
keywords = []
message_count = 17.0
messages = ['331837', '332567', '341826', '341844', '341938', '341939', '342115', '342122', '342124', '342129', '342150', '342196', '342370', '343603', '343710', '345916', '348129']
nosy_count = 4.0
nosy_names = ['bethard', 'paul.j3', 'rgov', 'mblahay']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35495'
versions = ['Python 3.9']

[rgov]

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('things', nargs=argparse.REMAINDER, default=['nothing'])
parser.parse_args([])
>>> Namespace(things=[])

Since there were no unparsed arguments remaining, the default setting for things should have been honored. However it silently ignores this setting.

If there's a reason why this wouldn't be desirable, it should raise an exception that the options aren't compatible.

[paulj3]

argparse.REMAINDER matches an empty list of arguments, just like '?' and '*'. So they are always 'filled', even by parse_args([]).

'?' and '*' have some special handling of defaults in this case, see in

argparse.ArgumentParser._get_values

the two

 value = action.default

REMAINDER has its own section in the function that does nothing with the default.

I think it should be left as is.

[mblahay]

Ryan, what are the exact steps to reproduce the problem? This is what I get when I run the code you included:

>>> import argparse
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('things', nargs=argparse.REMAINDER, default=['nothing'])
_StoreAction(option_strings=[], dest='things', nargs='...', const=None, default=['nothing'], type=None, choices=None, help=None, metavar=None)
>>> parser.parse_args([])
Namespace(things=[])
>>> Namespace(things=[])
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
NameError: name 'Namespace' is not defined

[rgov]

Just don’t run the last line which is just an echoing of the output of
parser.parse_args() repeated. The Namespace type would need to be imported
if you really wanted to but there’s no point.

On Tuesday, May 7, 2019, Michael Blahay <report@bugs.python.org> wrote:

>
> Michael Blahay <mblahay@gmail.com> added the comment:
>
> Ryan, what are the exact steps to reproduce the problem? This is what I
> get when I run the code you included:
>
> >>> import argparse
> >>> parser = argparse.ArgumentParser()
> >>> parser.add_argument('things', nargs=argparse.REMAINDER,
> default=['nothing'])
> _StoreAction(option_strings=[], dest='things', nargs='...', const=None,
> default=['nothing'], type=None, choices=None, help=None, metavar=None)
> >>> parser.parse_args([])
> Namespace(things=[])
> >>> Namespace(things=[])
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> NameError: name 'Namespace' is not defined
>
> 

---

nosy: +mblahay

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35495\>

---

[mblahay]

Okay, so the expected output after running parse.parse_args([]) is Namespace(['nothing'])

[mblahay]

Ryan, I have reviewed the documentation at https://docs.python.org/3/library/argparse.html#nargs and must admit that there is not a definitive answer that I can see regarding the defined behavior should there be no command line arguments that are in fact remaining. One could certainly argue that the empty list is the expression of the fact that no remaining arguments could be found. One can also argue that when seeking the remaining arguments, a list that may be zero to many elements in size, that by definition there cannot be a default.

Can you cite any documentation that would support your claim?

[mblahay]

For the purpose of facilitating continuing conversation, here are two tests that contrast the use of * versus REMAINDER

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.add_argument('baz', nargs='*', default=['nada'])
parser.parse_args('a b c'.split())

Out[7]: Namespace(bar=['b', 'c'], baz=['nada'], foo=['a'])

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
parser.add_argument('baz', nargs='*', default=['nada'])
parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.parse_args('a b c'.split())

Out[8]: Namespace(bar=[], baz=['b', 'c'], foo=['a'])

You can see that * and REMAINDER do differ in functionality when they are the last defined argument.

[mblahay]

Here is another take on the issue, this time illustrated through the lens of optional arguments.

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs=1,default=['none'])
parser.add_argument('--baz', nargs='*', default=['nada'])
parser.add_argument('--bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.parse_args('--foo a --bar b --baz c'.split())

Out[9]: Namespace(bar=['b', '--baz', 'c'], baz=['nada'], foo=['a'])

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs=1,default=['none'])
parser.add_argument('--baz', nargs='*', default=['nada'])
parser.add_argument('--bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.parse_args('--foo a --baz b --bar c'.split())

Out[10]: Namespace(bar=['c'], baz=['b'], foo=['a'])

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--foo', nargs=1,default=['none'])
parser.add_argument('--baz', nargs='*', default=['nada'])
parser.add_argument('--bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.parse_args([])

Out[11]: Namespace(bar=['nothing'], baz=['nada'], foo=['none'])

It is important to note that when an optional argument is not present then the default is always used, including for one using nargs=argparse.REMAINDER. In all three tests, bar is the argument using REMAIDER. In the first test, one can see that when bar isn't the last argument then anything else, including other arguments, are swept up as being arguments of bar. This greedy behavior for REMAINDER is something that * does not share (test 2).

[mblahay]

With the optional arguments, the determination about whether to use the default value is made based on whether the flag is present or not. When positional arguments are involved, the need for the defaults seems to in part be determined based on whether the argument exists. The fact that * and REMAINDER are zero-to-many in nature add some ambiguity into the situation. For the *, it seems that the positional argument only exists if there is at least one actual argument value that it can consume.

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
#parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.add_argument('baz', nargs='*', default=['nada'])
parser.parse_args('a b'.split())

Out[25]: Namespace(baz=['b'], foo=['a'])

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
#parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
parser.add_argument('baz', nargs='*', default=['nada'])
parser.parse_args('a'.split())

Out[26]: Namespace(baz=['nada'], foo=['a'])

Mean while, the REMAINDER option makes the argument act as if it exists regardless of whether an actual argument value exists.

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
#parser.add_argument('baz', nargs='*', default=['nada'])
parser.parse_args('a b'.split())

Out[27]: Namespace(bar=['b'], foo=['a'])

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo', nargs=1,default=['none'])
parser.add_argument('bar', nargs=argparse.REMAINDER,default=['nothing'])
#parser.add_argument('baz', nargs='*', default=['nada'])
parser.parse_args('a'.split())

Out[28]: Namespace(bar=[], foo=['a'])

To conclude, * and REMAINDER perform similar, but different, roles when used with positional arguments. With edge cases like the ones laid out above, it can be hard to conceptualize what the exact behavior should be. I will recommend that the documentation be updated to convey the following message: "When used with positional arguments, REMAINDER will never use the designated default value list. It will instead return an empty list if there are no values for the argument to consume. If the use of default values is desired, then * must be used."

[mblahay]

Much detail has been provided regarding why the default is ignored when user the REMAINDER option. The desire to add an exception has not. Is there anyone that can provide guidance on whether the combination of:

1. Positional Argument
2. nargs=REMAINDER
3. default=something
   should raise an exception upon execution of the add_argument method?

[paulj3]

At the start of parse_known_args, all defaults (except SUPPRESS ones) are placed in the namespace:

 # add any action defaults that aren't present
 for action in self._actions:
 if action.dest is not SUPPRESS:
 if not hasattr(namespace, action.dest):
 if action.default is not SUPPRESS:
 setattr(namespace, action.dest, action.default)

at the end of _parse_known_args there's a conditional expression that cleans up remaining defaults that are strings, by passing them through the 'type` callable:

 setattr(namespace, action.dest,
 self._get_value(action, action.default))

Read the comments to see why this default setting is done in two parts.

The pattern of defaults in msg342122 with optionals is consistent with that. If the argument is not provided, the default appears.

In the first example of that message, the REMAINDER is given all the remaining strings including the '--bar', so there is nothing left to trigger the '--bar' optional argument, and it retains the default.

The difference for positionals is due to how the '*' and '...' are handled in _getvalues. Both may be filled with an empty list of values.

 # when nargs='*' on a positional, if there were no command-line
 # args, use the default if it is anything other than None
 elif (not arg_strings and action.nargs == ZERO_OR_MORE and
 not action.option_strings):
 if action.default is not None:
 value = action.default
 else:
 value = arg_strings
 self._check_value(action, value)

 ....

 # REMAINDER arguments convert all values, checking none
 elif action.nargs == REMAINDER:
 value = [self._get_value(action, v) for v in arg_strings]

In the case of '*', the default is, effectively, placed back on the namespace. REMAINDER does not - the empty list is put in the namespace.

In _get_positional_kwargs,

 # mark positional arguments as required if at least one is
 # always required
 if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
 kwargs['required'] = True
 if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
 kwargs['required'] = True

That last conditional is a little puzzling, but I suspect it has to do with mutually_exclusive_groups, A '*' positional can be a member of a group if it has a default.

Anyways, we could add a test at this point like:

 if kwargs.get('nargs') == REMAINDER and 'default' in kwargs:
 msg = _("'default' is not allowed with a REMAINDER positional")
 raise TypeError(msg)

But reviewing the code I notice another difference. 'choices' are not honored for REMAINDER. That makes a lot of sense. REMAINDER is supposed to be a catch all, documented as something that might be passed on to another parser. This parser shouldn't be doing anything with those values.

The documentation reads:

argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:

I think REMAINDER has another quirk. It doesn't work as the first (and only?) argument. There should be a bug/issue to that effect.

https://bugs.python.org/issue17050, argparse.REMAINDER doesn't work as first argument

In https://bugs.python.org/issue17050#msg315716, I suggest removing REMAINDER from the docs. We can leave the code as is, in case anyone is using still using it. But it is probably too much work to make the code and docs match, both for that issue, and for this.

'*' plus '--' gives almost the same behavior. So does parse_known_args.