Argparse exceptions should include which argument has a problem

[vpython]

BPO	21666

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-06-05.04:25:08.064>
labels = ['type-feature']
title = 'Argparse exceptions should include which argument has a problem'
updated_at = <Date 2014-06-08.06:03:05.054>
user = 'https://bugs.python.org/vpython'

bugs.python.org fields:

activity = <Date 2014-06-08.06:03:05.054>
actor = 'paul.j3'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2014-06-05.04:25:08.064>
creator = 'v+python'
dependencies = []
files = []
hgrepos = []
issue_num = 21666
keywords = []
message_count = 5.0
messages = ['219778', '219949', '219969', '220011', '220016']
nosy_count = 2.0
nosy_names = ['v+python', 'paul.j3']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21666'
versions = []

Linked PRs

• gh-65865: Raise early errors for invalid help strings in argparse #124899

[vpython]

I coded up a new program, with a bunch of options, and got the following traceback when I tried to run it:

Traceback (most recent call last):
  File "D:\my\py\renmany.py", line 273, in <module>
    args = cmdl.parse_intermixed_args()
  File "D:\my\py\glu\glu.py", line 1695, in parse_intermixed_args
    args, argv = self.parse_known_intermixed_args(args, namespace)
  File "D:\my\py\glu\glu.py", line 1740, in parse_known_intermixed_args
    namespace, remaining_args = self.parse_known_args(args, namespace)
  File "C:\Python33\lib\argparse.py", line 1737, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "C:\Python33\lib\argparse.py", line 1943, in _parse_known_args
    start_index = consume_optional(start_index)
  File "C:\Python33\lib\argparse.py", line 1883, in consume_optional
    take_action(action, args, option_string)
  File "C:\Python33\lib\argparse.py", line 1811, in take_action
    action(self, namespace, argument_values, option_string)
  File "C:\Python33\lib\argparse.py", line 1015, in __call__
    parser.print_help()
  File "C:\Python33\lib\argparse.py", line 2339, in print_help
    self._print_message(self.format_help(), file)
  File "C:\Python33\lib\argparse.py", line 2323, in format_help
    return formatter.format_help()
  File "C:\Python33\lib\argparse.py", line 276, in format_help
    help = self._root_section.format_help()
  File "C:\Python33\lib\argparse.py", line 206, in format_help
    func(*args)
  File "C:\Python33\lib\argparse.py", line 206, in format_help
    func(*args)
  File "C:\Python33\lib\argparse.py", line 513, in _format_action
    help_text = self._expand_help(action)
  File "C:\Python33\lib\argparse.py", line 600, in _expand_help
    return self._get_help_string(action) % params
ValueError: unsupported format character ')' (0x29) at index 673

The only thing I can tell is that something went wrong in ArgParse. I had called a bunch of add_argument, and then a parse_known_args. I had passed parameters to the program to get a help message, so that is what I expect parse_known_args is trying to produce... and the call stack seems to confirm that. I didn't intentionally pass a format character ')' anywhere, but there are ')' characters in some of my help messages, so that is probably the source of the problem.

No doubt I can reduce the problem space by judiciously commenting out things until I can isolate the particular help message that is causing the failure (it may be more than one as several are similar). But it seems like the exception should include the name of the argument for which the failure occurred.

[OK, I isolated, and found a "%)" sequence in one of my messages that should have been "%%)". So this is not terribly urgent, just poor reporting.]

[paulj3]

First, 'parse_intermixed_args' on stack is not relevant. It's from an unreleased patch that we worked on.

What matters is the 'print_help', invoked probably with a '-h'.

The error message that normally specifies the problem argument is produced by ArgumentError. The HelpFormatter does not raise such an error. ArgumentError is usually used for parsing errors; this is a formatting one. It's not produced by faulty commandline values.

If you must put strings like '%)` in the help line, use RawTextHelpFormatter. Otherwise HelpFormatter assumes the help line has valid format expressions like '%(default)s'.

Or you could write your own HelpFormatter subclass with a modified '_expand_help' method, one which wraps the 'self._get_help_string(action) % params' in a 'try' block. Probably too draconian a measure for a rare problem. :)

It's an interesting problem, but I don't think it warrants any code changes.

[vpython]

Yes, I hope someday the parse_intermixed_args patch can be released... but I know it is not relevant to this issue.

I was aware of the %(substitution_variables) in the default help formatter, but I (1) goofed and entered % without escaping it (2) was surprised at how unhelpful the Traceback was at isolating the problem.

Happily, my code had only a few instances of %) so I was able to isolate it fairly quickly, but the error report certainly shows up at quite a distance (execution-wise) from the location of the source bug.

I haven't looked at the source for the HelpFormatter code: if it concatenates all the help text and then does substitutions en masse, then it would be difficult to isolate the error to a particular argument. If, on the other hand, it loops through the help text for each argument, doing the substitutions, and later formatting and concatenating, then surrounding the substitution attempt with a try: block so that the name of the argument with the faulty help text could be reported, that would be a big help to this situation, at little cost.

[paulj3]

The ''_expand_help' method formats one action (argument) at a time, so it could issue an error message that includes that action's name.

The disconnect that you noticed arises because your bad 'help' parameter wasn't tested until is was used in a 'print_help'.

http://bugs.python.org/issue9849 asks for better testing of nargs (and metavar) values. In the proposed patch, 'add_argument' creates a temporary HelpFormatter and tries to format a relevant portion of the help. I suppose that test could be extended to test the 'help' parameter as well.

[paulj3]

In http://bugs.python.org/file30010/nargswarn.patch adding the '_expand_help(action)' line should test the help string (during add_argument).

    def _check_argument(self, action):
        # check action arguments
        # focus on the arguments that the parent container does not know about
        # check nargs and metavar tuple
        try:
            self._get_formatter()._format_args(action, None)
        except ValueError as e:
            raise ArgumentError(action, str(e))
        except TypeError:
            #raise ValueError("length of metavar tuple does not match nargs")
            raise ArgumentError(action, "length of metavar tuple does not match nargs")
        # check the 'help' string
        try:
            self._get_formatter()._expand_help(action)
        except (ValueError, TypeError, KeyError) as e:
            raise ArgumentError(action, 'badly formed help string')

The 'except' clause may need to be changed to capture all (or just most?) of the possible errors in the format string. Besides your error I can imagine '%(error)s` (a KeyError). We also need to pay attention to the differences between Py2 and Py3 errors.

[serhiy-storchaka]

This should not be ArgumentError. That exception is for parsing errors, not for programming errors.

#124899 adds checks for invalid help arguments to add_argument(), add_subparsers() and add_parser(). The traceback will show the culprit.