argparse uses default encoding when read arguments from file

[serhiy-storchaka]

BPO	41136
Nosy	@rhettinger, @serhiy-storchaka, @ZackerySpytz
PRs	gh-85308: Add a parameter to argparse.ArgumentParser for file encoding #21444

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-27.07:13:47.818>
labels = ['type-feature', 'library', '3.10']
title = 'argparse uses default encoding when read arguments from file'
updated_at = <Date 2020-07-11.12:13:17.671>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-07-11.12:13:17.671>
actor = 'ZackerySpytz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-27.07:13:47.818>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 41136
keywords = ['patch']
message_count = 1.0
messages = ['372452']
nosy_count = 4.0
nosy_names = ['rhettinger', 'paul.j3', 'serhiy.storchaka', 'ZackerySpytz']
pr_nums = ['21444']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue41136'
versions = ['Python 3.10']

[serhiy-storchaka]

The fromfile_prefix_chars option allows you to read arguments from file. But open() without explicit encoding is used for this. Therefore the result is depending on the current locale or the PYTHONIOENCODING environment variable. On Linux this is rare a problem, because UTF-8 is a common locale encoding, but on Windows the locale encoding is usually 8-bit and may even be not able to encode all file names.

I think we need a new option to specify the encoding for files from which arguments are read.

[methane]

I think sys.getfilesystemencoding() and sys.getfilesystemencodeerrors() should be default.

I am not sure encoding/errors option is worth enough.
On Unix, fsencoding is locale encoding on Unix. UTF-8 mode affects both.
On Windows, PYTHONLEGACYWINDOWSFSENCODING can be used for backward compat option.

@vstinner How do you think?

[vstinner]

argparse has a big API, which specific API are you talking about? type=argparse.FileType accepts an encoding:

parser.add_argument('dest_file', type=argparse.FileType('w', encoding='latin-1'))

The documentation gives this example:

parser.add_argument('source_file', type=open)

You can use functools.partial(open, encoding='utf-8') to pass an encoding if you really don't want to use FileType.

[vstinner]

I expect that FileType() used with no encoding behaves the same than open() used with no encoding.

[methane]

@vstinner We are talking about here.

cpython/Lib/argparse.py

Lines 2164 to 2169 in 20d30ba

	with open(arg_string[1:]) as args_file:
	arg_strings = []
	for arg_line in args_file.read().splitlines():
	for arg in self.convert_arg_line_to_args(arg_line):
	arg_strings.append(arg)
	arg_strings = self._read_args_from_files(arg_strings)

argparse reads options from file with default text encoding for now.

On Unix, commandline arguments are decoded with fsencoding. So fsencoding should be used to parse option file too. This is almost backward compatible because locale encoding is same to fsencoding in most cases.

On Windows, fsencoding is UTF-8 by default and can be ANSI code page when PYTHONLEGACYWINDOWSFSENCODING is set. This is slightly backward incompatible, but I think it is perfect choice because:

• Commandline arguments and filesystem encoding are UTF-16 (WTF-16). UTF-8/surrogatepass is the best text encoding to represent it.
• Notepad.exe and most major text editors uses UTF-8 by default.
• User can set PYTHONLEGACYWINDOWSFSENCODING if they have option file with ANSI code page and they can not convert it to UTF-8.

So I don't think we need to add another option like fromfile_encoding in #21444.