.pth files cannot contain folders with utf-8 names

[einaren]

BPO	32921
Nosy	@pfmoore, @vstinner, @tjguk, @ezio-melotti, @zware, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-02-23.13:03:18.090>
labels = ['type-bug', 'expert-unicode', 'OS-windows']
title = '.pth files cannot contain folders with utf-8 names'
updated_at = <Date 2018-03-05.17:53:23.182>
user = 'https://bugs.python.org/einaren'

bugs.python.org fields:

activity = <Date 2018-03-05.17:53:23.182>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode', 'Windows']
creation = <Date 2018-02-23.13:03:18.090>
creator = 'einaren'
dependencies = []
files = []
hgrepos = []
issue_num = 32921
keywords = []
message_count = 2.0
messages = ['312635', '313273']
nosy_count = 7.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'steve.dower', 'einaren']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32921'
versions = ['Python 3.6']

Linked PRs

• gh-77102: site: try utf-8 and fallback to locale encoding when reading pth file #117802
• [3.12] gh-77102: site: try utf-8 and locale encoding when reading .pth file (GH-117802) #117917
• [3.11] gh-77102: site: try utf-8 and locale encoding when reading .pth file (GH-117802) #117918

[einaren]

Add "G:\русский язык" to a pth file and start python. it fails with

--------------

Failed to import the site module
Traceback (most recent call last):
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\site.py", line 546, in <module>
 main()
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\site.py", line 532, in main
 known_paths = addusersitepackages(known_paths)
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\site.py", line 287, in addusersitepackages
 addsitedir(user_site, known_paths)
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\site.py", line 209, in addsitedir
 addpackage(sitedir, name, known_paths)
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\site.py", line 165, in addpackage
 for n, line in enumerate(f):
 File "C:\Program Files\ROXAR\RMS dev_release\windows-amd64-vc_14_0-release\bin\lib\encodings\cp1252.py", line 23, in decode
 return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 8: character maps to <undefined>

---

This might very well have sideeffects, but adding "encoding='utf-8'" to the open() call in site.py def addpackage seems to fix the issue for me

[zooba]

Yes, it'll have significant side effects. The default file encoding on Windows is your configured code page (1252, in your case), and there's no good way around that default. The easiest immediate fix is to re-encode that file yourself.

Perhaps what we could do instead is allow the first line of a .pth file to be a coding comment? Then site.py can reopen the file with the specified encoding.

(FWIW, when I added the ._pth file, I explicitly made it UTF-8. But it had no history at that time so it was safe to do so.)

[abravalheri]

This seems to be related to the problem in pypa/setuptools#3937.

I wonder what is the position of the core maintainers about it. Is this a valid bug, or is it considered a wontfix/not a bug at all?

Considering the Python uses UTF-8 by default for source files, and that .pth are kind of source files (they can even contain valid Python code), my first reaction was to find the current behaviour weird...

One important point is that most of the times the .pth file is not produced by the users themselves but by the packaging system for editable installs.

While the packaging system could use locale.getpreferredencoding() to create the .pth files in the native encoding¹, if at a later stage the users set PYTHONUTF8 or invoke python -X utf8, these files would stop working. Ideally, we would like to produce valid .pth files that can be used regardless if the user decides to run Python in UTF-8 mode or not.

@zooba do you have any advice on how to handle this?

Footnotes

1. According to the docs locale.getpreferredencoding() has side effects and is not thread safe (it can be invoked with do_setlocale=False, but then, the docs seem to indicate the function may fail to get the correct value) -- Not sure if side-effects and thread-safety is something packaging have to worry about... ↩

[zooba]

Is this a valid bug, or is it considered a wontfix/not a bug at all?

It's valid, but unfortunately not fixable outside of changing the default encoding for the entire process.

Considering the Python uses UTF-8 by default for source files, and that .pth are kind of source files ...

Being able to include code in the files is the weird bit, and is likely to be removed well before the default encoding is changed. If it were meant to be a source file, it'd have a .py extension (like sitecustomize.py does).

if at a later stage the users set PYTHONUTF8 or invoke python -X utf8, these files would stop working

Yep, which is why users have to opt-in to these options, and if it breaks then they need to opt-out or fix it themselves. Other tools don't need to be concerned about this scenario - just use the default encoding.

do you have any advice on how to handle this?

Use a regular open(path, 'w') call to write to the file, suppressing a warning if needed.

Unfortunately, it seems it currently requests locale encoding explicitly to read, when it ought to be using the default. If it were the default, everyone would get onto UTF-8 when that default changes, but as it stands it'll be stuck on locale encoding forever. So I guess the advice is to explicitly use locale encoding until we fix that.

[abravalheri]

Thank you very much @zooba, I am working on pypa/setuptools#4009 trying to follow this recommendation.