Inconsistent behavior of pathlib.WindowsPath with drive paths

[kmaork]

BPO	36305
Nosy	@pfmoore, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @kmaork, @nmat
PRs	bpo-36305: Fixes to path handling and parsing in pathlib #12361

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-15.15:37:05.961>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = 'Inconsistent behavior of pathlib.WindowsPath with drive paths'
updated_at = <Date 2021-03-20.06:55:56.836>
user = 'https://github.com/kmaork'

bugs.python.org fields:

activity = <Date 2021-03-20.06:55:56.836>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2019-03-15.15:37:05.961>
creator = 'kmaork'
dependencies = []
files = []
hgrepos = []
issue_num = 36305
keywords = ['patch']
message_count = 18.0
messages = ['337999', '338007', '338011', '338663', '338667', '338669', '338675', '338677', '338680', '338685', '338708', '338727', '338768', '340388', '340396', '340826', '343621', '343784']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'kmaork', 'nmat']
pr_nums = ['12361']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36305'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• GH-80486: Fix handling of NTFS alternate data streams in pathlib #102454

[kmaork]

The behavior of WindowsPath is undefined when used with drive paths, specifically with no trailing slash:

WindowsPath('cc:').absolute() -> WindowsPath('C:/Users/maor/cc:')
WindowsPath('c:').absolute() -> WindowsPath('c:')
WindowsPath('c:').is_absolute() -> False
WindowsPath('c:') / 'p' -> WindowsPath('c:p')
WindowsPath('c:p').absolute() -> WindowsPath('c:p')
WindowsPath('c:p').is_absolute() -> False

[eryksun]

WindowsPath('cc:').absolute() -> WindowsPath('C:/Users/maor/cc:')

This is correct. "cc:" is not a drive, i.e. the drive attribute is an empty string. pathlib handles this as an unqualified filename that gets resolved relative to the process current working directory.

Note that Windows allows a trailing colon for DOS devices such as "con:". Colons can also be used for file streams, but "cc:" isn't valid. The anonymous stream can only be specified explicitly by including the stream type, e.g. "cc::$DATA".

WindowsPath('c:').is_absolute() -> False
WindowsPath('c:p').is_absolute() -> False
WindowsPath('c:') / 'p' -> WindowsPath('c:p')

These are correct. Drive-relative paths depend on the current working directory of the drive. By definition they cannot be absolute. And the last one can be no more resolved than WindowsPath('spam/eggs') / 'p'.

Note that Windows gets the working directory for drives from 'hidden' environment variables such as "=C:". If no value is set for a drive, it defaults to the root directory. Windows uses these values, but never sets them itself. Applications and libraries such as the C runtime library are responsible for setting the values. CMD and Python both set them, but PowerShell does not.

WindowsPath('c:').absolute() -> WindowsPath('c:')
WindowsPath('c:p').absolute() -> WindowsPath('c:p')

This is a bug. absolute() should resolve "C:" to the current directory on the drive and join it with the remaining parts.

[kmaork]

I agree.
So I'll open a PR that will:

1. Make the .absolute method return correct values in these cases
2. Add a regression test for this behavior

[kmaork]

Update after editing my PR - the bugs are:

1. WindowsPath('C:a').absolute() should return WindowsPath('C:\\d\\a') but returns WindowsPath('C:a').
   This is caused by flawed logic in the parse_parts method of the _Flavour class.

2. WindowsPath('./b:a').absolute() should return WindowsPath('C:\\d\\b:a') but returns WindowsPath('b:a').
   This is caused by the limited interface of parse_parts, and affects the Path.absolute, Path.expanduser and Path.__rtruediv__ methods.

3. WindowsPath('./b:a').resolve() should return WindowsPath('C:\\d\\b:a') but returns WindowsPath('b:a').
   This is caused by missing logic in the resolve method and in Path.__str__

It'd be great if someone could review the PR so we can make progress with fixing the bugs.

[pfmoore]

(Note: I consider all of these to be *extremely* obscure corner cases)

1. WindowsPath('C:a').absolute() should return WindowsPath('C:\\d\\a') but returns WindowsPath('C:a').
   This is caused by flawed logic in the parse_parts method of the _Flavour class.

2. WindowsPath('./b:a').absolute() should return WindowsPath('C:\\d\\b:a') but returns WindowsPath('b:a').
   This is caused by the limited interface of parse_parts, and affects the Path.absolute, Path.expanduser and Path.__rtruediv__ methods.

[Note there is no absolute() method - I assume you mean resolve()]

Why? If './b:a' resulted from joining '.' and 'b:a', then it seems to
me that 'b:a' is "absolute-ish" (drive-relative) and so any preceding
path should be ignored (just like joining '.' and '/a/b/c' on POSIX).
The problem here is more to do with the fact that the simple POSIX
"absolute or relative" dichotomy isn't complete for Windows -
drive-relative paths like C:foo have some of the characteristics of
absolute paths and some of the characteristics of relative paths.

To put it another way, I'm comfortable with WindowsPath("./b:a")
returning "WindowsPath("b:a"). Of course, that's because I read b:a as
a drive plus a filename. If you read it as a filename with a stream,
then your interpretation is correct. But unless you check for all of
the valid drives currently available, it's not possible to make that
choice - both interpretations are equally valid. In fact, if you have
a file "C", and add a stream "file1" to it, is "C:file1" a file on the
C drive, or a stream in the file C? The problem is genuinely ambiguous
in that case, and cannot be solved without making an arbitrary choice.
Worse, WindowsPath("w:fred").resolve(strict=False) technically can't
even take account of whether drive w exists or file w exists or has a
stream fred. In that case, the question is fundamentally unanswerable.

I'd be reluctant to "solve" this issue with a fix that doesn't address
that problem - it would simply be replacing one weird behaviour with
another in an obscure corner case. (It may be that the "fix" is simply
to document the choices that the code currently makes).

3. WindowsPath('./b:a').resolve() should return WindowsPath('C:\\d\\b:a') but returns WindowsPath('b:a').
   This is caused by missing logic in the resolve method and in Path.__str__

Same as (2)

It'd be great if someone could review the PR so we can make progress with fixing the bugs.

I don't think we should worry about the PR until it's clearly
established what the correct resolution actually is (or even whether
we consider all of these cases as bugs).

[kmaork]

(Note: I consider all of these to be *extremely* obscure corner cases)
One bug was enough for me :)

[Note there is no absolute() method - I assume you mean resolve()]
Of course there is an absolute() method, I'm not sure what you are saying...

it seems to me that 'b:a' is "absolute-ish" (drive-relative)
I think that is incorrect. As written here: https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file#fully-qualified-vs-relative-paths, "If a file name begins with only a disk designator but not the backslash after the colon, it is interpreted as a relative path to the current directory on the drive with the specified letter."
In that case, WindowsPath('C:a').is_absolute() should return False, (as it does today) and WindowsPath('C:a').absolute() should return a path on drive C:, with 'a' joined with the working directory in drive C:.

I'm comfortable with WindowsPath("./b:a") returning WindowsPath("b:a")
I disagree with that as well. "./b:a" is explicitly a path relative to the CWD (to a file named b:a). On the other hand, "b:a" should be (an is, in most windows programs) interpreted as a drive relative path.
For example, the ntpath module handles these cases correctly. When located in the directory C:\\d, this is the ntpath behavior:
ntpath.abspath('b:a') -> 'B:\\a'
ntpath.abspath('.\\b:a') -> 'C:\\d\\b:a'

In conclusion, I stand by my original fix offers. They are correct according to windows' documentation and behavior.

[pfmoore]

> [Note there is no absolute() method - I assume you mean resolve()]
Of course there is an absolute() method, I'm not sure what you are saying...

Huh, weird. It's not in
https://docs.python.org/3.7/library/pathlib.html But you're right, it
does exist...

> it seems to me that 'b:a' is "absolute-ish" (drive-relative)
I think that is incorrect. As written here: https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file#fully-qualified-vs-relative-paths, "If a file name begins with only a disk designator but not the backslash after the colon, it is interpreted as a relative path to the current directory on the drive with the specified letter."
In that case, WindowsPath('C:a').is_absolute() should return False, (as it does today) and WindowsPath('C:a').absolute() should return a path on drive C:, with 'a' joined with the working directory in drive C:.

OK, sure. My point is that "relative path to the current directory on
the drive with the specified letter" isn't a concept that matches how
"relative paths" are typically interpreted (most code interprets
"relative" as "relative to the CWD", and doesn't deal with the
possibility of per-drive CWDs).

> I'm comfortable with WindowsPath("./b:a") returning WindowsPath("b:a")
I disagree with that as well. "./b:a" is explicitly a path relative to the CWD (to a file named b:a). On the other hand, "b:a" should be (an is, in most windows programs) interpreted as a drive relative path.

Windows is inconsistent here - it can *also* be interpreted as a path
to a stream within a file. But it (virtually) never is.

For example, the ntpath module handles these cases correctly. When located in the directory C:\\d, this is the ntpath behavior:
ntpath.abspath('b:a') -> 'B:\\a'
ntpath.abspath('.\\b:a') -> 'C:\\d\\b:a'

That second case only results in a valid filename if b:a is viewed as
a file-with-stream, but the first only makes sense if b:a is viewed as
a directory-relative file.

Also, in effect it means that Path(".") / some_path can return a
completely different location than some_path. Following documented
behaviour or not, that violates a pretty fundamental assumption that
users would expect to hold. And I think the problem is that
"drive-relative paths" are somewhat odd things that don't fit well in
the POSIX-derived model that Python's path APIs follow.

In conclusion, I stand by my original fix offers. They are correct according to windows' documentation and behavior.

I remain of the view that the Windows documentation introduces a
concept ("relative path to the current directory on a specific drive")
that isn't well modelled by the current APIs, and the only "proper"
solution is to extend the API (like with the ideas of "drive" and
"reserved filenames", which are Windows-specific, but supported
everywhere). In the absence of that, I believe that any "fix" within
the existing model will have odd edge cases, and I don't think these
ones (specifically the "./b:a" cases) have *any* "obvious" answer.

I'm happy to agree to differ on this point, though. If the new
behaviour is just a side-effect of fixing absolute() to match the
cases Eryk commented on, then that's fine - I just wouldn't describe
the particular ./b:c cases as "bug fixes", rather as "changes in the
behaviour of cases no-one should actually care about" :-)

BTW, was there an actual use case for this issue, or was it simply a
theoretical concern? I'm actually much happier considering these cases
as "undefined" in practice (and just "not mentioned" in the docs),
rather than trying to pin it down precisely. I don't know of a case
where it actually benefits us to document this level of arguable
behaviour.

[kmaork]

OK, sure. My point is that "relative path to the current directory on
the drive with the specified letter" isn't a concept that matches how
"relative paths" are typically interpreted (most code interprets
"relative" as "relative to the CWD", and doesn't deal with the
possibility of per-drive CWDs).

In pathlib, each path object has a drive, a root and path parts. Seeing pathlib's logic and tests (many tests are specifically testing the behavior of drive-relative paths), I think pathlib completely supports drive relative paths, with the exception of a few small bugs.

> > I'm comfortable with WindowsPath("./b:a") returning WindowsPath("b:a")
> I disagree with that as well. "./b:a" is explicitly a path relative to the CWD (to a file named b:a). On the other hand, "b:a" should be (an is, in most windows programs) interpreted as a drive relative path.
Windows is inconsistent here - it can *also* be interpreted as a path
to a stream within a file. But it (virtually) never is.

I think that when parsing windows paths, pathlib should behave exactly like the windows API does. This is crucial for interaction with the windows API itself or with other applications that might use it. I don't see any other way to parse windows paths other than according to the normal windows behavior.
Having said that, pathlib does a pretty good keeping the compatibility with the windows API, except for the small cases I found and brought forward in this issue report. From the information I gathered, when a path starts with one letter followed by a colon, windows treats it as a drive and continues parsing the rest of the path separately. That means that if you want to specify a path to a file in the CWD, with a single-character name and a file stream, you must precede the path with a "./" (See eryksun's comment on my PR before I fixed it #12361 (comment)).
Here is an example for the behavior of the windows API in this case:
win32api.GetFullPathName('b:a') -> 'B:\\a'
win32api.GetFullPathName('./b:a') -> 'C:\\Users\\maor\\b:a'

Also, in effect it means that Path(".") / some_path can return a
completely different location than some_path.

This behavior is completely normal. Should WindowsPath('C:\\f') / WindowsPath('D:\\f2') return anything other than WindowsPath('D:/f2')?

And I think the problem is that "drive-relative paths" are somewhat odd things that don't fit well in the POSIX-derived model that Python's path APIs follow.

As I wrote earlier, I think this is incorrect as the pathlib.Path class holds the attributes _drv, _root and _parts, which allows it to fully support drive-relative paths, by having a _drv and not having a _root.

I'm happy to agree to differ on this point, though. If the new
behaviour is just a side-effect of fixing absolute() to match the
cases Eryk commented on, then that's fine - I just wouldn't describe
the particular ./b:c cases as "bug fixes", rather as "changes in the
behaviour of cases no-one should actually care about" :-)

I'm still that my case is convincing enough, but if not - does that require me to make any changes in order to make progress with my PR?

BTW, was there an actual use case for this issue, or was it simply a
theoretical concern?

I've had an annoying bug using pathlib, traced it to the first bug I've presented in this issue, and discovered a few similar unhandled edge cases. Again, the "bugginess" I set upon to fix (call it a bug or an undefined behavior) is an incompatibility issue with the way paths are normally treated in windows.

[pfmoore]

does that require me to make any changes in order to make progress with my PR?

I'm not going to block this PR. I'd prefer it if we at least
documented the agreed behaviour, so that in future people don't come
along and say the new behaviour is wrong, but again I'm not going to
insist. I doubt I'll ever hit this edge case myself (either in code of
my own, or when working with others) so my only real interest is in
flagging up the concern. I'd like to hear what Eryk has to say on the
matter, though.

Ultimately the only person whose views matter are yours (as the person
writing the code) whoever commits the change.

[kmaork]

Alright, documentation is always good :)
I'll be glad to add some, but could you please point me to the place in the code where you think it should go? (or just comment on the PR)