gh-67041: Allow to distinguish between empty and not defined URI components

[serhiy-storchaka]

Changes in the urllib.parse module:

• Add option allow_nonemissing_as_none in urlparse(), urlsplit() and urldefrag(). If it is true, represent not defined components as None instead of an empty string.
• Add option keep_empty in urlunparse() and urlunsplit(). If it is true, keep empty non-None components in the resulting string. By default it is the same as the allow_none value for the result of the urlparse() and urlsplit() calls.
• Add option keep_empty in the geturl() method of DefragResult, SplitResult, ParseResult and the corresponding bytes counterparts.

• Issue: urllib.parse wrongly strips empty #fragment, ?query, //netloc #67041

[serhiy-storchaka]

It is now ready to review. The status of allow_none is now saved in the DefragResult, SplitResult and ParseResult objects, so in most cases there is no need to pass the keep_empty argument. geturl() no longer needs the keep_empty parameter.

Unfortunately, these objects now have __dict__ and no longer immutable. This is because non-empty __slots__ is not compatible with tuple subclasses. This is a separate complex issue. I'll try to find a solution of it, but it may be difficult.

The long term plan is to make allow_none True by default, and later deprecate allow_none=False. keep_empty=False can still be useful.

[serhiy-storchaka]

I am sorry, I forget to copy the _keep_empty attribute in copying/encoding/decoding methods. Now the PR is ready for review.

@orsenthil, @barneygale, could you please make a review?

[merwok]

I think that allow_none is not the correct name: this is not about allowing nones in the input (like how allow_fragments allows fragment in the input), but about controlling the data returned.

Possible alternatives:

• use_none (short)
• missing_as_none
• use_none_for_empty

[serhiy-storchaka]

Maybe missing_as_empty? In future the behavior will be changed to use None for missing by default.

[merwok]

I think it should be missing_as_empty_string (short for return missing values as empty strings), which is long!

To change the default in the future, do you plan on adding a warning first?
I could see many projects not paying attention to the new param and getting surprised (or silent bugs) a few years after.

[serhiy-storchaka]

To change the default in the future, do you plan on adding a warning first?

On one hand, a warning will inform everyone about the change (it should be a FutureWarning). You will have to pass explicit True or False to silence it and get your behavior. This how we normally do. On other hand, the warning will unnecessary disturb those who are fine with any behavior. We will discuss this when the time come.

[serhiy-storchaka]

I think it should be missing_as_empty_string (short for return missing values as empty strings), which is long!

Currently, missing and empty component are not distinguishable. This parameter will allow to distinguish them. This PR adds also the keeps_empty parameter. So, "empty" refers not only to string, but to component. missing_component_as_empty_component, or simply missing_as_empty. Will it work?

[merwok]

Ah your’re right, there is empty component and empty string.

missing_as_none is good!

[orsenthil]

Looks like we maintain a patch for the iOS build which is failing to apply with this PR.

 export PATH=/Users/runner/work/cpython/cpython/Apple/iOS/Resources/bin:/Users/runner/work/cpython/cpython/cross-build/arm64-apple-ios-simulator/prefix:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
> make -j 3
arm64-apple-ios-simulator-clang -c -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -mios-version-min=13.0 -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include -DPy_BUILD_CORE -o Programs/python.o ../../Programs/python.c
arm64-apple-ios-simulator-clang -c -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -mios-version-min=13.0 -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include -DPy_BUILD_CORE -o Parser/token.o ../../Parser/token.c
1 out of 1 hunks failed while patching 'Lib/test/test_urlparse.py'

[orsenthil]

This file https://github.com/python/cpython/blob/main/Mac/Resources/app-store-compliance.patch (a meta-patch) needs to be updated for the new behavior (returning None instead of '') for it apply cleanly.

[orsenthil]

This file https://github.com/python/cpython/blob/main/Mac/Resources/app-store-compliance.patch should be updated with this content.

diff --git a/Mac/Resources/app-store-compliance.patch b/Mac/Resources/app-store-compliance.patch
index f4b7decc01c..e5d1e7da866 100644
--- a/Mac/Resources/app-store-compliance.patch
+++ b/Mac/Resources/app-store-compliance.patch
@@ -1,21 +1,21 @@
diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py
-index d6c83a75c1c..19ed4e01091 100644
+index 49a292df934..e1669a0c9b2 100644
--- a/Lib/test/test_urlparse.py
+++ b/Lib/test/test_urlparse.py
-@@ -237,11 +237,6 @@ def test_roundtrips(self):
- '','',''),
+@@ -282,11 +282,6 @@ def test_qs(self, orig, expect):
+ None,None,None),
('git+ssh', 'git@github.com','/user/project.git',
- '', '')),
+ None, None)),
- ('itms-services://?action=download-manifest&url=https://example.com/app',
-- ('itms-services', '', '', '',
-- 'action=download-manifest&url=https://example.com/app', ''),
+- ('itms-services', '', '', None,
+- 'action=download-manifest&url=https://example.com/app', None),
- ('itms-services', '', '',
-- 'action=download-manifest&url=https://example.com/app', '')),
+- 'action=download-manifest&url=https://example.com/app', None)),
('+scheme:path/to/file',
- ('', '', '+scheme:path/to/file', '', '', ''),
- ('', '', '+scheme:path/to/file', '', '')),
+ (None, None, '+scheme:path/to/file', None, None, None),
+ (None, None, '+scheme:path/to/file', None, None)),
diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py
-index 8f724f907d4..148caf742c9 100644
+index e917f8b61bb..8575172573f 100644
--- a/Lib/urllib/parse.py
+++ b/Lib/urllib/parse.py
@@ -59,7 +59,7 @@

And patch application is successful.

$patch -p1 < Mac/Resources/app-store-compliance.patch
patching file 'Lib/test/test_urlparse.py'
patching file 'Lib/urllib/parse.py'

And

 ./python.exe Lib/test/test_urlparse.py -v

was successful with this change.

[serhiy-storchaka]

Thank you @orsenthil. I was confused by this failure. The original issue is interesting: #120522.