Alternate RFC 3986 compliant URI parsing module

[ncoghlan]

BPO	1500504
Nosy	@ncoghlan, @orsenthil, @devdanzin, @merwok, @ambv, @berkerpeksag, @vadmium
Files	urischemes.py: v 0.4 of the urischemes module urischemes.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2006-06-04.14:50:18.000>
labels = ['type-feature', 'library']
title = 'Alternate RFC 3986 compliant URI parsing module'
updated_at = <Date 2015-01-01.01:25:35.044>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2015-01-01.01:25:35.044>
actor = 'berker.peksag'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2006-06-04.14:50:18.000>
creator = 'ncoghlan'
dependencies = []
files = ['7307', '32591']
hgrepos = []
issue_num = 1500504
keywords = ['patch']
message_count = 16.0
messages = ['50411', '50412', '50413', '50414', '50415', '83920', '86301', '86308', '86348', '109963', '109966', '120245', '120264', '120344', '120349', '202728']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'orsenthil', 'ajaksu2', 'eric.araujo', 'lukasz.langa', 'berker.peksag', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1500504'
versions = ['Python 3.5']

[ncoghlan]

Inspired by (and based on) Paul Jimenez's uriparse
module (http://python.org/sf/1462525), urischemes tries
to put a cleaner interface in front of the URI parsing
engine.

Most of the module works with a URI subclass of tuple
that is always a 5-tuple (scheme, authority, path,
query, fragment).

The authority component is either None, or a
URIAuthority subclass of tuple that is always a 4-tuple
(user, password, host, port).

The function make_uri will create a URI string from the
5 constituent components of a URI. The components do
not need to be strings - if they are not strings, str()
will be invoked on them (this allows the URIAuthority
tuple subclass to be used transparently instead of a
string for the authority component). The result is
checked to ensure it is an RFC-compliant URI.

The function split_uri accepts a string and returns a
URI object with strings as the individual elements.
Invoking str() on this object will recreate a URI
string using make_uri(). The regex underlying this
operation is now broken out and available as module
level attributes like URI_PATTERN.

The functions split_authority and make_authority are
similar, only working solely on the authority component
rather than the whole URI.

The function parse_uri digs into the internal structure
of a URI, also parsing the components. This will
replace a non-empty URI authority component string with
a URIAuthority tuple subclass. Depending on the scheme,
it may also replace other components (e.g. for mailto
links, the path is replaced with a (user, host) tuple
subclass).

The main parsing engine is still URIParser (much the
same as Paul's), but the root of the internal parser
hierarchy is now SchemeParser. This has two subclasses,
URLParser and MailtoParser. The various URL flavours
are now different instances of URLParser rather than
subclasses. All of the actual parsers are available as
module level attributes with the same name as the
scheme they parse. Additionally, each parser knows the
name of the scheme it is intended to parse.

The parse() methods of the individual parsers are now
expected to return a URI object (SchemeParser actually
takes care of this). The parse() method also takes a
dictionary of defaults, which can override the defaults
supplied by the parser instance. The unparse() method
is gone - instead, the scheme parser should ensure that
all components returned are either strings or produce
the right thing when __str__ is invoked (e.g. see
_MailtoURIPath)

The module level 'schemes' attribute is a mapping from
scheme names to parsers that is automatically populated
with all instances of SchemeParser that are found in
the module globals()

urljoin has been renamed to join_uri to match the style
of the other names in the module.

[ncoghlan]

Logged In: YES
user_id=1038590

Updated version attached which addresses some issues raised
by Mike Brown in private mail (the difference between a URI
and a URI reference and some major differences between URI
paths and posix paths).

Also settled on split/join for the component separation and
recombination operations and made the join methods all take
a tuple so that join_x(split_x(uri)) round trips.

Based on the terminology in the RFC, the function to combine
a URI reference with a base URI is now called "resolve_uriref".

[ncoghlan]

Logged In: YES
user_id=1038590

Uploaded version 0.3 which passes all the RFC tests, as well
as the failing 4Suite tests Mike sent me based on version
0.1 and 0.2.

The last 4suite failure went away when I realised those
tests expected to operate in strict mode :)

[ncoghlan]

Logged In: YES
user_id=1038590

Uploaded version 0.4

This version cleans up the logic in resolve_uripath a bit
(use a separate loop to strip the leading dot segments, add
comments explaining meaning of if statements when dealing
with dot segments).

It also exposes EmailPath (along with split_emailpath and
join_emailpath) as public objects, rather than treating them
as internal to the MailtoSchemeParser.

[ncoghlan]

Removed all versions prior to 0.4

[devdanzin]

I'll collect open issues that would be solved by this.

[ncoghlan]

The code itself is no longer the hard part here (hence the easy tag).

The problem is the fact that getting something like this into the
standard library is a tough sell on python-dev because it isn't really a
field tested module, but once people start downloading things from PyPI,
they're more likely to go for something like 4Suite rather than a mere
URI parsing module.

What the issue really needs is someone to champion the benefits of
having this in the standard library.

Now that it is available, it would also be worth looking at updating the
module to use collection.named_tuple instead of creating its own variant
of the same thing.

[devdanzin]

ISTM that gathering the issues where this would help is a good start,
but I haven't had the time to do it yet.

[orsenthil]

I am willing to review this/work on it. But I wonder if this can be
categorized as easy task.

1. Integration to Standard Library will involve compatibility with
   existing parsing, which will invariably involve certain tweaks (with
   discussions/buy-in from others).

2. There are other patches which tries to achieve this purpose;
   consolidation is required.