bpo-29729: uuid.UUID now accepts bytes-like object

[vstinner]

uuid.UUID doesn't require the 'bytes' argument to be an instance of
bytes: accept bytes-like types, bytearray and memoryview for example.

https://bugs.python.org/issue29729

[serhiy-storchaka]

Add a "versionchanged" directive in the module documentation.

Hmm, memoryview is not accepted as bytes_le. And the length check works correctly only with memoryview with byte format.

[vstinner]

Hmm, memoryview is not accepted as bytes_le

Oh, I didn't look at bytes_le. I also modified bytes_le to accept bytes-like objects.

And the length check works correctly only with memoryview with byte format.

I took care of such issues in my updated PR.

[serhiy-storchaka]

Add int.

[vstinner]

I don't understand why bytes_le should accept an int? Do you have an example please?

[serhiy-storchaka]

It should not! But accepts. Add a test for bytes_le=16. 😉

[serhiy-storchaka]

... and 'bytes_le'

[vstinner]

Oops, fixed.

[serhiy-storchaka]

A ValueError is raised if 'bytes_le' is a str of len != 16. To solve this duplicate int.from_bytes() for bytes_le instead of reusing the code for bytes.

[vstinner]

A ValueError is raised if 'bytes_le' is a str of len != 16.

I added an unit tests to ensure that a TypeError is raised if you pass a str of any length to bytes or bytes_le.

[vstinner]

Please don't merge my PR until PR #3796 is merged. IMHO PR #3796 is more important :-)

[serhiy-storchaka]

I added an unit tests to ensure that a TypeError is raised if you pass a str of any length to bytes or bytes_le.

Ah, yes, a bytes constructor raises an error.

But still there is a problem with memoryview with non-byte format. bytes_le=memoryview(b'\0'*16).cast('I') works, but bytes=memoryview(b'\0'*16).cast('I') is rejected and bytes=memoryview(b'\0'*64).cast('I') is accepted by mistake.

[vstinner]

PR #3796 has been merged. I merged master into my branch.

But still there is a problem with memoryview with non-byte format. memoryview(b'\0'*64).cast('I') is accepted by mistake.

I don't think that we should overengineer the code here. We are consenting adults here. You can also find a value to skip input validations. It's really hard to write the perfect validation. If you can a random memoryview, yeah, maybe we get a valid UUID object whereas you shouldn't. Well, I don't think that we should detect this corner case. Why would you do that on purpose?

[vadmium]

It may not be worth supporting every kind of invalid input (unless you have to handle untrusted input). But it is important to implement what the documentation promises (or fix the documentation). As of rev a169e89, bytes has to also support len.

[vadmium]

By above in the comment, do you mean this check, which is actually below the comment? :)

[vstinner]

I removed the comment.

[vstinner]

@vadmium: "It may not be worth supporting every kind of invalid input (unless you have to handle untrusted input). But it is important to implement what the documentation promises (or fix the documentation). As of rev a169e89, bytes has to also support len."

I'm sorry, but I don't understand what you are expecting from me. What do you mean by "bytes has to also support len"? Do you expect a nicer error message if len() fails?

[vstinner]

@serhiy-storchaka, @vadmium: Would you mind to review the latest version of my change, please?

[vadmium]

For your code to work, the object passed as the bytes argument has to implement __len__ returning 16. But your documentation promises any bytes-like object should work (as long as it is a string of 16 bytes). Since the glossary definition of bytes-like objects has no requirements about implementing __len__, I would expect len to not be used on the argument. You could change the code for bytes to make a copy like you do for bytes_le:

bytes=bytes_(bytes) # Convert bytes-like to bytes objectiflen(bytes) !=16 . . .

Or you could use memoryview:

withmemoryview(bytes) asview:
ifview.nbytes!=16 . . .

Or you could change the documentation:

bytes_le now accepts any bytes-like object, and bytes now accepts any bytes-like object for which len returns 16.

[vstinner]

Or you could change the documentation:

UUID constructor documentation is explicit:

 Create a UUID from either a string of 32 hexadecimal digits, a string of 16
bytes as the *bytes* argument, a string of 16 bytes in little-endian order as
the *bytes_le* argument, ...

I guess that your remarks was on my "versionchanged" markup and the NEWS entry. I added "of 16 bytes" there.

[pitrou]

>>> a = array.array('i', [-1]*4)
>>> x = int.from_bytes(a, byteorder='big')
>>> len(a)
4
>>> x.bit_length()/8
16.0

And conversely:

>>> a = array.array('i', [-1]*16)
>>> x = int.from_bytes(a, byteorder='big')
>>> len(a)
16
>>> x.bit_length()/8
64.0

[vadmium]

Yes I was talking about the versionchanged note, which now says "any bytes-like objects of 16 bytes". But I don't think this is clear enough. Antoine and Serhiy already gave examples of bytes-like objects containing exactly 16 bytes where len returns a different number.

Since you already copy bytes_le into a true bytes object, why not parallel this for the bytes parameter?

ifbytesisnotNone:
# Check for int so that bytes=16 triggers an exception belowifnotisinstance(bytes, (bytes_, bytearray, int_)):
bytes=bytes_(bytes) # Copy any bytes-like objectiflen(bytes) !=16:
raiseValueError('bytes is not a 16-byte string')
int=int_.from_bytes(bytes, 'big')

[vstinner]

I decided to work on https://bugs.python.org/issue29729 because I understand that the proposed change was just to remove an assertion. That's all.

It seems like you expect much more changes, much more complex code. It's already my 6th version of my pull request.

Honestly, I don't care much of the uuid module. I don't think that anyone ever tried to call UUID(bytes=value) with a type different than bytes (ok, apart the bug reporter ;-)). If I would really need to pass non-bytes, having to cast manually with bytes(value) wouldn't burn my hands.

For all these reasons, I abandon my change. Feel free to reuse my code if someone wants to pursue the effort on UUID enhancement ;-)

[serhiy-storchaka]

The problem with adding support of bytes-like objects is that in different cases this term has different meaning. The definition in the glossary is too wide, in many particular cases we need the support of not just the buffer protocol, but len(), indexing, slicing, iterating, startswith() or even lower().

I think we need different terms for different levels of "bytesness".

[vstinner]

I think that we should replace the assertion with a regular if and raise a proper TypeError, rather than trying to support "bytes-like" objects. It's fine to require bytes on UUID. UUID is not a common type, and passing bytes to UUID is even less common (IMHO).