Fix unicode message in exceptions

[daspecster]

See #2346

@dhermes looking at the magic methods and playing with __bytes__ and __unicode__, the solutions I had felt really hacky. i.e. checking for six.PY3 in __str__ and then just calling self.__bytes__() instead.

I'm not sure if it's good form to use __future__ here but it seems to be a valid way to solve the issue.

[dhermes]

@daspecster Ping me when Travis passing? (I'm working on release notes right now so a bit scatterbrained.)

[tseaver]

I'm not in favor of the from __future__ import unicode_literals wallpaper. Let's do the check for unicode ourselves.

[daspecster]

@tseaver ok, can do.

[dhermes]

@daspecster Agree with @tseaver. That won't help detect bytes passed in, just enforce it on whatever we type.

[daspecster]

Here's a cleaner way to handle it, however when running the follow test with python 2.7, you get an exception thrown by the print function.

exception_test.py

fromgoogle.cloudimportexceptionstry:
a=u'That\u2019s all we know'print(a)
raiseexceptions.NotFound(a)
exceptexceptions.GoogleCloudErrorase:
print(e)

Output:

╰─$ python test.py
That’s all we know
Traceback (most recent call last):
File "test.py", line 7, in<module>
print(e)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 8: ordinal not in range(128)

If that is not something we're concerned with then we are good to go.
Otherwise I can detect the python version(if six.PY2) and then convert accordingly.

[dhermes]

@daspecster You shouldn't have to branch for six.PY2. Hopefully there is a way to hook into __str__ for our exception classes. (DERP) Putting the following in foo.py

from __future__ importprint_functionclassNotFound(Exception):
def__init__(self, message):
super(NotFound, self).__init__(message)
self.message=messageself._errors= ()
def__str__(self):
return'%d %s'% (404, self.message)
VAL=u'it\u2019s all good'print(VAL)
print('='*40)
EXC=NotFound(VAL)
print(EXC)

we see the failure

$ python2.7 foo.py
it’s all good
========================================
Traceback (most recent call last):
File "foo.py", line 27, in <module>
print(EXC)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 6: ordinal not in range(128)
$ python3.4 foo.py
it’s all good
========================================
404 it’s all good
$ python3.5 foo.py
it’s all good
========================================
404 it’s all good

---

NOTE This min-repro was too min.

Putting the following in foo.py

from __future__ importprint_functionprint(u'it\u2019s all good')

it works as expected:

$ python2.7 foo.py
it’s all good
$ python3.4 foo.py
it’s all good
$ python3.5 foo.py
it’s all good

[dhermes]

OK, here is the problem: In Python 2, the result of __str__ can't be of type unicode if it can't be cast back to str/bytes using the ascii encoding. So we could use _helpers._to_bytes

[daspecster]

I was thinking about _to_bytes but it just seems like there should be a clean way to do this. _to_bytes > unicode_literals for sure though.

Should we used _to_bytes in __bytes__ and then just call __bytes__ from __str__?
Seems..not right?

Since the distinction between string and unicode has been done away with in Python 3, __unicode__ is gone and __bytes__ (which behaves similarly to __str__ and __unicode__ in 2.7) exists for a new built-in for constructing byte arrays.

From the bottom of that link you posted before.

[daspecster]

@dhermes, so is this just a REPL/print issue then?

That is the solution for #2346 is just to from __future__ import print_function?

[dhermes]

No. The issue is __str__. It needs a return value that is bytes / str or unicode that can be converted directly to ASCII (that's why you see the error, they use the default encoding which is ASCII)

[daspecster]

I'm apparently missing something here.

I have the code you demonstrated above, but I'm still getting the error.

from __future__ importprint_functionclassNotFound(Exception):
def__init__(self, message):
super(NotFound, self).__init__(message)
self.message=messageself._errors= ()
def__str__(self):
return'%d %s'% (404, self.message)
VAL=u'it\u2019s all good'print(VAL)
print('='*40)
EXC=NotFound(VAL)
print(EXC)

╰─$ python2.7 test.py 1 ↵
it’s all good
========================================
Traceback (most recent call last):
File "test.py", line 20, in<module>
print(EXC)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 6: ordinal not in range(128)

[dhermes]

@daspecster The example I posted doesn't fix it, I just posted it because my original snippet was busted.

The issue is that print(...) tries to cast to str in Python, which is essentially amounts to calling encode() on the unicode object.

Python2.7.12 (default, Jul172016, 01:21:14) [GCC4.8.4] onlinux2Type"help", "copyright", "credits"or"license"formoreinformation.
>>>classA(object):
... def__str__(self):
... returnu'\u2019'
... >>>a=A()
>>>a.__str__()
u'\u2019'>>>str(a)
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>UnicodeEncodeError: 'ascii'codeccan't encode character u'\u2019' inposition0: ordinalnotinrange(128)
>>>a.__str__().encode()
Traceback (mostrecentcalllast):
File"<stdin>", line1, in<module>UnicodeEncodeError: 'ascii'codeccan't encode character u'\u2019' inposition0: ordinalnotinrange(128)
>>>unicode(a)
u'\u2019'

[daspecster]

@dhermes PTAL when you have a chance.

[daspecster]

I had to remove this test for _to_bytes. Actually I can just remove the assertRaises.

deftest_unicode_non_ascii(self):
value=u'\u2013'# Long hyphenencoded_value=b'\xe2\x80\x93'self.assertRaises(UnicodeEncodeError, self._callFUT, value)
self.assertEqual(self._callFUT(value, encoding='utf-8'),
encoded_value)

Also, to get tests for py27 and py35 to pass and cover, I had to check for two message strings.

[dhermes]

I had to remove a test for_to_bytes to get this to pass.

Huh? Why should that be necessary?

[tseaver]

@daspecster Travis build cancelled. Is there another push coming?

[daspecster]

@tseaver yup! Sorry, I was trying to get my commits GPG verified.

[daspecster]

@dhermes, @tseaver PTAL. I'm ready to move on from this one lol.

[dhermes]

LGTM