[WIP] Fix UTF-8 handling by pyhalov · Pull Request #76 · OpenIndiana/pkg5

pyhalov · 2020-03-07T12:16:39Z

No description provided.

This reverts commit d3187bb.

AndWac

The changes look good so far. What makes you sure that you didn't miss an open() call? There are lots of other places where open() is being called...

pyhalov · 2020-03-08T07:45:49Z

# Ran 1132 tests in 18490.569s - skipped 1 tests.
Skipped tests:
cli.t_pkg_help.py TestPkgHelp.test_help_character_encoding: Test Skipped: The test system must have the ja_JP.eucJP locale installed to run this test.
======================================================================
BASELINE MATCH
======================================================================

pyhalov · 2020-03-08T08:30:21Z

@citrus-it What do you think about utf-8 test? The only issue I see is that now you should run test suite with UTF-8 locale

pyhalov · 2020-03-10T17:46:17Z

@citrus-it It's not enough.
Got this exception with patched pkg:

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/ca-certificates@1.0,5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) \
Traceback (most recent call last):
File "/usr/bin/pkg", line 5968, in handle_errors
__ret = func(*args, **kwargs)
File "/usr/bin/pkg", line 5954, in main_func
pargs=pargs, **opts)
File "/usr/bin/pkg", line 2276, in install
display_plan_cb=display_plan_cb, logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
logger=logger, **kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
display_plan_cb=display_plan_cb)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
for pd in api_plan_func(**kwargs):
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
log_op_end_all=True)
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
self._img.make_install_plan(**kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
reject_list=reject_list)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
self.__call_imageplan_evaluate(ip)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
ip.evaluate()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3879, in evaluate
self.__evaluate_pkg_plans()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4029, in __evaluate_pkg_plans
ccancel=self.__check_cancel, progtrack=self.__progtrack)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 536, in wrapper
return f(instance, *fargs, **f_kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1588, in prefetch_manifests
mfstlist, excludes)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1681, in _prefetch_manifests_list
dl_path)
File "/usr/lib/python3.5/vendor-packages/pkg/client/transport/transport.py", line 1780, in _verify_manifest
mcontent = mf.read()
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 36861: ordinal not in range(128)

pyhalov · 2020-03-10T19:11:11Z

It's still not complete, now it dies without exception...

$ env LC_ALL=C pfexec pkg install --no-backup-be pkg://userland/crypto/ca-certificates@1.0,5.11-2019.0.0.7:20200310T174048Z
Creating Plan (Download Manifests 1/1) |
pkg: 'ascii' codec can't encode character '\u0151' in position 36861: ordinal not in range(128)

pyhalov · 2020-03-10T19:56:36Z

The same issue - dies on writing to /var/pkg/cache/publisher/userland/pkg/crypto%2Fca-certificates/1.0%2C5.11-2019.0.0.7%3A20200310T174048Z/manifest.file.uosjwiqq

pyhalov · 2020-03-10T20:16:12Z

Now more issues:

Creating Plan (Checking for conflicting actions): -
Traceback (most recent call last):
File "/usr/bin/pkg", line 5968, in handle_errors
__ret = func(*args, **kwargs)
File "/usr/bin/pkg", line 5954, in main_func
pargs=pargs, **opts)
File "/usr/bin/pkg", line 2276, in install
display_plan_cb=display_plan_cb, logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1526, in _install
logger=logger)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1404, in __api_op
logger=logger, **kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1258, in __api_plan
display_plan_cb=display_plan_cb)
File "/usr/lib/python3.5/vendor-packages/pkg/client/client_api.py", line 1230, in __api_plan
for pd in api_plan_func(**kwargs):
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1579, in __plan_op
log_op_end_all=True)
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1207, in __plan_common_exception
six.reraise(exc_type, exc_value, exc_traceback)
File "/usr/lib/python3.5/vendor-packages/six.py", line 693, in reraise
raise value
File "/usr/lib/python3.5/vendor-packages/pkg/client/api.py", line 1513, in __plan_op
self._img.make_install_plan(**kwargs)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4511, in make_install_plan
reject_list=reject_list)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4492, in __make_plan_common
self.__call_imageplan_evaluate(ip)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 4404, in __call_imageplan_evaluate
ip.evaluate()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3880, in evaluate
self.__merge_actions()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 4747, in __merge_actions
self.__find_all_conflicts()
File "/usr/lib/python3.5/vendor-packages/pkg/client/imageplan.py", line 3371, in __find_all_conflicts
offset_dict = self.image._load_actdict(self.__progtrack)
File "/usr/lib/python3.5/vendor-packages/pkg/client/image.py", line 3881, in _load_actdict
for line in of:
File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5250: ordinal not in range(128)

pyhalov · 2020-03-10T20:28:12Z

And now we have to fix open() all over src/modules/client/image.py...

pyhalov · 2020-03-11T05:46:06Z

Now we can get another error:

Removing old actions 1/1
Action install failed for 'etc/certs/CA/NetLock_Arany_(Class_Gold)_F\u0151tan\xfas\xedtv\xe1ny.pem' (pkg://userland/crypto/ca-certificates):
UnicodeEncodeError: 'ascii' codec can't encode character '\u0151' in position 42: ordinal not in range(128)

Toasterson · 2021-11-03T14:52:52Z

@pyhalov With #93 could we merge this or are we still stuck on one case where it still calls open without encoding?

pyhalov · 2021-11-03T15:13:16Z

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

Toasterson · 2021-11-03T15:16:05Z

Not sure. What happens if someone launches IPS with 'C' locale? Can moving IPS to recent python (>=3.7) help with the issue?

No we needed to remove C locale and switch to C.UTF-8 one should not call IPS with just the C locale. We could make a patch to the python source to fix the if case that checks if the string contains .UTF-8 in locale and simply tell it to assume UTF-8 aswell in those cases.

pyhalovand others added 2 commits March 7, 2020 15:14

Revert "Fix decoding file text data as ASCII"
f006b53
This reverts commit d3187bb.

Fix errors resulting from UTF-8 characters in manifest files
72fa78d

pyhalov added the testing needed label Mar 7, 2020

AndWac reviewed Mar 7, 2020
View reviewed changes

Test pkg_verify with utf-8 pkg data
900da3c

pyhalov changed the title ~~Fix UTF-8 handling~~[WIP] Fix UTF-8 handlingMar 10, 2020

pyhalov added the needs_work label Mar 10, 2020

pyhalov force-pushed the utf-8 branch from d149651 to 0689757Compare March 10, 2020 18:29

pyhalov force-pushed the utf-8 branch from 0689757 to 12dd412Compare March 10, 2020 19:36

pyhalov force-pushed the utf-8 branch from 12dd412 to 94a7041Compare March 10, 2020 20:03

More fixes for UTF-8 characters in manifests files
a2b4a1f

pyhalov force-pushed the utf-8 branch from 94a7041 to a2b4a1fCompare March 11, 2020 05:06

Navigation Menu

Uh oh!

[WIP] Fix UTF-8 handling - #76

[WIP] Fix UTF-8 handling#76
pyhalov wants to merge 4 commits into
OpenIndiana:oifrom
pyhalov:utf-8

pyhalov commented Mar 7, 2020

Uh oh!

AndWac left a comment

Uh oh!

pyhalov commented Mar 8, 2020

Uh oh!

pyhalov commented Mar 8, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 11, 2020

Uh oh!

Toasterson commented Nov 3, 2021

Uh oh!

pyhalov commented Nov 3, 2021

Uh oh!

Toasterson commented Nov 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

pyhalov commented Mar 7, 2020

Uh oh!

AndWac left a comment

Choose a reason for hiding this comment

Uh oh!

pyhalov commented Mar 8, 2020

Uh oh!

pyhalov commented Mar 8, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 10, 2020

Uh oh!

pyhalov commented Mar 11, 2020

Uh oh!

Toasterson commented Nov 3, 2021

Uh oh!

pyhalov commented Nov 3, 2021

Uh oh!

Toasterson commented Nov 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants