OSM: Installing a proxy charm container fails, due to Python dependency issue

[dreibh]

I am not absolutely sure whether this bug belongs to "layer-basic", but it is strongly related to it:

I try to instantiate a NS with proxy charm in OSM Release EIGHT (basically: hackfest_proxycharm), and it fails instantiating:

12 May 2021 18:20:14Z workload maintenance installing charm software
12 May 2021 18:20:14Z juju-unit executing running install hook
12 May 2021 18:20:22Z juju-unit error hook failed: "install"

The not useful log output seems to be a known, but still not fixed, Juju bug reported here: https://bugs.launchpad.net/charm-ceph-osd/+bug/1883859 .

Further debugging shows that the Juju container instance tries to install some software, particularly Python PIP packages (this is probably by "layer-basic"?), which fails when doing this manually:

DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality. 
Looking in links: wheelhouse
Processing ./wheelhouse/paramiko-1.16.3.tar.gz
Processing ./wheelhouse/pbr-5.6.0.tar.gz
Processing ./wheelhouse/Tempita-0.5.2.tar.gz
Processing ./wheelhouse/PyYAML-5.2.tar.gz
Processing ./wheelhouse/pycrypto-2.6.1.tar.gz
Processing ./wheelhouse/charms.reactive-1.4.1.tar.gz
Processing ./wheelhouse/wheel-0.33.6.tar.gz
Processing ./wheelhouse/charmhelpers-0.20.21.tar.gz
Processing ./wheelhouse/ecdsa-0.16.1.tar.gz
Processing ./wheelhouse/six-1.16.0.tar.gz
Processing ./wheelhouse/Jinja2-2.10.1.tar.gz
Processing ./wheelhouse/MarkupSafe-2.0.0.tar.gz
 ERROR: Command errored out with exit status 1:
 command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8bcecu1o 
 cwd: /tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/ 
 Complete output (38 lines): 
 Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "/tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/setup.py", line 61, in <module> 
 run_setup(True) 
 File "/tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/setup.py", line 44, in run_setup 
 ext_modules=ext_modules if with_binary else [], 
 File "/usr/local/lib/python3.5/dist-packages/setuptools/__init__.py", line 145, in setup 
 return distutils.core.setup(**attrs)
 File "/usr/lib/python3.5/distutils/core.py", line 121, in setup
 dist.parse_config_files()
 File "/usr/local/lib/python3.5/dist-packages/setuptools/dist.py", line 701, in parse_config_files
 ignore_option_errors=ignore_option_errors)
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 121, in parse_configuration
 meta.parse()
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 426, in parse
 section_parser_method(section_options)
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 399, in parse_section
 self[name] = value
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 184, in __setitem__
 value = parser(value)
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 514, in _parse_version
 version = self._parse_attr(value, self.package_dir)
 File "/usr/local/lib/python3.5/dist-packages/setuptools/config.py", line 349, in _parse_attr
 module = import_module(module_name)
 File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
 return _bootstrap._gcd_import(name[level:], package, level)
 File "<frozen importlib._bootstrap>", line 986, in _gcd_import
 File "<frozen importlib._bootstrap>", line 969, in _find_and_load
 File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 661, in exec_module
 File "<frozen importlib._bootstrap_external>", line 767, in get_code
 File "<frozen importlib._bootstrap_external>", line 727, in source_to_code
 File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
 File "/tmp/pip-install-7v7mnyly/markupsafe_7e757b82a7424a1983d2fd842ac89a6d/src/markupsafe/__init__.py", line 111
 return f"{self.__class__.__name__}({super().__repr__()})"
 ^
 SyntaxError: invalid syntax
 ----------------------------------------
WARNING: Discarding file:///var/lib/juju/agents/unit-app-vnf-57c9c807d03f-vdu-mgmtvm-cnt-z0-0/charm/wheelhouse/MarkupSafe-2.0.0.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement MarkupSafe==2.0.0
ERROR: No matching distribution found for MarkupSafe==2.0.0

I am not sure whether this dependency "MarkupSafe" is caused by layer:options, layer:basic, layer:sshproxy or layer:vnfproxy. The issue, however, is that MarkupSafe-2.0.0 requires an up-to-date Python. Python 3.5, which is provided by the Ubuntu Xenial container, is deprecated since September 2020. It seems to not be possible to get OSM to use any other series than Ubuntu Xenial. The "series" in metadata.yaml seems to just be ignored. Furthermore, "charm build ..." just adds "trusty" and "xenial" to the series of the built charm (reported here: juju/charm-tools#608).

That is, the bug origin is that OSM uses a deprecated container installation, particularly using a deprecated version of Python. This may also be security-relevant.

[dreibh]

There is a simple work-around for the issue, for users who are affected by this bug and need a quick fix:
Add a file "wheelhouse.txt" in the directory of the charm to be used:

MarkupSafe==1.1.1

This ensures that an older version gets installed, which is still compatible with the deprecated Python 3.5.

This is of course not a solution to fix this bug.

[ajkavanagh]

Hi

Yup, MarkupSafe is required by Jinja2: https://github.com/pallets/jinja/blob/2.10.x/setup.py

Our best bet, as we pin Jinja2<=2.10.1 in the layer-basic wheelhouse.txt is to pin MarkupSafe as well.

[Jpfonseca]

I try to instantiate a NS with proxy charm in OSM Release EIGHT (basically: hackfest_proxycharm)

The issue was also present on release 9 and on the branch master/ release 10 due to a xenial dependency in the N2VC module... A fix was already merged into these branches.