Pin Jinja2 and MarkupSafe to fix issues building on Focal

[hloeung]

See #210

[hloeung]

Built charm on Jammy - https://paste.ubuntu.com/p/tP9X63bdWX/
Built charm on Focal - https://paste.ubuntu.com/p/GByT7j5Fcn/
Built charm on Bionic - https://paste.ubuntu.com/p/cs9J5HdSrp/
Built charm on Xenial - https://paste.ubuntu.com/p/SrGPv8tRjH/

[hloeung]

@ajkavanagh , free to take another look at this PR?

[ajkavanagh]

Thanks again for your work on the patch. I am a bit concerned with the re-pinning of various modules as this will change the builds on existing stable charms (should they be rebuilt), which would be a significant change in behaviour of the charm. e.g. the unpins of setuptools and setuptool-scm seems particularly problematic.

I wonder if, tactically, you just override these in your charm's wheelhouse.txt?

[ajkavanagh]

@hloeung I think this PR may be closed? I had a chat with @fnordahl and I think the issues you were seeing were to to with a juju bug?

[hloeung]

@hloeung I think this PR may be closed? I had a chat with @fnordahl and I think the issues you were seeing were to to with a juju bug?

@ajkavanagh different problem actually. The one @fnordahl & @darkalia saw and ran into was the Juju bug where it was selecting the charm with the highest revno and not one that was built for the specific series that they wanted.

[fnordahl]

Just to be clear, I've seen no issues fixed by this proposed change, and I repeat my stance that fine grained pinning does not belong in layer-basic. The wide range of different approaches to reactive charming within the teams we know about (not to mention those we don't know about) makes this impractical.

If you have specific pinning needs, doing so in individual charms or a different layer shared among a fleet of charms would be the preferred approach.

[hloeung]

The first issue is that charms fail to build on Focal. Here's quick steps I just came up with to reproduce:

From a clean Focal VM or LXD container:

sudo snap install charmcraft --classic
sudo snap install charm --classic
sudo apt-get install python3-venv
charm create haw-test # using charm here instead of charmcraft init to create a bare reactive charm instead of operator.
cd haw-test

Now edit metadata.yaml removing these lines:

provides:
provides-relation:
interface: interface-name
requires:
requires-relation:
interface: interface-name
peers:
peer-relation:
interface: interface-name

Then create a charmcraft.yaml file with:

type: charm
bases:
- build-on:
- name: ubuntu
channel: "20.04"
run-on:
- name: ubuntu
channel: "20.04"
parts:
charm:
source: "."
plugin: "reactive"
build-snaps: [charm]

Now try build it with:

charmcraft pack --destructive-mode

Unfortunately, it would fail with ModuleNotFoundError: No module named 'markupsafe'. Snippet of output being:

2022-11-01 00:41:36.959 :: build: Processing layer: layer:basic
2022-11-01 00:41:37.097 :: build: Processing layer: haw-test (from .)
2022-11-01 00:41:53.874 :: build: Ignoring pip: markers 'python_version < "3.8"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring Jinja2: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring Jinja2: markers 'python_version == "2.7" or python_version == "3.5"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring PyYAML: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring MarkupSafe: markers 'python_version < "3.6"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring MarkupSafe: markers 'python_version == "3.6"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring setuptools: markers 'python_version < "3.8"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring setuptools-scm: markers 'python_version < "3.8"' don't match your environment
2022-11-01 00:41:53.874 :: Ignoring wheel: markers 'python_version < "3.8"' don't match your environment
2022-11-01 00:41:53.874 :: Collecting pip<22.1
2022-11-01 00:41:53.874 :: Downloading pip-22.0.4.tar.gz (2.1 MB)
2022-11-01 00:41:53.874 :: Installing build dependencies: started
2022-11-01 00:41:53.874 :: Installing build dependencies: finished with status 'done'
2022-11-01 00:41:53.874 :: Getting requirements to build wheel: started
2022-11-01 00:41:53.874 :: Getting requirements to build wheel: finished with status 'done'
2022-11-01 00:41:53.874 :: Preparing wheel metadata: started
2022-11-01 00:41:53.874 :: Preparing wheel metadata: finished with status 'done'
2022-11-01 00:41:53.874 :: Saved /home/ubuntu/snap/charmcraft/common/tmpkzk9z42m/pip-22.0.4.tar.gz
2022-11-01 00:41:53.874 :: Downloading Jinja2-3.1.2.tar.gz (268 kB)
...
2022-11-01 00:41:53.875 :: File "/home/ubuntu/snap/charmcraft/common/pip-download-4j6lk0r8/Jinja2/src/jinja2/environment.py", line 14, in <module>
2022-11-01 00:41:53.875 :: from markupsafe import Markup
2022-11-01 00:41:53.875 :: ModuleNotFoundError: No module named 'markupsafe'

So we'll want to pin Jinja2, create the wheelhouse.txt file with:

Jinja2==2.11;python_version >= '3.6' and python_version <= '3.8' # py38, focal
Jinja2;python_version > '3.8'

Re-try build. That should build successfully with the bundled wheelhouse Python packages:

unzip -l haw-test_ubuntu-20.04-amd64.charm | grep wheelhouse | sort -k 4
1682 2022-11-01 00:45 wheelhouse.txt
257890 2022-11-01 00:46 wheelhouse/Jinja2-2.11.0.tar.gz
18668 2022-11-01 00:45 wheelhouse/MarkupSafe-2.1.1.tar.gz
269377 2022-11-01 00:45 wheelhouse/PyYAML-5.3.1.tar.gz
610181 2022-11-01 00:46 wheelhouse/charmhelpers-1.2.1.tar.gz
356307 2022-11-01 00:46 wheelhouse/charms.reactive-1.5.1.tar.gz
40330 2022-11-01 00:46 wheelhouse/flit_core-3.7.1.tar.gz
1622835 2022-11-01 00:46 wheelhouse/netaddr-0.7.19.tar.gz
84848 2022-11-01 00:46 wheelhouse/packaging-21.3.tar.gz
127505 2022-11-01 00:46 wheelhouse/pbr-5.11.0.tar.gz
2090742 2022-11-01 00:45 wheelhouse/pip-22.0.4.tar.gz
20466 2022-11-01 00:46 wheelhouse/pyaml-21.10.1.tar.gz
1999906 2022-11-01 00:46 wheelhouse/pyparsing-3.0.9.tar.gz
2512885 2022-11-01 00:45 wheelhouse/setuptools-62.1.0.tar.gz
61305 2022-11-01 00:46 wheelhouse/setuptools_scm-6.4.2.tar.gz
15164 2022-11-01 00:46 wheelhouse/tomli-2.0.1.tar.gz
66376 2022-11-01 00:46 wheelhouse/wheel-0.37.1.tar.gz

The other issue is that some of our charms use include_system_packages: true. This wasn't a problem until fb767dc. Here's quick steps to reproduce:

Edit layer.yaml adding:

options:
basic:
include_system_packages: true

Build charm:

charmcraft pack --destructive-mode

That should build successfully with the bundled wheelhouse Python packages:

$ unzip -l haw-test_ubuntu-20.04-amd64.charm | grep wheelhouse | sort -k 4
1682 2022-11-01 01:29 wheelhouse.txt
257890 2022-11-01 01:30 wheelhouse/Jinja2-2.11.0.tar.gz
18668 2022-11-01 01:29 wheelhouse/MarkupSafe-2.1.1.tar.gz
269377 2022-11-01 01:29 wheelhouse/PyYAML-5.3.1.tar.gz
610181 2022-11-01 01:30 wheelhouse/charmhelpers-1.2.1.tar.gz
356307 2022-11-01 01:30 wheelhouse/charms.reactive-1.5.1.tar.gz
40330 2022-11-01 01:30 wheelhouse/flit_core-3.7.1.tar.gz
1622835 2022-11-01 01:30 wheelhouse/netaddr-0.7.19.tar.gz
84848 2022-11-01 01:30 wheelhouse/packaging-21.3.tar.gz
127505 2022-11-01 01:30 wheelhouse/pbr-5.11.0.tar.gz
2090742 2022-11-01 01:29 wheelhouse/pip-22.0.4.tar.gz
20466 2022-11-01 01:30 wheelhouse/pyaml-21.10.1.tar.gz
1999906 2022-11-01 01:30 wheelhouse/pyparsing-3.0.9.tar.gz
2512885 2022-11-01 01:29 wheelhouse/setuptools-62.1.0.tar.gz
61305 2022-11-01 01:29 wheelhouse/setuptools_scm-6.4.2.tar.gz
15164 2022-11-01 01:30 wheelhouse/tomli-2.0.1.tar.gz
66376 2022-11-01 01:30 wheelhouse/wheel-0.37.1.tar.gz

Now using that newly built charm, try to deploy to Focal:

juju deploy ./haw-test_ubuntu-20.04-amd64.charm --series focal haw-test-focal-dd

It fails with:

2022-11-01 01:43:07 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Setting up virtualenv (20.0.17-1ubuntu0.4) ...
2022-11-01 01:43:08 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Processing triggers for man-db (2.9.1-1) ...
2022-11-01 01:43:11 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 created virtual environment CPython3.8.10.final.0-64 in 451ms
2022-11-01 01:43:11 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 creator CPython3Posix(dest=/var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv, clear=False, global=True)
2022-11-01 01:43:11 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
2022-11-01 01:43:11 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
2022-11-01 01:43:14 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Looking in links: wheelhouse
2022-11-01 01:43:14 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Processing ./wheelhouse/pip-22.0.4.tar.gz
2022-11-01 01:43:15 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Installing build dependencies: started
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 ERROR: Command errored out with exit status 1:
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 command: /var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/bin/python /var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-odg07h6m/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools wheel
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 cwd: None
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Complete output (40 lines):
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Looking in links: wheelhouse
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Processing ./wheelhouse/setuptools-62.1.0.tar.gz
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Getting requirements to build wheel: started
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Getting requirements to build wheel: finished with status 'error'
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 ERROR: Command errored out with exit status 1:
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 command: /var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/bin/python /tmp/tmpq0di12ca get_requires_for_build_wheel /tmp/tmp39w94f_4
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 cwd: /tmp/pip-install-c8kb4gvs/setuptools
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Complete output (30 lines):
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Traceback (most recent call last):
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/tmpq0di12ca", line 280, in <module>
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 main()
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/tmpq0di12ca", line 263, in main
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 json_out['return_val'] = hook(**hook_input['kwargs'])
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/tmpq0di12ca", line 114, in get_requires_for_build_wheel
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 return hook(config_settings)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 return self._get_build_requires(
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/build_meta.py", line 159, in _get_build_requires
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 self.run_setup()
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/build_meta.py", line 174, in run_setup
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 exec(compile(code, __file__, 'exec'), locals())
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "setup.py", line 87, in <module>
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 dist = setuptools.setup(**setup_params)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/__init__.py", line 87, in setup
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 return distutils.core.setup(**attrs)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/_distutils/core.py", line 109, in setup
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 _setup_distribution = dist = klass(attrs)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/dist.py", line 462, in __init__
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 _Distribution.__init__(
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/_distutils/dist.py", line 293, in __init__
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 self.finalize_options()
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/dist.py", line 885, in finalize_options
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 for ep in sorted(loaded, key=by_order):
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/dist.py", line 884, in <lambda>
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 loaded = map(lambda e: e.load(), filtered)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/tmp/pip-install-c8kb4gvs/setuptools/setuptools/_vendor/importlib_metadata/__init__.py", line 196, in load
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 return functools.reduce(getattr, attrs, module)
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'
2022-11-01 01:43:18 WARNING unit.haw-test-focal-dd/0.install logger.go:60 ----------------------------------------
2022-11-01 01:43:31 DEBUG unit.haw-test-focal-dd/0.install logger.go:60 Installing build dependencies: finished with status 'error'
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 ERROR: Command errored out with exit status 1: /var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/bin/python /var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ep8vjrs7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links wheelhouse -- setuptools wheel Check the logs for full command output.
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 Traceback (most recent call last):
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/var/lib/juju/agents/unit-haw-test-focal-dd-0/charm/hooks/install", line 8, in <module>
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 basic.bootstrap_charm_deps()
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/var/lib/juju/agents/unit-haw-test-focal-dd-0/charm/lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 _update_if_newer(pip, pre_install_pkgs)
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/var/lib/juju/agents/unit-haw-test-focal-dd-0/charm/lib/charms/layer/basic.py", line 366, in _update_if_newer
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 check_call([pip, 'install', '-U', '--no-index', '-f', 'wheelhouse',
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 raise CalledProcessError(retcode, cmd)
2022-11-01 01:43:31 WARNING unit.haw-test-focal-dd/0.install logger.go:60 subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-haw-test-focal-dd-0/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
2022-11-01 01:43:31 ERROR juju.worker.uniter.operation runhook.go:140 hook "install" (via explicit, bespoke hook script) failed: exit status 1

[hloeung]

@fnordahl I think you're right, I think going forward, for charms using include_system_packages: true we'll have to pin setuptools & setuptools-scm there within the charm. But can we land the Jinja2 pinning so we can at least build the charms using Focal (Launchpad per-series builds)?

[hloeung]

@ajkavanagh, @fnordahl, okay PR updated to just resolve the build issues on/with Focal.

[ajkavanagh]

Okay, I'm generally happy with this as it's fairly minimal and does allow a minimal charm to have sensible defaults for Jinja2/markupsafe on Focal. As @fnordahl has some key thoughts on this, I'd also like his +1 on this. Thanks again for your work on this.

[fnordahl]

Happy to see a more focused proposal.

Not going to block this change, but here are my further comments:

1. The root of the Jinja2 build issue on focal is a problem with the setuptools version/package as shipped with Focal (ref No control over virtualenv used to build wheels juju/charm-tools#646). So the proper fix for this issue is to extend charm-tools to allow controlling the virtualenv used for downloading/building wheels, or perhaps just upgrading the Python packages in the virtualenv (ref --upgrade-deps in https://docs.python.org/3/library/venv.html).
2. I'm not thrilled by the use of == pins here as previously stated strict pinning belongs in individual charm build.lock or other methods. I.e. what do you do if a upstream Jinja 2.x security fix is released?

[ajkavanagh]

2. I'm not thrilled by the use of == pins here as previously stated strict pinning belongs in individual charm build.lock or other methods. I.e. what do you do if a upstream Jinja 2.x security fix is released?

I think this is a valid critique and should be addressed. @hloeung could you update the PR to use "<" pins which would allow security fixes, please?

With charms.reactive, charm-tools, layer-basic and the rest of the charms.reactive charm building system being in maintenance mode effectively, we don't want to do invasive or breaking changes in these tools. This means being very conservative about changes and ensuring that existing work flows continue to work (as best we can) but recognising that there is a better way of building charms now.

[hloeung]

Updated the PR to use "<" pins as advised.

[hloeung]

Trying to get this moving along... @ajkavanagh ?

[fnordahl]

@hloeungjuju/charm-tools#650 has merged now, try using charm snap from edge channel and pass the --upgrade-buildvenv-core-deps argument, i.e.:

type: charmparts:
charm:
source: src/plugin: reactivereactive-charm-build-arguments:
- --upgrade-buildvenv-core-depsbuild-snaps:
- charm/3.x/edge

The new version will of course be promoted to stable, but most likely not until next year.

[ajkavanagh]

I'm approving this as Jinja is pinned by version and MarkupSafe is a dependency of that; either we unpin everything or we continue to hold things per python version as this review is doing.