ModuleNotFoundError: No module named 'pkg_resources' after series upgrade to jammy

[canonikamal]

Hello

After performing series upgrade from focal --> jammy on mysql-innodb-cluster charm, facing ModuleNotFoundError error for 'pkg_resources' as shown below:

2023-08-16 08:51:33 INFO juju.worker.uniter.operation runhook.go:159 ran "update-status" hook (via explicit, bespoke hook script)
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 Traceback (most recent call last):
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/hooks/update-status", line 7, in <module>
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 from charms.layer import basic # noqa
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charms/layer/basic.py", line 6, in <module>
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 from pkg_resources import Requirement
2023-08-16 08:56:20 WARNING unit.mysql-innodb-cluster/1.update-status logger.go:60 ModuleNotFoundError: No module named 'pkg_resources'
2023-08-16 08:56:20 ERROR juju.worker.uniter.operation runhook.go:153 hook "update-status" (via explicit, bespoke hook script) failed: exit status 1

series upgrade broke due to this issue and it also affected the other charm units on the same machine like logrotated.
Seems to be an issue of broken package/module requirements in the venv.

Please prioritize as this breaks the series upgrades which are part of larger plan for openstack upgrades in customer environments.

Thanks & regards

[ajkavanagh]

pkg_resources is part of setuptools, so this probably means that setuptools is 'broken' as part of the upgrade. It would be helpful if you could past the entire log-file (in debug mode) so that we can see where in the upgrade it is failing.

[canonikamal]

unit-mysql-innodb-cluster-1.log.gz
log file collected after enabling DEBUG logging.

[gustavosr98]

I am hitting something similar after a machine upgrade where charm is expecting to have pkg_resources
But in my case I do not seem to see any python package under dist-packages

root@k8s-worker-01:~# find /usr/local/lib/ -ls
 8650869 4 drwxr-xr-x 5 root root 4096 Sep 7 15:05 /usr/local/lib/
 10487400 4 drwxr-xr-x 3 root root 4096 Jun 5 2022 /usr/local/lib/nagios
 10487401 4 drwxr-xr-x 2 root root 4096 Jun 5 2022 /usr/local/lib/nagios/plugins
 10487404 4 -rwxr-xr-x 1 root root 1997 Jun 5 2022 /usr/local/lib/nagios/plugins/check_status_file.py
 10487402 16 -rwxr-xr-x 1 root root 15111 Jun 5 2022 /usr/local/lib/nagios/plugins/check_mem.pl
 10487409 8 -rwxr-xr-x 1 root root 5212 Jun 5 2022 /usr/local/lib/nagios/plugins/check_systemd_scopes.py
 10487414 4 -rwxr-xr-x 1 root root 4025 Jun 5 2022 /usr/local/lib/nagios/plugins/check_netlinks.py
 10487415 4 -rwxr-xr-x 1 root root 1338 Jun 5 2022 /usr/local/lib/nagios/plugins/check_xfs_errors.py
 10487419 4 -rwxr-xr-x 1 root root 2746 Jun 5 2022 /usr/local/lib/nagios/plugins/check_reboot.py
 10487416 8 -rwxr-xr-x 1 root root 5591 Jun 5 2022 /usr/local/lib/nagios/plugins/check_cis_audit.py
 10487403 4 -rwxr-xr-x 1 root root 1467 Jun 5 2022 /usr/local/lib/nagios/plugins/check_systemd.py
 10487420 4 -rwxr-xr-x 1 root root 1431 Jun 5 2022 /usr/local/lib/nagios/plugins/check_cpu_governor.py
 10487405 4 -rw-r--r-- 1 root root 2275 Jun 5 2022 /usr/local/lib/nagios/plugins/nagios_plugin3.py
 10487406 4 -rwxr-xr-x 1 root root 2651 Jun 5 2022 /usr/local/lib/nagios/plugins/check_upstart_job
 10487408 4 -rwxr-xr-x 1 root root 1135 Jun 5 2022 /usr/local/lib/nagios/plugins/check_netns.sh
 10487412 4 -rwxr-xr-x 1 root root 2169 Jun 5 2022 /usr/local/lib/nagios/plugins/check_swap_activity
 10487421 4 -rwxr-xr-x 1 root root 2694 Jun 3 2022 /usr/local/lib/nagios/plugins/check_ntpmon.py
 10487413 4 -rwxr-xr-x 1 root root 1974 Jun 5 2022 /usr/local/lib/nagios/plugins/check_conntrack.sh
 10487410 8 -rwxr-xr-x 1 root root 4783 Jun 5 2022 /usr/local/lib/nagios/plugins/check_lacp_bond.py
 10487411 8 -rwxr-xr-x 1 root root 5721 Jun 5 2022 /usr/local/lib/nagios/plugins/check_exit_status.pl
 10487418 4 -rwxr-xr-x 1 root root 2465 Jun 5 2022 /usr/local/lib/nagios/plugins/check_arp_cache.py
 10487417 4 -rwxr-xr-x 1 root root 2400 Jun 5 2022 /usr/local/lib/nagios/plugins/check_ro_filesystem.py
 10487407 8 -rwxr-xr-x 1 root root 4271 Jun 5 2022 /usr/local/lib/nagios/plugins/cron_cis_audit.py
 8651356 4 drwxr-xr-x 3 root root 4096 Sep 7 14:04 /usr/local/lib/python3.10
 8679962 4 drwxr-xr-x 2 root root 4096 Sep 7 14:04 /usr/local/lib/python3.10/dist-packages
 8916289 4 drwxr-xr-x 4 root root 4096 Jun 3 2022 /usr/local/lib/juju-charm-advanced-routing
 9181355 4 drwxr-xr-x 2 root root 4096 Jun 20 18:52 /usr/local/lib/juju-charm-advanced-routing/cleanup
 9177078 4 -rwxr-xr-x 1 root root 364 Jun 20 18:52 /usr/local/lib/juju-charm-advanced-routing/cleanup/95-juju_routing
 9057386 4 drwxr-xr-x 2 root root 4096 Jun 20 18:52 /usr/local/lib/juju-charm-advanced-routing/if-up
 9046023 4 -rwxr-xr-x 1 root root 334 Jun 20 18:52 /usr/local/lib/juju-charm-advanced-routing/if-up/95-juju_routing
root@k8s-worker-01:~# find /usr/local/lib/python3.10/dist-packages/
/usr/local/lib/python3.10/dist-packages/
root@k8s-worker-01:~# find /usr/local/lib/python3.10 -ls
 8651356 4 drwxr-xr-x 3 root root 4096 Sep 7 14:04 /usr/local/lib/python3.10
 8679962 4 drwxr-xr-x 2 root root 4096 Sep 7 14:04 /usr/local/lib/python3.10/dist-packages

[ajkavanagh]

@gustavosr98 most reactive charms use a virtualenv in a directory /var/lib/juju/agents/unit-<charm>-<rev>/.venv; thus any packages would be in there that were installed when the venv was created and then from the ./wheelhouse in the built charm. The venv creation is controlled in the layer.yaml in the options.basic.use_venv flag (setting it to True activates it).

The issue, I suspect, is that focal is py38 and jammy is py310 along with variations in what setuptools is installed in the venv. Now, during series upgrade the venv is supposed to be recreated so that this doesn't happen, and the charm code re-execed in the context of the new virtualenv/python version.

[ajkavanagh]

@gustavosr98 @canonikamal please could you indicate the versions of the charms that you've used (e.g. 8.0/stable + rev number for mysql, and same for yours @gustavosr98). What layer basic 'does' when building the charm affects what the versions of various modules get included depending on the base that the charm was built for.

[canonikamal]

@ajkavanagh

Charm version:
8.0/stable Revision: 56

Output for reference:
mysql-innodb-cluster 8.0.34 error 3 mysql-innodb-cluster 8.0/stable 56 no hook failed: "update-status" mysql-innodb-cluster/0 maintenance idle 3/lxd/13 10.11.2.128 Paused. Use 'resume' action to resume normal service. mysql-innodb-cluster/1* error idle 4/lxd/13 10.11.2.96 hook failed: "update-status" mysql-innodb-cluster/2 maintenance idle 5/lxd/14 10.11.2.55 Paused. Use 'resume' action to resume normal service.

[canonikamal]

Request for prioritization as we'll be planning some production upgrades in a couple of weeks.
Is it possible to workaround this in order to unblock the upcoming upgrades ?