Error on az connectedk8s connect: cannot import name '_psutil_linux' from partially initialized module 'psutil'

[derkoe]

Describe the bug

When running az connectedk8s connect I get an error on Linux (Fedora 34).

Command Name
az connectedk8s connect

Errors:

The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 556, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 517, in load_arguments
    self.command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 313, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/koc/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in <module>
    from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Install extension az extension add --name connectedk8s
• Register K8s Cluster az connectedk8s connect

Expected Behavior

The command works.

Environment Summary

Linux-5.12.6-300.fc34.x86_64-x86_64-with-glibc2.33
Python 3.9.5
Installer: RPM

azure-cli 2.24.0

Extensions:
k8sconfiguration 0.1.8
spring-cloud 1.2.0
aks-preview 0.4.64
connectedk8s 1.1.5

Additional Context

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc akashkeshari.

Issue Details

---

Describe the bug

When running az connectedk8s connect I get an error on Linux (Fedora 34).

Command Name
az connectedk8s connect

Errors:

The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 556, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 517, in load_arguments
    self.command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 313, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/koc/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in <module>
    from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py", line 99, in <module>
    from . import _pslinux as _psplatform
  File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Install extension az extension add --name connectedk8s
• Register K8s Cluster az connectedk8s connect

Expected Behavior

The command works.

Environment Summary

Linux-5.12.6-300.fc34.x86_64-x86_64-with-glibc2.33
Python 3.9.5
Installer: RPM

azure-cli 2.24.0

Extensions:
k8sconfiguration 0.1.8
spring-cloud 1.2.0
aks-preview 0.4.64
connectedk8s 1.1.5

Additional Context

Author:	derkoe
Assignees:	-
Labels:	Connected Kubernetes, Service Attention
Milestone:	-

[yonzhan]

route to service team

[redhatstuart]

Linux stkirk-fedora 5.12.6-300.fc34.x86_64 #1 SMP Sat May 22 20:42:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Fedora release 34 (Thirty Four)
azure-cli 2.24.0
core 2.24.0
telemetry 1.0.6

Extensions:
k8s-extension-private 0.2.1b2
connectedk8s 1.1.5

I have also experienced the exact same behavior as noted above. This issue does not seem to be present on debian/ubuntu type installs of the AZCLI. I have tried this with Fedora 33, Fedora 34 and RHEL 8 (all RPM based distributions) and the process fails each time:

[stkirk@stkirk-fedora aro4]$ az connectedk8s connect --name $arcName --resource-group $aroRG --location $arcLocation
The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 556, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/init.py", line 517, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/init.py", line 313, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python3.6/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/usr/lib64/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/usr/lib64/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 855, in exec_module
File "", line 228, in _call_with_frames_removed
File "/home/stkirk/.azure/cliextensions/connectedk8s/azext_connectedk8s/custom.py", line 22, in
from psutil import process_iter, NoSuchProcess, AccessDenied, ZombieProcess, net_connections
File "/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py", line 99, in
from . import _pslinux as _psplatform
File "/usr/lib64/az/lib/python3.6/site-packages/psutil/_pslinux.py", line 26, in
from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/init.py)

[derkoe]

I have found a workaround:

In /usr/bin/az replace the python command with pythond3.6 (and then re-install the connectedk8s extension):

3c3
< AZ_INSTALLER=RPM PYTHONPATH="$bin_dir"/../lib64/az/lib/python3.6/site-packages python3 -sm azure.cli "$@"
---
> AZ_INSTALLER=RPM PYTHONPATH="$bin_dir"/../lib64/az/lib/python3.6/site-packages python3.6 -sm azure.cli "$@"

It seems that the RPM packages uses Python 3.6 but the python3 command points to 3.9. This results in the wrong packages to be installed in the extension.

[redhatstuart]

Confirmed this works. I had a feeling that this was the issue given the higher rev of python. I tried using "alternatives" to get Fedora to set the default python to 3.6 however that didn't seem to work so was at a loss. cc: Yong Zhang (@yonzhan)

[jiasli]

Christian Köberl (@derkoe), you analysis helps a lot.

If you extract the rpm package azure-cli-2.24.0-1.el7.x86_64.rpm (downloadable from https://packages.microsoft.com/yumrepos/azure-cli/) and navigate to usr/lib64/az/lib/python3.6/site-packages/psutil, you will see only 2 so files:

• _psutil_linux.cpython-36m-x86_64-linux-gnu.so
• _psutil_posix.cpython-36m-x86_64-linux-gnu.so

This means psutil in Azure CLI only works for Python 3.6.

This is because Azure CLI is built using centos7:

azure-cli/scripts/release/rpm/Dockerfile.centos

Line 1 in f2855b8

ARG tag=centos7

Only python 3.6 is available in centos7.

I haven't tested Fedora 34. Could you verify if Python 3.6 exists on any Fedora 34 installation? If so we can fix it by using Python 3.6 for now:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 57 in 753a11c

printf "#!/usr/bin/env bash\nbin_dir=\`cd \"\$(dirname \"\$BASH_SOURCE[0]\")\"; pwd\`\nAZ_INSTALLER=RPM PYTHONPATH=\"\$bin_dir\"/../%{_lib}/az/lib/${python_version}/site-packages %{python_cmd} -sm azure.cli \"\$@\"" > %{buildroot}%{_bindir}/az

[derkoe]

Fedora 34 defaults to Python 3.9 but Python 3.6 is also available (as /usr/bin/python3.6).

[derkoe]

If you have a Docker (or another container runtime) you can do:

$ docker run -it --rm fedora:34

[root@d6bd91249557 /]# echo -e "[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc" |  tee /etc/yum.repos.d/azure-cli.repo
[root@d6bd91249557 /]# dnf install azure-cli
[root@d6bd91249557 /]# az connectedk8s connect
The command requires the extension connectedk8s. Do you want to install it now? The command will continue to run after the extension is installed. (Y/n): y
Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
The command failed with an unexpected error. Here is the traceback:
cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib64/az/lib/python3.6/site-packages/psutil/__init__.py)
...

[jiasli]

Christian Köberl (@derkoe), thanks for the info. We will join the the Kick-off event recently. We will look at it after we return.