importlib.metadata.version can return None

[DMRobertson]

BPO	47060
Nosy	@jaraco, @DMRobertson

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = None
created_at = <Date 2022-03-18.18:11:51.491>
labels = ['type-bug', 'library', '3.10']
title = 'importlib.metadata.version can return None'
updated_at = <Date 2022-03-20.19:58:49.043>
user = 'https://github.com/DMRobertson'

bugs.python.org fields:

activity = <Date 2022-03-20.19:58:49.043>
actor = 'jaraco'
assignee = 'jaraco'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-03-18.18:11:51.491>
creator = 'David Robertson'
dependencies = []
files = []
hgrepos = []
issue_num = 47060
keywords = []
message_count = 2.0
messages = ['415516', '415529']
nosy_count = 2.0
nosy_names = ['jaraco', 'David Robertson']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47060'
versions = ['Python 3.10']

[DMRobertson]

Originally written up at the typeshed repo: python/typeshed#7513. The conclusion was that this is a bug in the implementation rather than an incorrect annotation.

To my surprise, I discovered in matrix-org/synapse#12223 that it is possible for importlib.metadata.version(...) to return None. To reproduce this:

1. Create a new virtual environment. I'm using CPython 3.10.2 as my interpreter.
2. Within the venv, pip install bottle. (Any package will do; I choose bottle because it's small and doesn't have any dependencies).
3. Check importlib reports the version of bottle:

   >>> import importlib.metadata as m
   >>> m.version('bottle')
   '0.12.19'

4. Here's the dirty bit: remove the metadata files for that package but keep the metadata directory.
   • Use pip show bottle to find the site-packages location
   • From there, remove all files in the bottle-VERSION-.dist-info directory: `rm /path/to/site-packages/bottle-VERSION.dist-info/*'.
5. The version of bottle is now judged to be None:

   >>> import importlib.metadata as m
   >>> m.version("bottle") is None
   True

   pip show bottle now determines that bottle isn't installed:

   $ pip show bottle
   WARNING: Package(s) not found: bottle

As well as importlib.metadata.version, importlib.metadata.Distribution.version and importlib.metadata.Distribution.name return None in this situation.

I couldn't see any suggestion in the stdlib docs (https://docs.python.org/3.10/library/importlib.metadata.html#distribution-versions) that this was possible. (Aside: it'd be great if the docs mention that PackageNotFoundError is raised if a package is not installed.)

No-one in their right mind should do step 4 willingly, but I have seen it happen in the wild (matrix-org/synapse#12223). We suspected a botched backup or similar was to blame.

I'm not familiar with all the machinery of Python package management, but I think I'd expect there to be a PackageNotFoundError raised in this situation? (I can imagine a package that doesn't declare its version, where version() returning None might make sense; but that feels odd.) Is the behaviour as intended?

It looks like this might be related to python/importlib_metadata#371?

[jaraco]

Thanks for the report.

Yes, the issues are related, where .version and .name returning None are specific manifestations of the metadata not having that key and the behavior being ill-defined.

I haven't yet decided if metadata items being undefined should result in None or raise an Exception (maybe KeyError).

For the specific case of a missing Name or Version, however, the packaging spec says that these fields are required (https://packaging.python.org/en/latest/specifications/core-metadata/#core-metadata-specifications), so it may be reasonable for the behavior when the specification is not met that the resulting behavior would be undefined (i.e. importlib.metadata should be able to assume the specification). It's outside the scope of importlib.metadata to detect, report, and repair invalid metadata. I would welcome and even encourage a third-party package to take on the responsibility of validating all distributions in an environment and reporting on non-compliant aspects.

In that sense, the type declaration is correct. .name and .version should always return str or raise an exception.

This additional example leads me stronger toward the position that .metadata[missing] should raise a KeyError, which would also fix this issue.

I'd also argue that if the metadata file is missing altogether, that should perhaps be a different error. That is, missing metadata is different from null metadata. Right now, the two are indistinguishable from the interface.

I'd expect there to be a PackageNotFoundError raised in this situation

That doesn't sound quite right to me. If there's a .dist-info directory, that implies a package is present. e.g.:

~ $ mkdir foo.dist-info
~ $ py -c "import importlib.metadata as md; print(md.distribution('foo'))"
<importlib.metadata.PathDistribution object at 0x1032f8580>

I'm going to ponder this one some more and probably address the .metadata issue(s) first before making any pronouncements on the best approach here.

[rgommers]

I have a very similar case with importlib.metadata.version doing something unexpected. Probably the same root cause, but please tell me if it's better to open a separate issue instead.

I built an in-place version of numpy with python setup.py build_ext -i (note that that will not have a .dist-info dir at the root of the repo, distutils doesn't produce that), and then used PYTHONPATH to switch between the numpy installed into the conda env and this in-place build:

$ echo $PYTHONPATH
/home/rgommers/code/numpy

Then open IPython and:

>>> import numpy as np
>>> np.__version__
'1.24.0.dev0+291.g2c5f407cf6'
>>> import importlib.metadata
>>> importlib.metadata.version("numpy")
'1.22.3'

So rather than returning None or raising an exception, I'm getting the metadata for the wrong package.

I haven't yet decided if metadata items being undefined should result in None or raise an Exception (maybe KeyError).

Either way would be fine with me, as long as it's not silently returning incorrect metadata. There will be other cases like that other than numpy, and PYTHONPATH is pretty commonly used to switch around packages when developing. A .dist-info directory may be required to get correct metadata, but it being missing should not cause incorrect results - that's a clear bug.

[jaraco]

Hi Ralf. The issue you describe is different, so I'll move it to another bug.

[sswam]

FWIW, I noticed that I had two torch directories under /usr/local/lib/python3.10/dist-packages:

torch-1.13.1+cu116.dist-info (the installed version)
torch-1.12.1+cu116.dist-info (containing only an empty file REQUESTED)

importlib_metadata.version("torch") was retuning None in this scenario. I think it was looking at the directory which only had the REQUESTED file.

After I removed that torch-1.12.1+cu116.dist-info directory, it fixed the problem, i.e. now: importlib_metadata.version("torch") == '1.13.1+cu116'