Fix missing dependecy definition of 'packaging' by s-weigand · Pull Request #6207 · pydata/xarray

s-weigand · 2022-01-29T08:19:09Z

Hi there, we just wanted to upgrade xarray to version 0.21.0 and our ASV workflow on the CI started failing.

File"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/dask_array_compat.py", line4, in<module>frompackaging.versionimportVersionModuleNotFoundError: Nomodulenamed'packaging'

The reason is that packaging isn't a direct dependency of xarray.

$ johnnydep xarray2022-01-29 08:28:38 [info ] init johnnydist [johnnydep.lib] dist=xarray parent=None2022-01-29 08:28:40 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18 parent=xarray2022-01-29 08:28:44 [info ] init johnnydist [johnnydep.lib] dist=pandas>=1.1 parent=xarray2022-01-29 08:28:47 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=1.12022-01-29 08:28:51 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=1.12022-01-29 08:28:53 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=1.12022-01-29 08:28:56 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1name summary------------------------------ -----------------------------------------------------------------------xarray N-D labeled arrays and datasets in Python├── numpy>=1.18 NumPy is the fundamental package for array computing with Python.└── pandas>=1.1 Powerful data structures for data analysis, time series, and statistics ├── numpy>=1.18.5 NumPy is the fundamental package for array computing with Python. ├── python-dateutil>=2.8.1 Extensions to the standard Python datetime module │ └── six>=1.5 Python 2 and 3 compatibility utilities └── pytz>=2020.1 World timezone definitions, modern and historical

The problem why your tests didn't catch this is that packaging is a dependency of pytest itself.

$ johnnydep pytest2022-01-29 08:25:03 [info ] init johnnydist [johnnydep.lib] dist=pytest parent=None2022-01-29 08:25:07 [info ] init johnnydist [johnnydep.lib] dist=atomicwrites>=1.0 parent=pytest2022-01-29 08:25:09 [info ] init johnnydist [johnnydep.lib] dist=attrs>=19.2.0 parent=pytest2022-01-29 08:25:11 [info ] init johnnydist [johnnydep.lib] dist=colorama parent=pytest2022-01-29 08:25:14 [info ] init johnnydist [johnnydep.lib] dist=iniconfig parent=pytest2022-01-29 08:25:16 [info ] init johnnydist [johnnydep.lib] dist=packaging parent=pytest2022-01-29 08:25:18 [info ] init johnnydist [johnnydep.lib] dist=pluggy<2.0,>=0.12 parent=pytest2022-01-29 08:25:20 [info ] init johnnydist [johnnydep.lib] dist=py>=1.8.2 parent=pytest2022-01-29 08:25:23 [info ] init johnnydist [johnnydep.lib] dist=toml parent=pytest2022-01-29 08:25:25 [info ] init johnnydist [johnnydep.lib] dist=pyparsing!=3.0.5,>=2.0.2 parent=packagingname summary-------------------------------- ---------------------------------------------------------------------pytest pytest: simple powerful testing with Python├── atomicwrites>=1.0 Atomic file writes.├── attrs>=19.2.0 Classes Without Boilerplate├── colorama Cross-platform colored terminal text.├── iniconfig iniconfig: brain-dead simple config-ini parsing├── packaging Core utilities for Python packages│ └── pyparsing!=3.0.5,>=2.0.2 Python parsing module├── pluggy<2.0,>=0.12 plugin and hook calling mechanisms for python├── py>=1.8.2 library with cross-python path, ini-parsing, io, code, log facilities└── toml Python Library for Tom's Obvious, Minimal Language

We also only saw our ASV workflow fail since ASV only installs the package and all direct runtime dependencies in the venv it runs the benchmark with.

I prepared a dummy PR on my fork to demonstrate that this fixes the issue.

Since the release is only 10h old maybe just do a quick post-release so most users won't even notice.

User visible changes (including notable bug fixes) are documented in whats-new.rst

s-weigand · 2022-01-29T08:44:10Z

The failing tests are unrelated to this change and originate from this commit.

mathause · 2022-01-29T10:30:23Z

Thanks. I think it makes sense to revert #6200 then, it's not critical. I opened #6208 for this.

Thanks for the report and the PR.

mathause · 2022-01-29T10:30:38Z

cc @pydata/xarray

mathause · 2022-01-29T10:46:12Z

On second thought I think the problem is the release of dask 2022.01.1.

s-weigand · 2022-01-29T11:11:22Z

On second thought I think the problem is the release of dask 2022.01.1.

We don't have dask as a dependency

$ johnnydep pyglotaran2022-01-29 10:55:08 [info ] init johnnydist [johnnydep.lib] dist=pyglotaran parent=None2022-01-29 10:55:11 [info ] init johnnydist [johnnydep.lib] dist=asteval>=0.9.21 parent=pyglotaran2022-01-29 10:55:24 [info ] init johnnydist [johnnydep.lib] dist=click>=7.0 parent=pyglotaran2022-01-29 10:55:26 [info ] init johnnydist [johnnydep.lib] dist=netCDF4>=1.5.3 parent=pyglotaran2022-01-29 10:55:29 [info ] init johnnydist [johnnydep.lib] dist=numba>=0.52 parent=pyglotaran2022-01-29 10:55:32 [info ] init johnnydist [johnnydep.lib] dist=numpy<1.21,>=1.20.0 parent=pyglotaran2022-01-29 10:55:36 [info ] init johnnydist [johnnydep.lib] dist=pandas>=0.25.2 parent=pyglotaran2022-01-29 10:55:39 [info ] init johnnydist [johnnydep.lib] dist=rich>=10.9.0 parent=pyglotaran2022-01-29 10:55:42 [info ] init johnnydist [johnnydep.lib] dist=ruamel.yaml>=0.17.17 parent=pyglotaran2022-01-29 10:55:45 [info ] init johnnydist [johnnydep.lib] dist=scipy>=1.3.2 parent=pyglotaran2022-01-29 10:55:49 [info ] init johnnydist [johnnydep.lib] dist=sdtfile>=2020.8.3 parent=pyglotaran2022-01-29 10:55:51 [info ] init johnnydist [johnnydep.lib] dist=setuptools>=41.2 parent=pyglotaran2022-01-29 10:55:54 [info ] init johnnydist [johnnydep.lib] dist=tabulate>=0.8.8 parent=pyglotaran2022-01-29 10:55:56 [info ] init johnnydist [johnnydep.lib] dist=xarray!=0.20.0,!=0.20.1,>=0.16.2 parent=pyglotaran2022-01-29 10:55:58 [info ] init johnnydist [johnnydep.lib] dist=colorama parent=click>=7.02022-01-29 10:56:00 [info ] init johnnydist [johnnydep.lib] dist=cftime parent=netCDF4>=1.5.32022-01-29 10:56:03 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.9 parent=netCDF4>=1.5.32022-01-29 10:56:07 [info ] init johnnydist [johnnydep.lib] dist=numpy>1.13.3 parent=cftime2022-01-29 10:56:11 [info ] init johnnydist [johnnydep.lib] dist=llvmlite<0.39,>=0.38.0rc1 parent=numba>=0.522022-01-29 10:56:15 [info ] init johnnydist [johnnydep.lib] dist=numpy<1.22,>=1.18 parent=numba>=0.522022-01-29 10:56:18 [info ] init johnnydist [johnnydep.lib] dist=setuptools parent=numba>=0.522022-01-29 10:56:21 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=0.25.22022-01-29 10:56:25 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=0.25.22022-01-29 10:56:27 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=0.25.22022-01-29 10:56:29 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.12022-01-29 10:56:31 [info ] init johnnydist [johnnydep.lib] dist=colorama<0.5.0,>=0.4.0 parent=rich>=10.9.02022-01-29 10:56:34 [info ] init johnnydist [johnnydep.lib] dist=commonmark<0.10.0,>=0.9.0 parent=rich>=10.9.02022-01-29 10:56:36 [info ] init johnnydist [johnnydep.lib] dist=pygments<3.0.0,>=2.6.0 parent=rich>=10.9.02022-01-29 10:56:38 [info ] init johnnydist [johnnydep.lib] dist=ruamel.yaml.clib>=0.2.6 parent=ruamel.yaml>=0.17.172022-01-29 10:56:40 [info ] init johnnydist [johnnydep.lib] dist=numpy<1.23.0,>=1.16.5 parent=scipy>=1.3.22022-01-29 10:56:44 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.15.1 parent=sdtfile>=2020.8.32022-01-29 10:56:48 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18 parent=xarray!=0.20.0,!=0.20.1,>=0.16.22022-01-29 10:56:51 [info ] init johnnydist [johnnydep.lib] dist=pandas>=1.1 parent=xarray!=0.20.0,!=0.20.1,>=0.16.22022-01-29 10:56:55 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=1.12022-01-29 10:56:55 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=1.12022-01-29 10:56:55 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=1.12022-01-29 10:56:55 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1name summary------------------------------------ ----------------------------------------------------------------------------------------------------------------------------pyglotaran The Glotaran fitting engine.├── asteval>=0.9.21 Safe, minimalistic evaluator of python expression using ast module├── click>=7.0 Composable command line interface toolkit│ └── colorama Cross-platform colored terminal text.├── netCDF4>=1.5.3 Provides an object-oriented python interface to the netCDF version 4 library.│ ├── cftime Time-handling functionality from netcdf4-python│ │ └── numpy>1.13.3 NumPy is the fundamental package for array computing with Python.│ └── numpy>=1.9 NumPy is the fundamental package for array computing with Python.├── numba>=0.52 compiling Python code using LLVM│ ├── llvmlite<0.39,>=0.38.0rc1 lightweight wrapper around basic LLVM functionality│ ├── numpy<1.22,>=1.18 NumPy is the fundamental package for array computing with Python.│ └── setuptools Easily download, build, install, upgrade, and uninstall Python packages├── numpy<1.21,>=1.20.0 NumPy is the fundamental package for array computing with Python.├── pandas>=0.25.2 Powerful data structures for data analysis, time series, and statistics│ ├── numpy>=1.18.5 NumPy is the fundamental package for array computing with Python.│ ├── python-dateutil>=2.8.1 Extensions to the standard Python datetime module│ │ └── six>=1.5 Python 2 and 3 compatibility utilities│ └── pytz>=2020.1 World timezone definitions, modern and historical├── rich>=10.9.0 Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal│ ├── colorama<0.5.0,>=0.4.0 Cross-platform colored terminal text.│ ├── commonmark<0.10.0,>=0.9.0 Python parser for the CommonMark Markdown spec│ └── pygments<3.0.0,>=2.6.0 Pygments is a syntax highlighting package written in Python.├── ruamel.yaml>=0.17.17 ruamel.yaml is a YAML parser/emitter that supports roundtrip preservation of comments, seq/map flow style, and map key order│ └── ruamel.yaml.clib>=0.2.6 C version of reader, parser and emitter for ruamel.yaml derived from libyaml├── scipy>=1.3.2 SciPy: Scientific Library for Python│ └── numpy<1.23.0,>=1.16.5 NumPy is the fundamental package for array computing with Python.├── sdtfile>=2020.8.3 Read Becker & Hickl SDT files│ └── numpy>=1.15.1 NumPy is the fundamental package for array computing with Python.├── setuptools>=41.2 Easily download, build, install, upgrade, and uninstall Python packages├── tabulate>=0.8.8 Pretty-print tabular data└── xarray!=0.20.0,!=0.20.1,>=0.16.2 N-D labeled arrays and datasets in Python ├── numpy>=1.18 NumPy is the fundamental package for array computing with Python. └── pandas>=1.1 Powerful data structures for data analysis, time series, and statistics ├── numpy>=1.18.5 NumPy is the fundamental package for array computing with Python. ├── python-dateutil>=2.8.1 Extensions to the standard Python datetime module │ └── six>=1.5 Python 2 and 3 compatibility utilities └── pytz>=2020.1 World timezone definitions, modern and historical

As far as I can see xarray/__init__.py triggers the following import chain :
xarray.testing->xarray.core.duck_array_ops->xarray.core.dask_array_compat which triggers the import of packaging

Traceback starting from our package

File"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/glotaran/io/prepare_dataset.py", line6, in<module>importxarrayasxrFile"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/__init__.py", line1, in<module>from . importtesting, tutorial, ufuncsFile"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/testing.py", line8, in<module>fromxarray.coreimportduck_array_ops, formatting, utilsFile"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/duck_array_ops.py", line24, in<module>from . importdask_array_compat, dask_array_ops, dtypes, npcompat, nputilsFile"/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/dask_array_compat.py", line4, in<module>frompackaging.versionimportVersionModuleNotFoundError: Nomodulenamed'packaging'

s-weigand · 2022-01-29T11:20:49Z

packaging is imported in 16 other python files e.g. xarray.core.indexing, I think there is no way around adding it as a dependency.
Hiding all the imports in closures will be quite error prawn especially since tests won't catch it since pytest bring packaging along with it.

Illviljan · 2022-01-29T11:47:29Z

https://docs.python.org/3/whatsnew/3.10.html#distutils-deprecated recommends importing packaging instead, so if we want to do version checking maybe we should just add it as a dependency?

mathause · 2022-01-29T11:47:42Z

Yes, I think I just expressed it wrong. What I wanted to say: zhe failure you see here is not caused by #6200 but by a new release of dask. And this is independent of your issue with packaging.

s-weigand · 2022-01-29T11:48:47Z

Yes, I think I just expressed it wrong. What I wanted to say: zhe failure you see here is not caused by #6200 but by a new release of dask. And this is independent of your issue with packaging.

Ahhh now I get it 😅

max-sixty · 2022-01-29T17:50:02Z

Thanks everyone! Are we ok to close this then?

keewis · 2022-01-29T21:55:50Z

no, I think we should still add packaging to the dependencies, even if it doesn't cause any failures: we unconditionally import it in at least xarray.core.pycompat. We need to add a reasonable minimum version, though.

s-weigand · 2022-01-29T22:47:26Z

Dask has packaging>=20 which was released over 2y ago guess this is kinda reasonable.
I also ran the tests with packaging=17.1 (first py_0 version compatible with py38 on conda) released about 3y ago and all passed (windows but all imports should have been hit). For older versions, conda couldn't resolve the version.

❤️ 17.1 👍 20

Illviljan · 2022-01-30T18:23:53Z

Following dask seems like a good idea! We probably should add packaging in our ci installs as well:
https://github.com/pydata/xarray/tree/main/ci/requirements
and the asv config:
https://github.com/pydata/xarray/blob/main/asv_bench/asv.conf.json

keewis · 2022-01-30T20:54:39Z

just for the record, we decided that packaging related tools (i.e. setuptools, not sure if that also applies to packaging) should be released just before the earliest supported python version, so we would have to decide between 19.2 and 20. However, since the release notes of packaging state that 20 is the first version to officially support py38 I'd say using packaging>=20 should be uncontroversial.

keewis · 2022-01-30T21:01:12Z

other than the environment files and the asv config we also need to update doc/getting-started/installing.rst: the Required dependencies section, and maybe also the Minimum dependency versions. For the latter, we need to figure out a rule, though. For setuptools we had "42 months, but no older than 40.4", where 40.4 was the oldest version of setuptools installable from conda-forge.

…ndency

for more information, see https://pre-commit.ci

jhamman · 2022-01-31T16:54:46Z

@pydata/xarray - we should try to get this merged and a new release up ASAP as bug reports are starting to pile up. I've pushed changes to the CI configs, documentation, and elsewhere. I've also pulled in the recent changes on main which should hopefully fix the failing CI in the first commit.

Also, thanks @s-weigand for opening this PR and raising the issue.

max-sixty · 2022-01-31T16:57:00Z

I can do the release later today if no one gets there first

…/xarray into fix-missing-dependency

keewis

a few more suggestions, and I think we should also update requirements.txt, but after that we should be ready to merge!

…ndency

for more information, see https://pre-commit.ci

jhamman · 2022-01-31T23:16:11Z

This is ready to merge. @max-sixty - whenever you are ready, go ahead and merge and make the release.

max-sixty · 2022-01-31T23:51:54Z

I'll merge now though do the release this eve

🩹 Added packaging to install_requires
1e4ae63

s-weigand changed the title ~~🩹[BREAKING] Fix missing dependecy definition of 'packaging'~~🩹[BREAKING BUG] Fix missing dependecy definition of 'packaging'Jan 29, 2022

mathause mentioned this pull request Jan 29, 2022
MNT: prepare h5netcdf backend for (coming) change in dimension handling #6200
Merged
1 task

mathause mentioned this pull request Jan 29, 2022
Revert "MNT: prepare h5netcdf backend for (coming) change in dimension handling" #6208
Closed

s-weigand mentioned this pull request Jan 29, 2022
🩹 Disallow xarray 0.21.0 in setup.cfg glotaran/pyglotaran#976
Merged
1 task

Merge branch 'main' into pr/6207
1e0901c

max-sixty reviewed Jan 30, 2022
View reviewed changes

Comment threadsetup.cfg Outdated

max-sixty changed the title ~~🩹[BREAKING BUG] Fix missing dependecy definition of 'packaging'~~Fix missing dependecy definition of 'packaging'Jan 30, 2022

This was referenced Jan 30, 2022
Import of xarray conda package fails for 0.21 conda-forge/xarray-feedstock#71
Closed
add packaging dep conda-forge/xarray-feedstock#72
Merged

H0R5E mentioned this pull request Jan 31, 2022
[Bug]: ModuleNotFoundError: No module named 'packaging' #6215
Closed

TomNicholas mentioned this pull request Jan 31, 2022
[Bug]: No module named 'packaging' #6216
Closed

Joseph Hamman added 3 commits January 31, 2022 08:49

add packaging dep to ci, install instructions, and whatsnew page
958638a

lint
26421a8

Merge branch 'main' of github.com:pydata/xarray into fix-missing-depe…
09c4222
…ndency

[pre-commit.ci] auto fixes from pre-commit.com hooks
d9e5861
for more information, see https://pre-commit.ci

mathause reviewed Jan 31, 2022
View reviewed changes

Comment threadci/install-upstream-wheels.sh

jhamman mentioned this pull request Jan 31, 2022
Added missing dependency packaging #6217
Closed
4 tasks

Joseph Hamman added 2 commits January 31, 2022 09:20

uninstall packaging before installing upstream
ac0cc94

Merge branch 'fix-missing-dependency' ofhttps://github.com/s-weigand…
31c4cd5
…/xarray into fix-missing-dependency

keewis reviewed Jan 31, 2022
View reviewed changes

Comment threaddoc/contributing.rst Outdated
Comment threaddoc/contributing.rst Outdated
Comment threaddoc/whats-new.rst Outdated

Joseph Hammanand others added 4 commits January 31, 2022 13:10

Merge branch 'main' of github.com:pydata/xarray into fix-missing-depe…
69567a6
…ndency

update requirements.txt
abbccb5

update whats new and rerun linter
4406f36

[pre-commit.ci] auto fixes from pre-commit.com hooks
1c64bd8
for more information, see https://pre-commit.ci

max-sixty merged commit 5973ef3 into pydata:mainJan 31, 2022

s-weigand deleted the fix-missing-dependency branch February 1, 2022 18:29

jhamman mentioned this pull request Feb 2, 2022
Replace distutils.version with packaging.version #6096
Merged
3 tasks

Uh oh!

Uh oh!

Conversation

s-weigand commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-weigand commented Jan 29, 2022

Uh oh!

mathause commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mathause commented Jan 29, 2022

Uh oh!

mathause commented Jan 29, 2022

Uh oh!

s-weigand commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-weigand commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Illviljan commented Jan 29, 2022

Uh oh!

mathause commented Jan 29, 2022

Uh oh!

s-weigand commented Jan 29, 2022

Uh oh!

max-sixty commented Jan 29, 2022

Uh oh!

keewis commented Jan 29, 2022

Uh oh!

s-weigand commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Illviljan commented Jan 30, 2022

Uh oh!

Uh oh!

keewis commented Jan 30, 2022

Uh oh!

keewis commented Jan 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jhamman commented Jan 31, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

max-sixty commented Jan 31, 2022

Uh oh!

Uh oh!

keewis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhamman commented Jan 31, 2022

Uh oh!

max-sixty commented Jan 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

s-weigand commented Jan 29, 2022•
edited
Loading

mathause commented Jan 29, 2022•
edited
Loading

s-weigand commented Jan 29, 2022•
edited
Loading

s-weigand commented Jan 29, 2022•
edited
Loading

s-weigand commented Jan 29, 2022•
edited
Loading

keewis commented Jan 30, 2022•
edited
Loading

jhamman commented Jan 31, 2022•
edited
Loading