Use _unstack_once for valid dask and sparse versions by Illviljan · Pull Request #5315 · pydata/xarray

Illviljan · 2021-05-15T11:42:57Z

Requires Add version variable for optional imports in pycompat #5314
Passes pre-commit run --all-files

pep8speaks · 2021-05-15T11:43:03Z

Hello @Illviljan! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-05-17 21:56:42 UTC

…stacking

Illviljan · 2021-05-15T20:50:19Z

pint doesn't seem to be playing nice still @max-sixty. Seems some of your issues from #4746 are still there? Eventhough some of the issues in #4751 seems to have been solved.

max-sixty · 2021-05-15T22:54:01Z

pint doesn't seem to be playing nice still @max-sixty. Seems some of your issues from #4746 are still there? Eventhough some of the issues in #4751 seems to have been solved.

@keewis any thoughts?

One (kinda bad) idea would be to branch on whether pint was installed — presumably if it's not installed then it can't cause the import issues. But changing dask performance based on whether pint is installed is not great!

keewis · 2021-05-16T00:43:49Z

yep, to really fix this we would have to change pint. I think it would be worth verifying whether we can check the interface (with hasattr) instead of type checking using isinstance, though.

max-sixty · 2021-05-16T03:25:34Z

yep, to really fix this we would have to change pint. I think it would be worth verifying whether we can check the interface (with hasattr) instead of type checking using isinstance, though.

Great, that could be a good workaround. What attr would we check for?

…stacking

Illviljan · 2021-05-16T16:10:45Z

@keewis any idea what it means when only 1 check has been run?

keewis · 2021-05-16T16:47:50Z

github actions currently has "degraded performance" (earlier today it was a "major outage")

max-sixty · 2021-05-17T21:30:23Z

Looks great! Let's merge on green. Feel free to add a whatsnew @Illviljan !

max-sixty · 2021-05-17T22:49:12Z

Great @Illviljan ! Feel free to add a whatsnew in a different PR — this is good news for lots of dask users

…5315)" This reverts commit 9165c26.

* Revert "Use _unstack_once for valid dask and sparse versions (#5315)" This reverts commit 9165c26. * 0.18.2 release notes

* Revert "Use _unstack_once for valid dask and sparse versions (#5315)" This reverts commit 9165c26. * 0.18.2 release notes * fix RTD [skip-ci] (#5518) * v0.19.0 release notes * Update doc/whats-new.rst [skip-ci] * remove empty sections Co-authored-by: Maximilian Roos <m@maxroos.com> Co-authored-by: keewis <keewis@users.noreply.github.com>

Illviljan added 5 commits May 15, 2021 12:36

optional import versions to pycompat
8497d91

Update indexing.py
2c11948

move dtypes to avoid circular import
aeea5cc

Update pycompat.py
8fa535c

faster unstacking
ec6c301

Illviljan added 2 commits May 15, 2021 13:43

Update dataset.py
3f2784d

Update dataset.py
de15c39

Illviljan marked this pull request as draft May 15, 2021 11:51

Illviljan added 2 commits May 15, 2021 15:17

have to check that the array type is in use
682922f

Update dataset.py
dba4693

Illviljan changed the title ~~Faster unstacking with dask index assignment~~Use _unstack_once for valid dask and sparse versionsMay 15, 2021

sparse arg requires the slow path?
bb74dc9

max-sixty reviewed May 15, 2021
View reviewed changes

Comment threadxarray/core/dataset.py Outdated

Illviljan added 6 commits May 15, 2021 18:19

cupy.__version__
8f15a28

Merge branch 'Illviljan-version_to_pycompat' into Illviljan-faster_un…
cd39b67
…stacking

test pint_array_type
c2ce0cf

Update dataset.py
9992824

Add pint check in pycompat
95fc10e

Update pycompat.py
2597619

revert pint test
cf80360

Illviljan added 3 commits May 16, 2021 06:36

import whole dtypes
b41c113

Merge branch 'Illviljan-version_to_pycompat' into Illviljan-faster_un…
df425a4
…stacking

Test turning off the ndarray check
a31754b

Illviljan closed this May 16, 2021

Illviljan reopened this May 16, 2021

keewis closed this May 16, 2021

keewis reopened this May 16, 2021

Illviljan closed this May 16, 2021

Illviljan reopened this May 16, 2021

Illviljan mentioned this pull request May 16, 2021
Convert new_shape from list to tuple in _unstack_once #5319
Merged
1 task

Illviljan added 2 commits May 16, 2021 23:31

sparse and pint doesn't work. Switch to a restrictive if
207609b

Update dataset.py
7bb5aa5

max-sixty reviewed May 16, 2021
View reviewed changes

Comment threadxarray/core/dataset.py

Illviljan added 3 commits May 17, 2021 23:02

Merge branch 'master' into Illviljan-faster_unstacking
0d08e35

Add back some comments and add some relevant issue links
d5b31ae

lint
35db4a8

Update dataset.py
7ee325d

Illviljan marked this pull request as ready for review May 17, 2021 21:36

Illviljan closed this May 17, 2021

Illviljan reopened this May 17, 2021

max-sixty merged commit 9165c26 into pydata:masterMay 17, 2021

Illviljan deleted the Illviljan-faster_unstacking branch May 18, 2021 18:13

aulemahal mentioned this pull request May 19, 2021
Fast-track unstack doesn't work with dask #5346
Closed

max-sixty added a commit to max-sixty/xarray that referenced this pull request May 19, 2021
Revert "Use _unstack_once for valid dask and sparse versions (pydata#…
5533a67
…5315)" This reverts commit 9165c26.

max-sixty added a commit that referenced this pull request May 19, 2021
Merge 0.18.2 hotfix into master (#5349)
f58e76c
* Revert "Use _unstack_once for valid dask and sparse versions (#5315)" This reverts commit 9165c26. * 0.18.2 release notes

Illviljan mentioned this pull request Jul 5, 2021
Faster unstacking to sparse #5577
Merged
3 tasks

Navigation Menu

Uh oh!

Uh oh!

Use _unstack_once for valid dask and sparse versions - #5315

Use _unstack_once for valid dask and sparse versions#5315
max-sixty merged 26 commits into
pydata:masterfrom
Illviljan:Illviljan-faster_unstacking

Illviljan commented May 15, 2021•
edited
Loading

Uh oh!

pep8speaks commented May 15, 2021•
edited
Loading

Uh oh!

Uh oh!

Illviljan commented May 15, 2021

Uh oh!

max-sixty commented May 15, 2021

Uh oh!

keewis commented May 16, 2021

Uh oh!

max-sixty commented May 16, 2021

Uh oh!

Illviljan commented May 16, 2021

Uh oh!

keewis commented May 16, 2021

Uh oh!

Uh oh!

max-sixty commented May 17, 2021

Uh oh!

max-sixty commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

Illviljan commented May 15, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pep8speaks commented May 15, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comment last updated at 2021-05-17 21:56:42 UTC

Uh oh!

Uh oh!

Illviljan commented May 15, 2021

Uh oh!

max-sixty commented May 15, 2021

Uh oh!

keewis commented May 16, 2021

Uh oh!

max-sixty commented May 16, 2021

Uh oh!

Illviljan commented May 16, 2021

Uh oh!

keewis commented May 16, 2021

Uh oh!

Uh oh!

max-sixty commented May 17, 2021

Uh oh!

max-sixty commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Illviljan commented May 15, 2021•
edited
Loading

pep8speaks commented May 15, 2021•
edited
Loading