Grouper, Resampler as public api by dcherian · Pull Request #8840 · pydata/xarray

dcherian · 2024-03-15T05:16:05Z

Expose UniqueGrouper, BinGrouper and TimeResampler as public API under xarray.groupers
Allow providing these objects to .resample and .groupby
This is a step toward grouping by multiple variables, grouping by a dask array, and unblocking "chunking to a frequency" (Support specifying chunk sizes using labels (e.g. frequency string) #7559)

We do not yet officially support custom Grouper or Resampler objects. I'd like to clear up the API a bit before exposing that.

Consider avoiding IndexVariable (groupby: remove some internal use of IndexVariable #9123)
ClosesUpdate GroupBy constructor for grouping by multiple variables, dask arrays #6610
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

dcherian · 2024-06-21T22:36:29Z

This is finally ready for review! Please take a look at the API and docs.

* main: promote floating-point numeric datetimes to 64-bit before decoding (pydata#9182) also pin `numpy` in the all-but-dask CI (pydata#9184) temporarily remove `pydap` from CI (pydata#9183) temporarily pin `numpy<2` (pydata#9181) Change np.core.defchararray to np.char (pydata#9165) (pydata#9166) Fix example code formatting for CachingFileManager (pydata#9178) Slightly improve DataTree repr (pydata#9064) switch to unit `"D"` (pydata#9170) Docs: Add page with figure for navigating help resources (pydata#9147) Add test for pydata#9155 (pydata#9161) Remove mypy exclusions for a couple more libraries (pydata#9160) Include numbagg in type checks (pydata#9159) Improve zarr chunks docs (pydata#9140)

Illviljan · 2024-07-11T11:31:35Z

These looks related, same problem as before I guess:

xarray/core/dataset.py: note: In member "groupby" of class "Dataset":
xarray/core/dataset.py:10321: error: Incompatible types in assignment (expression has type "Mapping[Hashable, Any]", variable has type "dict[str, Grouper]") [assignment]
xarray/core/dataarray.py: note: In member "groupby" of class "DataArray":
xarray/core/dataarray.py:6778: error: Incompatible types in assignment (expression has type "Mapping[Hashable, Any]", variable has type "dict[str, Grouper]") [assignment]

* main: exclude the bots from the release notes (pydata#9235) switch the documentation to run with `numpy>=2` (pydata#9177) `numpy` 2 compatibility in the iris code paths (pydata#9156) `numpy` 2 compatibility in the `netcdf4` and `h5netcdf` backends (pydata#9136) Fix time indexing regression in `convert_calendar` (pydata#9192) Use duckarray assertions in test_coding_times (pydata#9226) Use reshape and ravel from duck_array_ops in coding/times.py (pydata#9225) Cleanup test_coding_times.py (pydata#9223) Only use necessary dims when creating temporary dataarray (pydata#9206) Fix two bugs in DataTree.update() (pydata#9214) Use numpy 2.0-compat `np.complex64` dtype in test (pydata#9217)

dcherian · 2024-07-11T16:57:38Z

I ended up using type: ignore after struggling for a while...

* main: Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" Test main push Revert "Update _typing.py" Update _typing.py Add a `.drop_attrs` method (pydata#8258)

dcherian · 2024-07-17T04:24:22Z

I'll merge tomorrow if there are no comments.

One thing to think about is whether we want UniqueGrouper or CategoricalGrouper or CategoryGrouper.

headtr1ck

Looks good, only couple of minor details.
Feel free to ignore these as well :P

Co-authored-by: Michael Niklas <mick.niklas@gmail.com>

* main: add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234)

* main: (54 commits) Adding `open_datatree` backend-specific keyword arguments (#9199) [pre-commit.ci] pre-commit autoupdate (#9202) Restore ability to specify _FillValue as Python native integers (#9258) add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234) Allow mypy to run in vscode (#9239) Revert "Test main push" ...

…monotonic-variable * main: (995 commits) Adding `open_datatree` backend-specific keyword arguments (pydata#9199) [pre-commit.ci] pre-commit autoupdate (pydata#9202) Restore ability to specify _FillValue as Python native integers (pydata#9258) add backend intro and how-to diagram (pydata#9175) Fix copybutton for multi line examples in double digit ipython cells (pydata#9264) Update signature for _arrayfunction.__array__ (pydata#9237) Add encode_cf_datetime benchmark (pydata#9262) groupby, resample: Deprecate some positional args (pydata#9236) Delete ``base`` and ``loffset`` parameters to resample (pydata#9233) Update dropna docstring (pydata#9257) Grouper, Resampler as public api (pydata#8840) Fix mypy on main (pydata#9252) fix fallback isdtype method (pydata#9250) Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" ...

* main: (54 commits) Adding `open_datatree` backend-specific keyword arguments (pydata#9199) [pre-commit.ci] pre-commit autoupdate (pydata#9202) Restore ability to specify _FillValue as Python native integers (pydata#9258) add backend intro and how-to diagram (pydata#9175) Fix copybutton for multi line examples in double digit ipython cells (pydata#9264) Update signature for _arrayfunction.__array__ (pydata#9237) Add encode_cf_datetime benchmark (pydata#9262) groupby, resample: Deprecate some positional args (pydata#9236) Delete ``base`` and ``loffset`` parameters to resample (pydata#9233) Update dropna docstring (pydata#9257) Grouper, Resampler as public api (pydata#8840) Fix mypy on main (pydata#9252) fix fallback isdtype method (pydata#9250) Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" ...

dcherian mentioned this pull request Mar 15, 2024
Proof of concept - public Grouper objects #8509
Closed

Illviljan reviewed Mar 21, 2024
View reviewed changes

Comment threadxarray/core/dataarray.py Outdated
Comment threadxarray/tests/test_groupby.py Outdated

dcherian force-pushed the grouper-public-api branch from 3c6d2d5 to 8fc7f5fCompare April 17, 2024 03:44

dcherian commented Apr 17, 2024
View reviewed changes

Comment threadxarray/core/groupers.py

dcherian commented Apr 17, 2024
View reviewed changes

Comment threadxarray/core/types.py Outdated

dcherian mentioned this pull request Jun 13, 2024
Grouper refactor #9122
Merged

dcherian force-pushed the grouper-public-api branch from 04b433a to 235b553Compare June 13, 2024 22:55

dcherian mentioned this pull request Jun 13, 2024
groupby: remove some internal use of IndexVariable #9123
Merged

dcherian force-pushed the grouper-public-api branch from b8085f8 to 384818fCompare June 14, 2024 16:50

TomNicholas added the topic-groupby label Jun 14, 2024

dcherian force-pushed the grouper-public-api branch 3 times, most recently from 436997b to 7100907Compare June 21, 2024 22:08

dcherian added 6 commits June 21, 2024 16:35

Grouper, Resampler as public API
65d500c

Add test
9f5b9fd

Add docs
f338e45

Fix test
09fa705

fix types.
110517a

bugfix
01fbf50

dcherian force-pushed the grouper-public-api branch from 7100907 to 01fbf50Compare June 21, 2024 22:35

dcherian added the needs review label Jun 21, 2024

dcherian marked this pull request as ready for review June 21, 2024 22:35

dcherian commented Jun 21, 2024
View reviewed changes

Comment threadxarray/tests/test_groupby.py Outdated

dcherian added 3 commits June 28, 2024 10:47

Better binning API
e250895

docs.fixes
0dc1663

dcherian closed this Jun 30, 2024

dcherian reopened this Jun 30, 2024

Illviljan reviewed Jun 30, 2024
View reviewed changes

Comment threadxarray/core/groupers.py Outdated
Comment threadxarray/core/groupers.py Outdated
Comment threadxarray/core/groupers.py Outdated
Comment threadxarray/core/dataset.py Outdated
Comment threadxarray/core/groupers.py Outdated
Comment threadxarray/core/groupby.py

dcherian added 2 commits July 11, 2024 22:25

Try to fix typing
1142663

dcherian force-pushed the grouper-public-api branch from 12c4316 to c94d9c2Compare July 11, 2024 16:57

dcherian mentioned this pull request Jul 11, 2024
groupby, resample: Deprecate some positional args #9236
Merged

dcherian force-pushed the grouper-public-api branch from a38206d to c94d9c2Compare July 11, 2024 17:09

Nicer error
7838d57

dcherian added plan to merge Final call for comments and removed needs review labels Jul 17, 2024

dcherian added 2 commits July 17, 2024 09:34

Try fixing types
cb4faad

dcherianand others added 2 commits July 17, 2024 09:56

fix
792098c

Merge branch 'main' into grouper-public-api
8e1ed58

headtr1ck approved these changes Jul 17, 2024
View reviewed changes

Comment threadxarray/core/dataarray.py Outdated
Comment threadxarray/core/dataset.py Outdated
Comment threadxarray/core/groupers.py Outdated
Comment threadxarray/core/types.py Outdated
Comment threadxarray/core/types.py Outdated
Comment threadxarray/core/groupers.py Outdated

dcherianand others added 6 commits July 17, 2024 20:27

Apply suggestions from code review
8b40396
Co-authored-by: Michael Niklas <mick.niklas@gmail.com>

Review comments
a2b0f05

Add whats-new note
d521849

Fix
bf23f3d

Add more types
25c897d

Fix link
1cafa67

dcherian enabled auto-merge (squash) July 18, 2024 03:18

FIx docs
78252b2

dcherian merged commit b55c783 into pydata:mainJul 18, 2024

dcherian deleted the grouper-public-api branch November 17, 2024 16:15

Uh oh!

Uh oh!

Conversation

dcherian commented Mar 15, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcherian commented Jun 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Illviljan commented Jul 11, 2024

Uh oh!

dcherian commented Jul 11, 2024

Uh oh!

dcherian commented Jul 17, 2024

Uh oh!

headtr1ck left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dcherian commented Mar 15, 2024•
edited
Loading