Enable pandas type checking by headtr1ck · Pull Request #9213 · pydata/xarray

headtr1ck · 2024-07-07T20:17:51Z

Since pandas-stubs is making good progress, I would say it is slowly advantageous to enable type checking for it.
About 200 mypy errors have to be fixed....

User visible changes (including notable bug fixes) are documented in whats-new.rst

headtr1ck · 2024-07-07T21:11:08Z

It seems that mypy is now resolving the operators more precisely and also it seems that this causes some issues with the current implementation.

It looks like we have to completely rework them... :/
This is gonna take some time to get it right!

headtr1ck · 2024-07-08T19:59:56Z

After some debugging I noticed that DatasetGroupby + Dataset-subclass will return Dataset and not Dataset-subclass.
So our current typing of Self is wrong.

I think refactoring the code to actually return the subclass is blocked by the Coordinates refactor...

max-sixty · 2024-07-08T20:20:11Z

It seems that mypy is now resolving the operators more precisely and also it seems that this causes some issues with the current implementation.
It looks like we have to completely rework them... :/

For the pandas items or for all? 🫨

FWIW IIRC the latest mypy fails on main (though that could be a much much smaller & separate change...)

max-sixty · 2024-07-08T20:23:48Z

After some debugging I noticed that DatasetGroupby + Dataset-subclass will return Dataset and not Dataset-subclass.
So our current typing of Self is wrong.

I remember hitting similar things in the past. Ideally we'd return the subclass; but to the extent we don't I would heavily deprioritize any big effort to change the typing to state we return Dataset — instead I would vote to push it off until we can return the subclass...

headtr1ck · 2024-07-08T20:44:30Z

For the pandas items or for all? 🫨

I think I have fixed all mypy issues related to pandas.

FWIW IIRC the latest mypy fails on main (though that could be a much much smaller & separate change...)

I though of quickly fixing these issues here as well, turns out not to be so easy...

headtr1ck · 2024-07-08T21:32:12Z

Issue seems to be in xarray.core.types with

try:
fromdask.arrayimportArrayasDaskArrayexceptImportError:
DaskArray=np.ndarray

Somehow that gets evaluated to Any, which in turn makes mypy think that DataArray + anything = DataArray, even when this "anything" equals a Dataset/DatasetGroupby...

If I try to recreate this locally, I don't get Any, so no idea what is going on there...

Done for today, lets see how one solves this issue.

headtr1ck · 2024-07-11T20:25:22Z

After hours of trying to figure out what is going on, I am close to giving up :/
If I create a file xarray.core.types2.py with the exact same content as xarray.core.types.py, the DaskArray gets correctly resolved in types2 but not types.... WTH is going on?
Is that a mypy bug???? :/

max-sixty · 2024-07-11T23:06:55Z

That doesn't sound fun...

If I create a file xarray.core.types2.py with the exact same content as xarray.core.types.py, the DaskArray gets correctly resolved in types2 but not types.... WTH is going on?

Where are the errors from types.py? I don't see them in https://github.com/pydata/xarray/actions/runs/9845800832/job/27182145496?pr=9213#step:8:58

headtr1ck · 2024-07-12T08:16:19Z

Where are the errors from types.py? I don't see them in https://github.com/pydata/xarray/actions/runs/9845800832/job/27182145496?pr=9213#step:8:58

As I was saying some comments up, the mypy errors are the ones in _typed_ops. But these are only a symptom not the root of the error...

Mainly: Variable + Any = Variable, but should not be the case for e.g. Dataset.
This is because we type the "other" as Union[ArrayLike, Variable, DaskArray], but since DaskArray = Any thats what happens...

max-sixty · 2024-07-12T18:10:35Z

Something does seem confusing with DaskArray.

One thing that does seem to fix almost all errors is just excluding DaskArray. Arguably even if we can't improve from here this would be a quite worthwhile tradeoff. In particular there is a lot of code here it would be great to merge before we get conflicts etc, to the extent we can push dask typing precision off...

diff --git a/xarray/core/types.py b/xarray/core/types.py
index 20d94de5..5b231780 100644
--- a/xarray/core/types.py+++ b/xarray/core/types.py@@ -178,7 +178,8 @@ def copy(
T_ExtensionArray = TypeVar("T_ExtensionArray", bound=pd.api.extensions.ExtensionArray)
-ScalarOrArray = Union["ArrayLike", np.generic, np.ndarray, "DaskArray"]+# ScalarOrArray = Union[ArrayLike, np.generic, np.ndarray, DaskArray]+ScalarOrArray = Union[ArrayLike, np.generic, np.ndarray]
VarCompatible = Union["Variable", "ScalarOrArray"]
DaCompatible = Union["DataArray", "VarCompatible"]
DsCompatible = Union["Dataset", "DaCompatible"]

FWIW I also get weirdness around using dmypy with these — after a few changes it reports quite different results than mypy. Maybe it's a version thing...

headtr1ck · 2024-07-12T19:22:21Z

@max-sixty thank you! That was the solution, and it does not even matter, because a DaskArray is ArrayLike anyway!
Wow, that was some wasted hours.
Still I believe that this is some kind of mypy bug, but too difficult to trace!

headtr1ck · 2024-07-12T19:25:15Z

remaining mypy errors are from numpy2, see #9231

max-sixty

Very impressive @headtr1ck ! Big lift!

dcherian · 2024-07-17T04:01:03Z

Force-merged since the auto-merge won't work till mypy on main is fixed.

* main: Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" Test main push Revert "Update _typing.py" Update _typing.py Add a `.drop_attrs` method (pydata#8258)

* main: add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234)

* main: (54 commits) Adding `open_datatree` backend-specific keyword arguments (#9199) [pre-commit.ci] pre-commit autoupdate (#9202) Restore ability to specify _FillValue as Python native integers (#9258) add backend intro and how-to diagram (#9175) Fix copybutton for multi line examples in double digit ipython cells (#9264) Update signature for _arrayfunction.__array__ (#9237) Add encode_cf_datetime benchmark (#9262) groupby, resample: Deprecate some positional args (#9236) Delete ``base`` and ``loffset`` parameters to resample (#9233) Update dropna docstring (#9257) Grouper, Resampler as public api (#8840) Fix mypy on main (#9252) fix fallback isdtype method (#9250) Enable pandas type checking (#9213) Per-variable specification of boolean parameters in open_dataset (#9218) test push Added a space to the documentation (#9247) Fix typing for test_plot.py (#9234) Allow mypy to run in vscode (#9239) Revert "Test main push" ...

…monotonic-variable * main: (995 commits) Adding `open_datatree` backend-specific keyword arguments (pydata#9199) [pre-commit.ci] pre-commit autoupdate (pydata#9202) Restore ability to specify _FillValue as Python native integers (pydata#9258) add backend intro and how-to diagram (pydata#9175) Fix copybutton for multi line examples in double digit ipython cells (pydata#9264) Update signature for _arrayfunction.__array__ (pydata#9237) Add encode_cf_datetime benchmark (pydata#9262) groupby, resample: Deprecate some positional args (pydata#9236) Delete ``base`` and ``loffset`` parameters to resample (pydata#9233) Update dropna docstring (pydata#9257) Grouper, Resampler as public api (pydata#8840) Fix mypy on main (pydata#9252) fix fallback isdtype method (pydata#9250) Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" ...

* main: (54 commits) Adding `open_datatree` backend-specific keyword arguments (pydata#9199) [pre-commit.ci] pre-commit autoupdate (pydata#9202) Restore ability to specify _FillValue as Python native integers (pydata#9258) add backend intro and how-to diagram (pydata#9175) Fix copybutton for multi line examples in double digit ipython cells (pydata#9264) Update signature for _arrayfunction.__array__ (pydata#9237) Add encode_cf_datetime benchmark (pydata#9262) groupby, resample: Deprecate some positional args (pydata#9236) Delete ``base`` and ``loffset`` parameters to resample (pydata#9233) Update dropna docstring (pydata#9257) Grouper, Resampler as public api (pydata#8840) Fix mypy on main (pydata#9252) fix fallback isdtype method (pydata#9250) Enable pandas type checking (pydata#9213) Per-variable specification of boolean parameters in open_dataset (pydata#9218) test push Added a space to the documentation (pydata#9247) Fix typing for test_plot.py (pydata#9234) Allow mypy to run in vscode (pydata#9239) Revert "Test main push" ...

headtr1ck added 12 commits July 1, 2024 22:38

remove pandas from ignore missing imports
678536c

add Any to dim arg of concat as placeholder
20066c4

allow sequence of np.ndarrays as coords in dataArray constructor
9053b60

fix several typing issues in tests
27bd0ab

fix more types
0dcb2a1

more fixes
a2b5b4c

more typing...
6a75746

we are getting there?
ce657e4

who might have guessed it... more typing
5589156

continue fixing typing issues
e45d28b

fix some typed_ops
faa8bcc

fix last non-typed-ops errors
2b678b1

headtr1ck added the topic-typing label Jul 7, 2024

Merge branch 'pydata:main' into pandas-stubs
7b08a94

Merge branch 'main' into pandas-stubs
69b93dd

headtr1ck added 2 commits July 12, 2024 21:19

update typed ops
ab4dbf5

remove useless DaskArray type in scalar or array type
27db395

headtr1ck marked this pull request as ready for review July 12, 2024 19:23

headtr1ck added 10 commits July 12, 2024 21:27

fix missing import in type_checking
5977107

fix import
64765c8

improve cftime offsets typing
5b1d98c

fix classvars
16bee67

fix some checks
602c3cf

fix a broken test
07a52ae

Merge branch 'main' into pandas-stubs
a93bba6

Merge branch 'main' into pandas-stubs
f2f5dfb

improve typing of test_concat
a5eaa82

fix broken concat
2f12f88

max-sixty approved these changes Jul 15, 2024
View reviewed changes

add whats-new
4bfec5e

headtr1ck added the plan to merge Final call for comments label Jul 15, 2024

Merge branch 'main' into pandas-stubs
2f95dbd

headtr1ck enabled auto-merge (squash) July 16, 2024 18:40

dcherian disabled auto-merge July 17, 2024 04:00

dcherian merged commit 71fce9b into pydata:mainJul 17, 2024

headtr1ck deleted the pandas-stubs branch July 18, 2024 18:51

Illviljan mentioned this pull request Aug 4, 2024
Use assert_duckarray_allclose instead of rounding the array #9313
Merged

Uh oh!

Uh oh!

Conversation

headtr1ck commented Jul 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

headtr1ck commented Jul 7, 2024

Uh oh!

headtr1ck commented Jul 8, 2024

Uh oh!

max-sixty commented Jul 8, 2024

Uh oh!

max-sixty commented Jul 8, 2024

Uh oh!

headtr1ck commented Jul 8, 2024

Uh oh!

headtr1ck commented Jul 8, 2024

Uh oh!

headtr1ck commented Jul 11, 2024

Uh oh!

max-sixty commented Jul 11, 2024

Uh oh!

headtr1ck commented Jul 12, 2024

Uh oh!

max-sixty commented Jul 12, 2024

Uh oh!

headtr1ck commented Jul 12, 2024

Uh oh!

headtr1ck commented Jul 12, 2024

Uh oh!

max-sixty left a comment

Choose a reason for hiding this comment

Uh oh!

dcherian commented Jul 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

headtr1ck commented Jul 7, 2024•
edited
Loading