FIX: correct dask array handling in _calc_idxminmax

[kmuehlbauer]

Fixes dask handling for implementation in #3871.

• Closes #xxxx
• Tests added
• Passes isort -rc . && black . && mypy . && flake8
• Fully documented, including whats-new.rst for all changes and api.rst for new API

[pep8speaks]

Hello @kmuehlbauer! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-05-09 14:03:57 UTC

[dcherian]

What breaks if you don't compute?

[kmuehlbauer]

further down, this will break:

# The dim is gone but we need to remove the corresponding coordinate.delres.coords[dim]
# Copy attributes from argmin/argmax, if anyres.attrs=indx.attrs

[kmuehlbauer]

My dask-knowledge lacks, so I was not able to come up with a better solution, unfortunately

[dcherian]

ah this was quite broken. I just pushed a commit. Please see if that works. Clearly we need to add some tests with dask-backed objects.

[kmuehlbauer]

@dcherian Thanks, I'll have a look first thing next morning (CET).

[shoyer]

+1 for adding dask tests

[kmuehlbauer]

@dcherian This seemed to work until computation:

IndexError: Unlabeled multi-dimensional array cannot be used for indexing: array_bin

where array_bin is the dimension over which idxmax is calculated. I tried to wrap my head around this to no avail. I keep trying but appreciate any hints...

Full Traceback:

/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataarray.pyincompute(self, **kwargs)
839""" 840 new = self.copy(deep=False)--> 841 return new.load(**kwargs) 842  843 def persist(self, **kwargs) -> "DataArray":/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataarray.py in load(self, **kwargs) 813 dask.array.compute 814 """-->815ds=self._to_temp_dataset().load(**kwargs)
816new=self._from_temp_dataset(ds)
817self._variable=new._variable/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataset.pyinload(self, **kwargs)
654655# evaluate all the dask arrays simultaneously-->656evaluated_data=da.compute(*lazy_data.values(), **kwargs)
657658fork, datainzip(lazy_data, evaluated_data):
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/base.pyincompute(*args, **kwargs)
435keys= [x.__dask_keys__() forxincollections]
436postcomputes= [x.__dask_postcompute__() forxincollections]
-->437results=schedule(dsk, keys, **kwargs)
438returnrepack([f(r, *a) forr, (f, a) inzip(results, postcomputes)])
439/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/threaded.pyinget(dsk, result, cache, num_workers, pool, **kwargs)
74pools[thread][num_workers] =pool75--->76results=get_async(
77pool.apply_async,
78len(pool._pool),
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/local.pyinget_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)
484_execute_task(task, data) # Re-execute locally485else:
-->486raise_exception(exc, tb)
487res, worker_id=loads(res_info)
488state["cache"][key] =res/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/local.pyinreraise(exc, tb)
314ifexc.__traceback__isnottb:
315raiseexc.with_traceback(tb)
-->316raiseexc317318/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/local.pyinexecute_task(key, task_info, dumps, loads, get_id, pack_exception)
220try:
221task, data=loads(task_info)
-->222result=_execute_task(task, data)
223id=get_id()
224result=dumps((result, id))
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/core.pyin_execute_task(arg, cache, dsk)
119# temporaries by their reference count and can execute certain120# operations in-place.-->121returnfunc(*(_execute_task(a, cache) forainargs))
122elifnotishashable(arg):
123returnarg/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/optimization.pyin__call__(self, *args)
980ifnotlen(args) ==len(self.inkeys):
981raiseValueError("Expected %d args, got %d"% (len(self.inkeys), len(args)))
-->982returncore.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
983984def__reduce__(self):
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/core.pyinget(dsk, out, cache)
149forkeyintoposort(dsk):
150task=dsk[key]
-->151result=_execute_task(task, cache)
152cache[key] =result153result=_execute_task(out, cache)
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/dask/core.pyin_execute_task(arg, cache, dsk)
119# temporaries by their reference count and can execute certain120# operations in-place.-->121returnfunc(*(_execute_task(a, cache) forainargs))
122elifnotishashable(arg):
123returnarg/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/computation.pyin<lambda>(ind, coord)
1387res=indx.copy(
1388data=indx.data.map_blocks(
->1389lambdaind, coord: coord[(ind,)], coordarray, dtype=coordarray.dtype1390 )
1391 )
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataarray.pyin__getitem__(self, key)
642else:
643# xarray-style array indexing-->644returnself.isel(indexers=self._item_key_to_dict(key))
645646def__setitem__(self, key: Any, value: Any) ->None:
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataarray.pyinisel(self, indexers, drop, **indexers_kwargs)
1020indexers=either_dict_or_kwargs(indexers, indexers_kwargs, "isel")
1021ifany(is_fancy_indexer(idx) foridxinindexers.values()):
->1022ds=self._to_temp_dataset()._isel_fancy(indexers, drop=drop)
1023returnself._from_temp_dataset(ds)
1024/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataset.pyin_isel_fancy(self, indexers, drop)
1962# Note: we need to preserve the original indexers variable in order to merge the1963# coords below->1964indexers_list=list(self._validate_indexers(indexers))
19651966variables: Dict[Hashable, Variable] = {}
/home/kai/miniconda/envs/wradlib_38_01/lib/python3.8/site-packages/xarray/core/dataset.pyin_validate_indexers(self, indexers)
18051806ifv.ndim>1:
->1807raiseIndexError(
1808"Unlabeled multi-dimensional array cannot be "1809"used for indexing: {}".format(k)
IndexError: Unlabeledmulti-dimensionalarraycannotbeusedforindexing: array_bin

[max-sixty]

@kmuehlbauer what's the code that's running to generate that traceback? I can try and help in lieu of @dcherian

Thanks for giving this a go

And CC @toddrjen if they have any insight

[kmuehlbauer]

@max-sixty Thanks! I'll really appreciate your help. I've tracked the possible source down to a dimension problem. I've tried to create a minimal example as follows using the current idxmax implementation from above. I copied only the dask related lines of code:

# create dask backed 3d arraydarray=da.from_array(np.random.RandomState(0).randn(10*20*30).reshape(10, 20, 30), chunks=(10, 20, 30), name='data_arr')
array=xr.DataArray(darray, dims=["x", "y", 'z'])
array=array.assign_coords({'x': (['x'], np.arange(10)),
'y': (['y'], np.arange(20)),
'z': (['z'], np.arange(30)),
})
func=lambdax, *args, **kwargs: x.argmax(*args, **kwargs)
indx=func(array, dim='z', axis=None, keep_attrs=True, skipna=False)
coordarray=array['z']
res=indx.copy(
data=indx.data.map_blocks(
lambdaind, coord: coord[(ind,)], coordarray, dtype=coordarray.dtype
)
)
print(res)
# the following line break breaksprint(res.compute())
# using only 2dim array everything works as intendedarray2d=array.sel(y=0, drop=True)
indx=func(array2d, dim='z', axis=None, keep_attrs=True, skipna=False)
coordarray=array['z']
res=indx.copy(
data=indx.data.map_blocks(
lambdaind, coord: coord[(ind,)], coordarray, dtype=coordarray.dtype
)
)
print(res)
# this works for two dim dataprint(res.compute())

[keewis]

The issue is that argmax produces a ndim - 1 array of indexes. In the case of 2D input data that would be 1D, so since indexing only allows to index with 1D arrays, your code works. For ndim > 2, we'd be trying to index with arrays with ndim > 1, so the indexing fails.

To get your 3D example (and potentially every N-D example) to work, simply fall back to the wrapped array's integer indexing (using coordarray.data):

In [2]: darray=da.from_array( ...: np.random.RandomState(0).randn(10*20*30).reshape(10, 20, 30), ...: chunks=(1, 20, 30), # so we actually have multiple blocks
...: name='data_arr' ...: ) ...: array=xr.DataArray( ...: darray, ...: dims=["x", "y", 'z'], ...: coords={"x": np.arange(10), "y": np.arange(20), "z": np.arange(30)}, ...: ) ...: arrayOut[2]: <xarray.DataArray'data_arr' (x: 10, y: 20, z: 30)>dask.array<data_arr, shape=(10, 20, 30), dtype=float64, chunksize=(1, 20, 30), chunktype=numpy.ndarray>Coordinates:
*x (x) int640123456789*y (y) int64012345678910111213141516171819*z (z) int64012345678910 ... 20212223242526272829In [3]: indx=array.argmin(dim='z', keep_attrs=True, skipna=False) ...: res=indx.copy( ...: data=indx.data.map_blocks( ...: lambdaind, coord: coord[(ind,)], ...: array.z.data, ...: dtype=array.z.dtype ...: ) ...: )
In [4]: res.compute()
Out[4]: <xarray.DataArray'data_arr' (x: 10, y: 20)>array([[20, 3, 3, 11, 20, 17, 3, 27, 24, 1, 7, 4, 22, 14, 7, 18,
5, 18, 7, 19],
[10, 21, 25, 3, 15, 25, 28, 8, 10, 9, 13, 3, 24, 17, 19, 23,
12, 19, 19, 28],
[ 1, 26, 10, 9, 16, 8, 17, 8, 6, 24, 28, 13, 23, 22, 26, 13,
28, 11, 6, 16],
[ 6, 9, 26, 27, 1, 2, 21, 8, 10, 19, 14, 14, 20, 25, 24, 4,
18, 12, 20, 2],
[22, 5, 12, 17, 13, 23, 23, 8, 27, 22, 1, 19, 26, 16, 12, 17,
19, 28, 8, 12],
[20, 8, 25, 13, 4, 12, 23, 13, 27, 18, 15, 28, 10, 10, 0, 12,
5, 14, 5, 27],
[29, 0, 19, 7, 15, 2, 8, 8, 13, 4, 12, 1, 7, 19, 14, 0,
3, 7, 12, 9],
[ 9, 8, 4, 9, 17, 6, 7, 5, 29, 0, 15, 28, 22, 6, 24, 24,
20, 0, 24, 23],
[ 1, 19, 12, 20, 4, 26, 5, 13, 21, 26, 25, 10, 5, 1, 11, 21,
6, 18, 4, 21],
[15, 27, 13, 7, 25, 3, 14, 14, 17, 15, 11, 4, 16, 22, 22, 23,
0, 16, 26, 13]])
Coordinates:
*x (x) int640123456789*y (y) int64012345678910111213141516171819

Note that in this case res == indx since z = np.arange(30)

[kmuehlbauer]

@keewis Thanks a bunch for the explanation. Would we be on the safe side, if we use your proposed N-D example? It also works in the 2d-case.

[keewis]

👍

[kmuehlbauer]

@keewis I checked with my datasets, works like a charm. I'll try to add dask tests for this as @shoyer suggested. Where should these tests go? Currently the idxmax/idxmin tests are in test_dataarray and test_dataset:

xarray/xarray/tests/test_dataarray.py

Line 4512 in b3bafee

deftest_idxmin(self, x, minindex, maxindex, nanindex):

xarray/xarray/tests/test_dataarray.py

Line 4608 in b3bafee

deftest_idxmax(self, x, minindex, maxindex, nanindex):

xarray/xarray/tests/test_dataset.py

Lines 4603 to 4610 in 1416d5a

	expected=Dataset({"x": -10})
	actual=ds.idxmin()
	assert_identical(expected, actual)
	expected=Dataset({"x": 10})
	actual=ds.idxmax()
	assert_identical(expected, actual)

Any pointers?

[keewis]

I'd put the dask tests directly below the idxmin / idxmax tests, decorated with requires_dask.

Edit: that's easy for test_dataarray.py, but not as much for test_dataset.py. Since Dataset delegates to the DataArray methods we might get away with not adding dask tests to Dataset, but I'm not really sure.

[kmuehlbauer]

@keewis I've started by adding the dask tests to the existing idxmin/idxmaxtests (test_dataarray). This works locally, but one tests (data is datetimes) fails for the dask test. Any immediate thought on this?

[keewis]

it seems you can't use argmin with dask arrays if the dtype is datetime64 (M8):

In [24]: time=np.asarray(pd.date_range("2019-07-17", periods=10)) ...: array=xr.DataArray( ...: time, ...: dims="x", ...: coords={"x": np.arange(time.size) *4}, ...: ).chunk({}) ...: arrayOut[24]: <xarray.DataArray (x: 10)>dask.array<xarray-<this-array>, shape=(10,), dtype=datetime64[ns], chunksize=(10,), chunktype=numpy.ndarray>Coordinates:
*x (x) int6404812162024283236In [25]: array.compute().argmin(dim="x") Out[25]: <xarray.DataArray ()>array(0)
In [26]: array.argmin(dim="x") ---------------------------------------------------------------------------UFuncTypeErrorTraceback (mostrecentcalllast)
<ipython-input-26-e665d5b1b9b4>in<module>---->1array.argmin(dim="x")
.../xarray/core/common.pyinwrapped_func(self, dim, axis, skipna, **kwargs)
4445defwrapped_func(self, dim=None, axis=None, skipna=None, **kwargs):
--->46returnself.reduce(func, dim, axis, skipna=skipna, **kwargs)
4748else:
.../xarray/core/dataarray.pyinreduce(self, func, dim, axis, keep_attrs, keepdims, **kwargs)
2260 """
2261->2262var=self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs)
2263returnself._replace_maybe_drop_dims(var)
2264 .../xarray/core/variable.pyinreduce(self, func, dim, axis, keep_attrs, keepdims, allow_lazy, **kwargs)
15731574ifaxisisnotNone:
->1575data=func(input_data, axis=axis, **kwargs)
1576else:
1577data=func(input_data, **kwargs)
.../xarray/core/duck_array_ops.pyinf(values, axis, skipna, **kwargs)
302303try:
-->304returnfunc(values, axis=axis, **kwargs)
305exceptAttributeError:
306ifnotisinstance(values, dask_array_type):
.../xarray/core/duck_array_ops.pyinf(*args, **kwargs)
45else:
46wrapped=getattr(eager_module, name)
--->47returnwrapped(*args, **kwargs)
4849else:
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinwrapped(x, axis, split_every, out)
10021003defwrapped(x, axis=None, split_every=None, out=None):
->1004returnarg_reduction(
1005x, chunk, combine, agg, axis, split_every=split_every, out=out1006 )
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinarg_reduction(x, chunk, combine, agg, axis, split_every, out)
980tmp=Array(graph, name, chunks, dtype=x.dtype)
981dtype=np.argmin([1]).dtype-->982result=_tree_reduce(tmp, agg, axis, False, dtype, split_every, combine)
983returnhandle_out(out, result)
984~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyin_tree_reduce(x, aggregate, axis, keepdims, dtype, split_every, combine, name, concatenate, reduced_meta)
243ifconcatenate:
244func=compose(func, partial(_concatenate2, axes=axis))
-->245returnpartial_reduce(
246func,
247x,
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinpartial_reduce(func, x, split_every, keepdims, dtype, name, reduced_meta)
314ifis_arraylike(meta) andmeta.ndim!=len(out_chunks):
315iflen(out_chunks) ==0:
-->316meta=meta.sum()
317else:
318meta=meta.reshape((0,) *len(out_chunks))
~/.conda/envs/xarray/lib/python3.8/site-packages/numpy/core/_methods.pyin_sum(a, axis, dtype, out, keepdims, initial, where)
36def_sum(a, axis=None, dtype=None, out=None, keepdims=False,
37initial=_NoValue, where=True):
--->38returnumr_sum(a, axis, dtype, out, keepdims, initial, where)
3940def_prod(a, axis=None, dtype=None, out=None, keepdims=False,
UFuncTypeError: ufunc'add'cannotuseoperandswithtypesdtype('<M8[ns]') anddtype('<M8[ns]')

I guess that's a dask bug?

Edit: you can reproduce it without xarray, so definitely a bug in dask:

MWE with only numpy / dask.array

In [32]: time=np.asarray(pd.date_range("2019-07-17", periods=10)) ...: np.argmin(da.from_array(time)) ---------------------------------------------------------------------------UFuncTypeErrorTraceback (mostrecentcalllast)
<ipython-input-32-190cb901ff65>in<module>1time=np.asarray(pd.date_range("2019-07-17", periods=10))
---->2np.argmin(da.from_array(time))
<__array_function__internals>inargmin(*args, **kwargs)
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/core.pyin__array_function__(self, func, types, args, kwargs)
1348ifda_funcisfunc:
1349returnhandle_nonmatching_names(func, args, kwargs)
->1350returnda_func(*args, **kwargs)
13511352 @property~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinwrapped(x, axis, split_every, out)
10021003defwrapped(x, axis=None, split_every=None, out=None):
->1004returnarg_reduction(
1005x, chunk, combine, agg, axis, split_every=split_every, out=out1006 )
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinarg_reduction(x, chunk, combine, agg, axis, split_every, out)
980tmp=Array(graph, name, chunks, dtype=x.dtype)
981dtype=np.argmin([1]).dtype-->982result=_tree_reduce(tmp, agg, axis, False, dtype, split_every, combine)
983returnhandle_out(out, result)
984~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyin_tree_reduce(x, aggregate, axis, keepdims, dtype, split_every, combine, name, concatenate, reduced_meta)
243ifconcatenate:
244func=compose(func, partial(_concatenate2, axes=axis))
-->245returnpartial_reduce(
246func,
247x,
~/.conda/envs/xarray/lib/python3.8/site-packages/dask/array/reductions.pyinpartial_reduce(func, x, split_every, keepdims, dtype, name, reduced_meta)
314ifis_arraylike(meta) andmeta.ndim!=len(out_chunks):
315iflen(out_chunks) ==0:
-->316meta=meta.sum()
317else:
318meta=meta.reshape((0,) *len(out_chunks))
~/.conda/envs/xarray/lib/python3.8/site-packages/numpy/core/_methods.pyin_sum(a, axis, dtype, out, keepdims, initial, where)
36def_sum(a, axis=None, dtype=None, out=None, keepdims=False,
37initial=_NoValue, where=True):
--->38returnumr_sum(a, axis, dtype, out, keepdims, initial, where)
3940def_prod(a, axis=None, dtype=None, out=None, keepdims=False,
UFuncTypeError: ufunc'add'cannotuseoperandswithtypesdtype('<M8[ns]') anddtype('<M8[ns]')

[kmuehlbauer]

@keewis OK, how should I handle this? Shall we XFAIL the these tests then?

[keewis]

I think so? For now, xfail if use_dask and x.dtype.kind == "M"

[kmuehlbauer]

Seems that everything goes well, besides the upstream-dev run.

If this is ready for merge, should I extend the idxmin/idxmax section of whats-new.rst ? And how should I distribute the credit for all contributors @dcherian, @keewis, @max-sixty?

[keewis]

I'd say add a new entry. Also, I think we're all just reviewers so adding just your name should be fine.

[kmuehlbauer]

This is ready from my end for final review.

Should be merged before #3936, IMHO.

[dcherian]

If your tests are passing now, it's likely that they're computing things to make things work. We should add the with raise_if_dask_computes() context (imported from test_dask.py) when testing with dask variables but it looks non-trivial given how the tests are structured at the moment.

[kmuehlbauer]

@dcherian Thanks for the suggestion with the dask compute context, I'll have a look the next day.

Nevertheless, I've debugged locally and the res-output of the idxmin/idxmax holds dask data.

Anyway, I'll revert to the former working status and leave a comment to this PR in the code.

[dcherian]

I suggest updating the tests before reverting anything. This solution may work ...

[kmuehlbauer]

@dcherian I've tried to apply the with raise_if_dask_computes() context, but as you already mentioned, it is quite hard to incorporate into the tests. Every once in a while we would need to introduce some if/else clause to check for use_dask and also the computation count inside idxmin/idxmax varies based on source data layout (because of usage of argmin/argmax inside).

I'm now totally unsure how to proceed from here. Any guidance very much appreciated.

[dcherian]

@kmuehlbauer I've pushed a commit adding the decorator to just the 2D test_idxmax. The decorator does nothing for numpy so all the numpy tests pass but the dask tests fail now (because it is indeed computing things). Prior to this commit the 2D tests were passing, so we should go back to the map_blocks solution I guess.

because you can't index a NumPy array with a dask array?

I think @shoyer is right here.

[kmuehlbauer]

@dcherian Thanks for explaining the decorator a bit more. So it's indeed simpler than I thought. I'll revert to the map_blocks solution. I'll not have time today, so this will have to wait a bit.

[dcherian]

Not a problem. Thanks for working on this!

[shoyer]

A simpler option than using map_blocks might be to cast the array being indexed into a Dask array if needed, using dask.array.asarray(). Then both objects would be dask arrays, so indexing should work.

[kmuehlbauer]

@shoyer Thanks for the hint.

I'm currently experimenting with the different possibilities.

non-dask:

<xarray.DataArray (y: 3, x: 7)>
array([[ 0, 1, 2, 0, -2, -4, 2],
[ 1, 1, 1, 1, 1, 1, 1],
[ 0, 0, -10, 5, 20, 0, 0]])
Coordinates:
* x (x) int64 0 4 8 12 16 20 24
* y (y) int64 1 0 -1
Attributes:
attr1: value1
attr2: 2929

dask:

<xarray.DataArray (y: 3, x: 7)>
dask.array<xarray-<this-array>, shape=(3, 7), dtype=int64, chunksize=(3, 7), chunktype=numpy.ndarray>
Coordinates:
* x (x) int64 0 4 8 12 16 20 24
* y (y) int64 1 0 -1
Attributes:
attr1: value1
attr2: 2929

The relevant code inside idxmin/idxmax:

# This will run argmin or argmax.# indx will be dask if array is daskindx=func(array, dim=dim, axis=None, keep_attrs=keep_attrs, skipna=skipna)
# separated out for debugging# with the current test layout coords will not be dask since the array's coords are not daskcoords=array[dim]
# try to make dask of coords as per @shoyer's suggestion # the below fails silently, but cannot be forced even with trying to # do something like dask.array.asarray(), this errors out with# "Cannot assign to the .data attribute of dimension coordinate a.k.a IndexVariable 'x'"ifisinstance(indx.data, dask_array_type):
coords=coords.chunk({})
res=coords[(indx,)]

It seems that the map-blocks approach is the only one which seem to work throughout the tests, but one. It fails with array set as array.astype("object") in the test fixture. Reason: dask gets computed within argmin/argmax.

I'll revert to the map-blocks now and add the with raise_if_dask_computes() , context where neccessary. Any hints appreciated for the dask compute-error.

[kmuehlbauer]

Error log of the compute error:
https://dev.azure.com/xarray/xarray/_build/results?buildId=2629&view=logs&j=78b48a04-306f-5a15-9ac3-dd2fdb28db5e&t=5160aa4e-6217-5012-6424-4f17180b374b&l=412

Details

xarray/tests/test_dask.py:52: RuntimeError
_______ TestReduce2D.test_idxmax[True-x2-minindex2-maxindex2-nanindex2] ________
self = <xarray.tests.test_dataarray.TestReduce2D object at 0x7fbf71ecba58>
xarray/core/variable.py:1579: in reduce
data = func(input_data, axis=axis, **kwargs)
xarray/core/duck_array_ops.py:304: in f
return func(values, axis=axis, **kwargs)
xarray/core/nanops.py:104: in nanargmax
return _nan_argminmax_object("argmax", fill_value, a, axis=axis)
xarray/core/nanops.py:57: in _nan_argminmax_object
if (valid_count == 0).any():
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/array/core.py:1375: in __bool__
return bool(self.compute())
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/base.py:175: in compute
(result,) = compute(self, traverse=False, **kwargs)
/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/dask/base.py:446: in compute
results = schedule(dsk, keys, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <xarray.tests.test_dask.CountingScheduler object at 0x7fbf718e9160>
dsk = {('all-all-aggregate-any-getitem-invert-sum-sum-aggregate-any-aggregate-a1eb5c32ca3225c37fddde3d1aa0d2f0',): (Compose(... -4.0, 2.0],
[-4.0, nan, 2.0, nan, -2.0, -4.0, 2.0],
[nan, nan, nan, nan, nan, nan, nan]], dtype=object)}
keys = [[('any-aggregate-a1eb5c32ca3225c37fddde3d1aa0d2f0',)]], kwargs = {}
def __call__(self, dsk, keys, **kwargs):
self.total_computes += 1
if self.total_computes > self.max_computes:
raise RuntimeError(
"Too many computes. Total: %d > max: %d."
> % (self.total_computes, self.max_computes)
)
E RuntimeError: Too many computes. Total: 1 > max: 0.

[dcherian]

Yeah interestingly we don't raise an error when trying to chunk IndexVariables.

I've pushed a commit where we extract the underlying numpy array, chunk that, index it and then wrap it up in a DataArray o_O.

[dcherian]

The compute error is from here:

xarray/xarray/core/nanops.py

Lines 48 to 60 in 6a6f2c8

	def_nan_argminmax_object(func, fill_value, value, axis=None, **kwargs):
	""" In house nanargmin, nanargmax for object arrays. Always return integer
	type
	"""
	valid_count=count(value, axis=axis)
	value=fillna(value, fill_value)
	data=_dask_or_eager_func(func)(value, axis=axis, **kwargs)
	# TODO This will evaluate dask arrays and might be costly.
	if (valid_count==0).any():
	raiseValueError("All-NaN slice encountered")
	returndata

I think we'll have to rethink the skipna conditions for dask arrays so that the compute doesn't happen.

Or figure out why we do this check in the first place. hmm...

[dcherian]

The test fails for object arrays because we compute eagerly in nanops._nan_argminmax_object to raise an error for all-NaN slices.

To solve this we could

1. fix Allow for All-NaN in argmax, argmin #3884 so that nanargmin, nanargmax never raise an error for all-NaN slices.
2. Figure out some clever way to raise the error at compute time rather than graph construction time.

For now, I bumped up max_computes to 1 for object arrays.

[kmuehlbauer]

Thanks @dcherian for getting back to this. To my bad, this adventure went too far for my capabilities. Nevertheless I hope to catch up learning xarray inside.