Challenges running xarray wrapped netcdf files

[pwolfram]

This is a traceback from calling compute on an XArray computation on dask.distributed.

We're able to use dask.array on a NetCDF4 object without locks if our workers have single threads. However, when computing on the .data attribute backed by a NetCDF object wrapped by a few XArray containers we run into the following error. It appears to be coming from computing the shape, which is odd. Traceback below:

cc @mrocklin @shoyer

In [168]: ds = xr.open_mfdataset(fname, lock=False)

In [169]: ds.yParticle.data.sum().compute()

/net/scratch3/pwolfram/miniconda2/lib/python2.7/site-packages/dask/array/core.pyc in getarray()
 47 lock.acquire()
 48 try:
---> 49 c = a[b]
 50 if type(c) != np.ndarray:
 51 c = np.asarray(c)

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in __getitem__()
 396 
 397 def __getitem__(self, key):
--> 398 return type(self)(self.array, self._updated_key(key))
 399 
 400 def __setitem__(self, key, value):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in _updated_key()
 372 
 373 def _updated_key(self, new_key):
--> 374 new_key = iter(canonicalize_indexer(new_key, self.ndim))
 375 key = []
 376 for size, k in zip(self.array.shape, self.key):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in ndim()
 380 @property
 381 def ndim(self):
--> 382 return len(self.shape)
 383 
 384 @property

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
 384 def shape(self):
 385 shape = []
--> 386 for size, k in zip(self.array.shape, self.key):
 387 if isinstance(k, slice):
 388 shape.append(len(range(*k.indices(size))))

/users/pwolfram/lib/python2.7/site-packages/xarray/conventions.pyc in shape()
 447 @property
 448 def shape(self):
--> 449 return self.array.shape[:-1]
 450 
 451 def __str__(self):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
 384 def shape(self):
 385 shape = []
--> 386 for size, k in zip(self.array.shape, self.key):
 387 if isinstance(k, slice):
 388 shape.append(len(range(*k.indices(size))))

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in shape()
 407 @property
 408 def shape(self):
--> 409 return self.array.shape
 410 
 411 def __array__(self, dtype=None):

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:32778)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._getdims (netCDF4/_netCDF4.c:31870)()

RuntimeError: NetCDF: Not a valid ID

[mrocklin]

I worked with @pwolfram to produce that traceback. We were able to get things working if we dove into the .dask graph to get the NetCDF file and then wrap it with a dask.array, something like the following:

xr_var = ds.yParticle.data.dask['some-key']
var = xr_var.array.array.array # unpack

x = da.from_array(var, chunks=(...))
x.sum().compute() # works ok

[shoyer]

This makes sense now that I'm looking at the full traceback.

This making use of conventions.CharToStringArray, another array-like type that xarray uses for decoding string data, for which we are relying on default serialization (pickle). The netCDF4.Variable get unpickled an no longer points to an open file.

Let's try this again opening the dataset with decode_cf=False.

[pwolfram]

It looks the same.

[shoyer]

Somehow we seem to be using a CharToStringArray....

On Sun, Nov 6, 2016 at 2:33 PM, Phillip Wolfram notifications@github.com
wrote:

It looks the same.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#629 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKS1stI69DmsWS6PvJ_rjuZMoWRurNxks5q7lWngaJpZM4Kqsny
.

[shoyer]

What are the types of each level of xr_var.array.array.array?

Also try concat_characters=False

[shoyer]

Also: xr.open_dataset(fname, lock=False, chunks={}) should make a dask array without the mulit-file logic.

[pwolfram]

concat_characters=False is the same error.

[pwolfram]

levels of xr_var.array.array.array:

In [204]: type(xr_var.array)
Out[204]: xarray.core.indexing.LazilyIndexedArray

In [205]: type(xr_var.array.array)
Out[205]: xarray.backends.netCDF4_.NetCDF4ArrayWrapper

In [206]: type(xr_var.array.array.array)
Out[206]: netCDF4._netCDF4.Variable

[pwolfram]

xr.open_dataset(fname, lock=False, chunks={}) is the same error.

[shoyer]

Following your stacktrace down, I see five array types:

LazilyIndexedArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
--> 386 for size, k in zip(self.array.shape, self.key):

CharToStringArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/conventions.pyc in shape()
--> 449 return self.array.shape[:-1]

LazilyIndexedArray:

/users/pwolfram/lib/python2.7/site-packages/xarray/core/indexing.pyc in shape()
--> 386 for size, k in zip(self.array.shape, self.key):

NetCDF4ArrayWrapper (most likely, definitely a NDArrayMixin subclass):

/users/pwolfram/lib/python2.7/site-packages/xarray/core/utils.pyc in shape()
--> 409 return self.array.shape

netCDF4.Variable:

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.shape.__get__ (netCDF4/_netCDF4.c:32778)()

So it really looks like dask-distributed is choking on CharToStringArray.

In particular, it appears that the original error is from a different array than xr_var. Are you sure you're pulling out the top level key from the dask dict?

To be doubly sure, what is the dtype of ds.yParticle? It is also conceivable (though I think unlikely) that some other array with string data has been pulled into it's dask graph (ds.yParticle.data.dask). Some use of dask.optimize.cull could test that hypothesis.

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?

[pwolfram]

In particular, it appears that the original error is from a different array than xr_var. Are you sure you're pulling out the top level key from the dask dict?

In [214]: ds.yParticle.data.dask
Out[214]: 
{(u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eb',
 0,
 0): (<function dask.array.core.getarray>,
 u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a',
 (slice(0, 31, None), slice(0, 1012000, None))),
 u'fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a': LazilyIndexedArray(array=LazilyIndexedArray(array=NetCDF4ArrayWrapper(array=<type 'netCDF4._netCDF4.Variable'>
 float64 yParticle(Time, nParticles)
 unlimited dimensions: Time
 current shape = (31, 1012000)
 filling off
 ), key=(slice(None, None, None), slice(None, None, None))), key=(slice(None, None, None), slice(None, None, None)))}

In [215]: x = ds.yParticle.data.dask['fname:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394
 ...: a']

So I think this is the top level key from the dask dict, but note that this may not be the case

In [219]: ds.yParticle.data.dask.keys()
Out[219]: 
[(u'filename:/yParticle-846a0722e86ecac24903e03f48aa35eb',
 0,
 0),
 u'filename:/yParticle-846a0722e86ecac24903e03f48aa35eboriginal-f52ad42f43568bf502049f452c4b394a']

In [220]: 

In [220]: x = ds.yParticle.data.dask['In [219]: ds.yParticle.data.dask.keys()
 File "<ipython-input-220-f7fc92c41ba8>", line 1
 x = ds.yParticle.data.dask['In [219]: ds.yParticle.data.dask.keys()
 ^
SyntaxError: EOL while scanning string literal


In [221]: x = ds.yParticle.data.dask['filename:/yParticle-846a0722e86ecac24903e03f48aa35eb']
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-221-94db2c484015> in <module>()
----> 1 x = ds.yParticle.data.dask['filename:/yParticle-846a0722e86ecac24903e03f48aa35eb']

KeyError: 'filename:/yParticle-846a0722e86ecac24903e03f48aa35eb'

[pwolfram]

In [223]: type(ds.yParticle)
Out[223]: xarray.core.dataarray.DataArray

[pwolfram]

@shoyer, can you please clarify:

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?

I don't think this is entirely what you mean:

In [244]: x.compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-244-f4c69c9c1276> in <module>()
----> 1 x.compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'compute'

In [245]: x.array.compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-245-d906770c195f> in <module>()
----> 1 x.array.compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'compute'

In [246]: x.array.array.compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-246-a1bb7b5fa51a> in <module>()
----> 1 x.array.array.compute()

AttributeError: 'NetCDF4ArrayWrapper' object has no attribute 'compute'

In [247]: x.array.array.array.compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-247-975c3b146207> in <module>()
----> 1 x.array.array.array.compute()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getattr__ (netCDF4/_netCDF4.c:36798)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.getncattr (netCDF4/_netCDF4.c:34035)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._get_att (netCDF4/_netCDF4.c:4265)()

AttributeError: NetCDF: Attribute not found

or

In [249]: x.array.array.array.sum().compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-249-63306814fa5b> in <module>()
----> 1 x.array.array.array.sum().compute()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getattr__ (netCDF4/_netCDF4.c:36798)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.getncattr (netCDF4/_netCDF4.c:34035)()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._get_att (netCDF4/_netCDF4.c:4265)()

AttributeError: NetCDF: Attribute not found

In [250]: x.array.array.sum().compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-250-77cf68e730e3> in <module>()
----> 1 x.array.array.sum().compute()

AttributeError: 'NetCDF4ArrayWrapper' object has no attribute 'sum'

In [251]: x.array.sum().compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-251-2714bd5c5439> in <module>()
----> 1 x.array.sum().compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'sum'

In [252]: x.sum().compute()
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-252-7ce0149d8b09> in <module>()
----> 1 x.sum().compute()

AttributeError: 'LazilyIndexedArray' object has no attribute 'sum'

but hopefully this is helpful.

[shoyer]

One other thing to try: at what level of xr_var[.array[.array[.array]]], if any, does the dask array fail to compute?
I don't think this is entirely what you mean:

I mean, e.g.,

chunks = 1e7
da.from_array(xr_var, chunks=chunks, dtype=xr_var.dtype).sum().compute()
da.from_array(xr_var.array, chunks=chunks, dtype=xr_var.array.dtype).sum().compute()
da.from_array(xr_var.array.array, chunks=chunks, dtype=xr_var.array.array.dtype).sum().compute()

I can also make CharToStringArray serialized with dask-distributed on my xarray branch (though as I discussed with @mrocklin today, we will want a slightly different solution later).

[shoyer]

Try the latest version of my xarray branch which implements CharToStringArray serialization. If you're still getting the same error, I will be surprised!