Avoid loading entire dataset by getting the nbytes in an array

[hmaarrfk]

Using .data accidentally tries to load the whole lazy arrays into memory.

Sad.

• Closes #xxxx
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[hmaarrfk]

I personally do not even think the hasattr is really that useful. You might as well use size and itemsize

[hmaarrfk]

Looking into the history a little more. I seem to be proposing to revert:
60f8c3d

I think this is important since many users have arrays that are larger than memory. For me, I found this bug when trying to access the number of bytes in a 16GB dataset that I'm trying to load on my wimpy laptop. Not fun to start swapping. I feel like others might be hitting this too.

xref:
#6797
#4842

[hmaarrfk]

I think that at the very lease, the current implementation works as well as the old one for arrays that are defined by the sparse package.

[hmaarrfk]

It seems that checking hasattr on the _data variable achieves both purposes.

[Illviljan]

Is that test targetting your issue with RAM crashing the laptop? Shouldn't there be some check if the values were loaded?

How did you import your data? self.data looks like this:

xarray/xarray/core/variable.py

Lines 420 to 435 in ed60c6c

	@property
	defdata(self) ->Any:
	"""
	The Variable's data as an array. The underlying array type
	(e.g. dask, sparse, pint) is preserved.
	See Also
	--------
	Variable.to_numpy
	Variable.as_numpy
	Variable.values
	"""
	ifis_duck_array(self._data):
	returnself._data
	else:
	returnself.values

I was expecting your data to be a duck_array?

[hmaarrfk]

No explicit test was added to ensure that the data wasn't loaded. I just experienced this bug enough (we would accidentally load 100GB files in our code base) that I knew exactly how to fix it.

If you want i can add a test to ensure that future optimizations to nbytes do not trigger a data load.

I was hoping the 1 line fix would be a shoe in.

[hmaarrfk]

The data is loaded from an NetCDF store through open_dataset

[Illviljan]

I'm not really opposed to this change, shape and dtype uses self._data aswell.

Without using chunks={} in open_dataset? I just find it a little odd that it's not a duck_array, what type is self._data?

This test just looked so similar to the tests in #6797. I think you can do a similar lazy test taking inspiration from:

xarray/xarray/tests/test_formatting.py

Lines 715 to 727 in ed60c6c

	deftest_lazy_array_wont_compute() ->None:
	fromxarray.core.indexingimportLazilyIndexedArray
	classLazilyIndexedArrayNotComputable(LazilyIndexedArray):
	def__array__(self, dtype=None):
	raiseNotImplementedError("Computing this array is not possible.")
	arr=LazilyIndexedArrayNotComputable(np.array([1, 2]))
	var=xr.DataArray(arr)
	# These will crash if var.data are converted to numpy arrays:
	var.__repr__()
	var._repr_html_()

[hmaarrfk]

Very smart test!

[hmaarrfk]

Yes, without chunks of anything

[dcherian]

LGTM. thanks!

[hmaarrfk]

👍🏾

[hmaarrfk]

Any chance of a release, this is quite breaking for large datasets that can only be out of memory.

[TomNicholas]

This came up in the xarray office hours today, and I'm confused why this PR made any difference to the behavior at all? The .data property just points to ._data, so why would it matter which one we check?

[dcherian]

Because we have lazy data reading functionality

importxarrayasxrds=xr.tutorial.open_dataset("air_temperature")
var=ds.air.variableprint(type(var._data)) # memory cached arrayprint(type(var._data.array.array)) # ah that's wrapping a lazy array, no data read in yetprint(var._data.size) # can access sizeprint(type(var._data.array.array)) # still a lazy array#.data forces a disk loadprint(type(var.data)) # oops disk-loadprint(type(var._data)) # "still memory cached array"print(type(var._data.array.array)) # but that's wrapping numpy data in memory

<class 'xarray.core.indexing.MemoryCachedArray'>
<class 'xarray.core.indexing.LazilyIndexedArray'>
3869000
<class 'xarray.core.indexing.LazilyIndexedArray'>
<class 'numpy.ndarray'>
<class 'xarray.core.indexing.MemoryCachedArray'>
<class 'numpy.ndarray'>