Skip to content

Variable.load() is a no-op for numpy data; can skip dispatch entirely #8

Description

@FBumann

What

For a Variable whose ._data is already an in-memory numpy.ndarray, Variable.load() does no useful work — it walks a dispatch chain that always returns the same array unchanged. The cost is paid once per variable inside Dataset.load() / DataArray.load(), on top of (and after) is_chunked_array has already concluded the variable isn't chunked.

Code path

Variable.load (xarray/core/variable.py:1022):

defload(self, **kwargs) ->Self:
self._data=to_duck_array(self._data, **kwargs)
returnself

to_duck_array (xarray/namedarray/pycompat.py:139):

defto_duck_array(data, **kwargs):
ifis_chunked_array(data):
...
ifisinstance(data, ExplicitlyIndexed|ImplicitToExplicitIndexingAdapter):
returndata.get_duck_array()
elifis_duck_array(data):
returndataelse:
returnnp.asarray(data)

For np.ndarray:

  1. is_chunked_array(data)False
  2. isinstance(data, ExplicitlyIndexed | ImplicitToExplicitIndexingAdapter)False (np.ndarray is neither)
  3. is_duck_array(data)True → returns data unchanged

So self._data = to_duck_array(self._data) reduces to self._data = self._data. The entire call is dispatch overhead.

Why it matters

Dataset.load() calls Variable.load() per variable. The same dispatch is also reached implicitly by:

  • Dataset.compute() / DataArray.compute()dataset.py:803, dataarray.py:1247
  • xr.load_dataset() / xr.load_dataarray() / xr.load_datatree()backends/api.py:166,192,218
  • DataArray.persist()dataarray.py:1173
  • Some backend writer paths — backends/writers.py:768

On many-variable datasets — common with open_mfdataset / concat results — this overhead compounds. (Note: arithmetic does not call .load(). Lazy stays lazy. .values, .to_dataframe(), .plot(), repr etc. materialize data via to_duck_array directly without going through Variable.load, so they are unaffected by this issue.)

Proposed fix

Add a numpy fast-path at the top of Variable.load() and Variable.load_async():

defload(self, **kwargs) ->Self:
ifisinstance(self._data, np.ndarray):
returnselfself._data=to_duck_array(self._data, **kwargs)
returnself

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions