Add support for fancy indexing on get/setitem

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!

[jni]

Addresses #657

This matches NumPy behaviour in that basic, boolean, and vectorized integer (fancy) indexing are all accessible from __{get,set}item__. Users still have access to all the indexing methods if they want to be sure to use only basic indexing (integer + slices).

I'm not 100% sure about the approach, but it seemed much easier to use a try/except than to try to detect all the cases when fancy indexing should be used. Happy to hear some guidance about how best to arrange that.

I still need to update docstrings + docs, will do that now — thanks for the checklist below. 😂

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[pep8speaks]

Hello @jni! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-10-19 19:51:41 UTC

[jni]

Questions for maintainers:

• how to update release.rst
• do I need to update any of the notebooks? git grep "unsupported selection item for basic indexing" has some hits in there.
• have I missed anything?

Thanks! 😊 Here's a picture of me painting into a zarr array with napari. 😃

[joshmoore]

• how to update release.rst

Add a commit like 03dce69 to this PR?

[jni]

But is the next version 2.8.2 or 2.9? =) Or 3.0! Surely this momentous change merits such an upgrade. 😂

[joshmoore]

🙄 (:smile:)

I'd think 2.8.2 but could get behind 2.9 as well.

[codecov]

Codecov Report

Merging #725 (cf9051c) into master (4f8cb35) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head cf9051c differs from pull request most recent head b9c8f59. Consider uploading reports for the commit b9c8f59 to get more accurate results

@@ Coverage Diff @@## master #725 +/- ##
=======================================
Coverage 99.94% 99.94% =======================================
Files 31 31 Lines 10936 10986 +50 =======================================
+ Hits 10930 10980 +50 
Misses 6 6 

Impacted Files	Coverage Δ
zarr/core.py	100.00% <100.00%> (ø)
zarr/indexing.py	100.00% <100.00%> (ø)
zarr/tests/test_indexing.py	100.00% <100.00%> (ø)

[jakirkham]

Yeah 2.9 makes sense. It's a new feature to some extent

Am hoping that 3.0 will also be when we are using the v3 spec 😉

[jni]

Ok, all done. I used 2.9.0 based on @jakirkham's logic that enhancements belong in minor version bumps, not patch. This should be ready for a full review now. 🙏

[jni]

kind ping! 🙏 Just rebased on latest master.

[joshmoore]

Looking at #657, @shoyer, any thoughts?

[joshmoore]

import numpy as np
import pytest as pt
import zarr as zr
@pt.fixture
def na():
return np.arange(60).reshape((3, 4, 5))
@pt.fixture
def za(na):
return zr.array(data=na)
def test_base(na, za):
np.equal(na, za)
def test_base(na, za):
np.equal(na[0], za[0])
np.equal(na[1], za[1])
np.equal(na[2, 3], za[2, 3])
@pt.mark.skip
def test_integer_array(na, za):
print(na[[0, 0]], za[[0, 0]])
def test_int_arr_arr(na, za):
x = [0, 0, 0]
y = [1, 1, 1]
z = [2, 2, 2]
np.equal(na[x, y, z], za[x, y, z])
def test_simple(na, za):
np.equal(na[0, 1, [2, 3]], za[0, 1, [2, 3]])
np.equal(na[[0, 1], 1, [2, 3]], za[[0, 1], 1, [2, 3]])
@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

Assuming there are no objections by tomorrow, I'm inclined to move forward with this.

[shoyer]

I don't think Zarr should fall back to using vindex. There are some cases where the dimension order from vindex differs from NumPy, so this could perpetuate exactly the sort of confusion we were trying to solve with vindex :).

We could definitely leverage vindex for normal array indexing, but we should be sure reorder dimensions in a way that is consistent with NumPy.

[joshmoore]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

[shoyer]

There are some cases where the dimension order from vindex differs from NumPy,...

Thanks @shoyer. Do you have an example handy? And does this count as a reversal of #657 (comment) ?

This is probably the canonical edge case:
https://numpy.org/neps/nep-0021-advanced-indexing.html#mixed-indexing

I think I'm being consistent with my previous suggestion to "copy NumPy's behavior for fancy indexing" here :). It's just important to recognize that vindex is not consistent with NumPy's behavior for fancy indexing, in the case where slices and arrays are mixed.

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

Or we could support mixed slice/array indexing, but only if we're careful to re-order array dimensions the same way that NumPy does.

[joshmoore]

That matches:

@pt.mark.skip
def test_slicing(na, za):
np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])

right? which if I remove the skip fails with

IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>

full run

(base) /tmp $pytest -svk slicing t.py
========================================================= test session starts =========================================================
platform darwin -- Python 3.7.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp
plugins: testinfra-3.4.0, napari-plugin-engine-0.1.7
collected 5 items / 4 deselected / 1 selected
t.py::test_slicing FAILED
============================================================== FAILURES ===============================================================
____________________________________________________________ test_slicing _____________________________________________________________
na = array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]... [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
za = <zarr.core.Array (3, 4, 5) int64>
def test_slicing(na, za):
> np.equal(na[1:, 2:, [3, 4]], za[1:, 2:, [3, 4]])
t.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:571: in __getitem__
return self.get_basic_selection(selection, fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:697: in get_basic_selection
fields=fields)
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/core.py:737: in _get_basic_selection_nd
indexer = BasicIndexer(selection, self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <zarr.indexing.BasicIndexer object at 0x7fc3c8217950>, selection = (slice(1, None, None), slice(2, None, None), [3, 4])
array = <zarr.core.Array (3, 4, 5) int64>
def __init__(self, selection, array):
# handle ellipsis
selection = replace_ellipsis(selection, array._shape)
# setup per-dimension indexers
dim_indexers = []
for dim_sel, dim_len, dim_chunk_len in \
zip(selection, array._shape, array._chunks):
if is_integer(dim_sel):
dim_indexer = IntDimIndexer(dim_sel, dim_len, dim_chunk_len)
elif is_slice(dim_sel):
dim_indexer = SliceDimIndexer(dim_sel, dim_len, dim_chunk_len)
else:
raise IndexError('unsupported selection item for basic indexing; '
'expected integer or slice, got {!r}'
> .format(type(dim_sel)))
E IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
/usr/local/anaconda3/lib/python3.7/site-packages/zarr/indexing.py:285: IndexError
======================================================= short test summary info =======================================================
FAILED t.py::test_slicing - IndexError: unsupported selection item for basic indexing; expected integer or slice, got <class 'list'>
=================================================== 1 failed, 4 deselected in 4.60s ===================================================
(base) /tmp $

[jni]

So as long as that edge case is avoided, we could support "array only" fancy indexing. (This would probably be a good place to start.)

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

[jni]

or should I just check that there are no instances of slice in the indexing tuple?

[shoyer]

or should I just check that there are no instances of slice in the indexing tuple?

NumPy says "When there is at least one slice (:), ellipsis (...) or newaxis in the index (or the array has more dimensions than there are advanced indexes)"
https://numpy.org/doc/stable/reference/arrays.indexing.html#combining-advanced-and-basic-indexing

So this is the case to avoid.

[shoyer]

@shoyer I didn't really know the best way to "check" that we were in "array only" fancy indexing mode, since arrays and scalars can be broadcast together. Also, are lists supported?

From NumPy's perspective:

• lists are arrays
• scalars are arrays if doing "mixed" or "advanced indexing" (if there is at least one other array)

[jni]

Ok I think I've caught that case and raised an appropriate error then. Let me know if that meets requirements now! 🤞

[shoyer]

another case would worth checking would be something like:

 z = zarr.zeros((20, 20, 20))
with pytest.raises(IndexError):
z[[1, 2, 3], 0] = 2

This doesn't look like mixed indexing but it actually is because of the implicit slice at the end.

[jni]

great point! 😬 Will work on this.

[shoyer]

I think this should probably have the same check you put in __setitem__?

[shoyer]

I think you probably have to "expand" the indexer to a tuple with length equal to the number of array dimensions (i.e., by replacing Ellipsis and padding by slice(None)) in order to determine if vindex will work.

[jni]

Thanks @shoyer, very good point. Is there a function to do this already in zarr?

[jni]

actually, how about just checking that the length of the tuple matches the dimension of the array?

[shoyer]

Here's a helper I wrote for Xarray:
https://github.com/pydata/xarray/blob/234b40a37e484a795e6b12916315c80d70570b27/xarray/core/indexing.py#L31

Conceivably you could copy it into Zarr as long as you are compliant with the Apache 2.0 license.

[shoyer]

There may well be something like this in Zarr already, I'm not too familiar with the Zarr codebase here.

[jni]

Ok, take 3. 😂 This time I went with an explicit check for fancy indexing rather than a fallback, now that I understand the exact requirements better.

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

In [6]: arr=zarr.zeros((5, 5))
In [7]: %timeitarr[0, 1] =4157µs ± 5.87µsperloop (mean ± std. dev. of7runs, 10000loopseach)
In [8]: idx.is_pure_fancy_indexing((0, 1), 2)
Out[8]: FalseIn [9]: %timeitidx.is_pure_fancy_indexing((0, 1), 2)
8.42µs ± 430nsperloop (mean ± std. dev. of7runs, 100000loopseach)

Two other questions:

1. It turns out that zarr.indexing.is_integer(x) returns True for bools. Is this desired/expected? Or should I add a check to exclude them explicitly?
2. For lists, I have only added a check that the first element is an int, since I don't think we want a check to take O(n) time. Alternatives welcome.

[jni]

not sure why the CI isn't running, too... 🤷

[jni]

nvmd just read the notice on top of the checks. 🤦

[jni]

Ok I've rebased on master. The failure in the previous CI run was:

assert a[42] == z[np.uint64(42)] in TestArrayWithFSStorePartialRead.test_array_1d

________________ TestArrayWithFSStorePartialRead.test_array_1d _________________
self = <zarr.tests.test_core.TestArrayWithFSStorePartialRead testMethod=test_array_1d>
deftest_array_1d(self):
a = np.arange(1050)
z =self.create_array(shape=a.shape, chunks=100, dtype=a.dtype)
# check propertiesassertlen(a) ==len(z)
assert a.ndim == z.ndim
assert a.shape == z.shape
assert a.dtype == z.dtype
assert (100,) == z.chunks
assert a.nbytes == z.nbytes
assert11== z.nchunks
assert0== z.nchunks_initialized
assert (11,) == z.cdata_shape
# check empty
b = z[:]
assertisinstance(b, np.ndarray)
assert a.shape == b.shape
assert a.dtype == b.dtype
# check attributes
z.attrs['foo'] ='bar'assert'bar'== z.attrs['foo']
# set data
z[:] = a
# check propertiesassert a.nbytes == z.nbytes
assert11== z.nchunks
assert11== z.nchunks_initialized
# check slicing
assert_array_equal(a, np.array(z))
assert_array_equal(a, z[:])
assert_array_equal(a, z[...])
# noinspection PyTypeChecker
assert_array_equal(a, z[slice(None)])
assert_array_equal(a[:10], z[:10])
assert_array_equal(a[10:20], z[10:20])
assert_array_equal(a[-10:], z[-10:])
assert_array_equal(a[:10, ...], z[:10, ...])
assert_array_equal(a[10:20, ...], z[10:20, ...])
assert_array_equal(a[-10:, ...], z[-10:, ...])
assert_array_equal(a[..., :10], z[..., :10])
assert_array_equal(a[..., 10:20], z[..., 10:20])
assert_array_equal(a[..., -10:], z[..., -10:])
# ...across chunk boundaries...
assert_array_equal(a[:110], z[:110])
assert_array_equal(a[190:310], z[190:310])
assert_array_equal(a[-110:], z[-110:])
# single itemassert a[0] == z[0]
assert a[-1] == z[-1]
# unusual integer itemsassert a[42] == z[np.int64(42)]
assert a[42] == z[np.int32(42)]
> assert a[42] == z[np.uint64(42)]
zarr/tests/test_core.py:212: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ zarr/core.py:675: in __getitem__
result =self.vindex[selection]
zarr/indexing.py:818: in __getitem__
returnself.array.get_coordinate_selection(selection, fields=fields)
zarr/core.py:1032: in get_coordinate_selection
out =self._get_selection(indexer=indexer, out=out, fields=fields)
zarr/core.py:1141: in _get_selection
self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
zarr/core.py:1868: in _chunk_getitems
self._process_chunk(
zarr/core.py:1752: in _process_chunk
index_selection = PartialChunkIterator(chunk_selection, self.chunks)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <zarr.indexing.PartialChunkIterator object at 0x7f1ef83bec70>
selection = [slice(42, 43.0, 1)], arr_shape = (100,)
def__init__(self, selection, arr_shape):
selection = make_slice_selection(selection)
self.arr_shape = arr_shape
# number of selection dimensions can't be greater than the number of chunk dimensionsiflen(selection) >len(self.arr_shape):
raiseValueError(
"Selection has more dimensions then the array:\n"f"selection dimensions = {len(selection)}\n"f"array dimensions = {len(self.arr_shape)}"
)
# any selection can not be out of the range of the chunk
> selection_shape = np.empty(self.arr_shape)[tuple(selection)].shape
E TypeError: slice indices must be integers or None or have an __index__ method
zarr/indexing.py:958: TypeError

which I can't reproduce locally:

 $ pytest zarr/tests/test_core.py::TestArrayWithFSStorePartialRead
============================= test session starts ==============================
platform darwin -- Python 3.8.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
PySide2 5.14.2.1 -- Qt runtime 5.14.2 -- Qt compiled 5.14.2
rootdir: /Users/jni/projects/zarr-python, configfile: pytest.ini
plugins: timeout-1.3.4, localserver-0.5.0, napari-plugin-engine-0.1.9, hypothesis-6.0.0, mypy-plugins-1.6.1, ordering-0.6, order-0.11.0, qt-3.3.0, cov-2.8.1, napari-0.4.8.dev133+g46f295b5
collected 48 items
zarr/tests/test_core.py ................................................ [100%]

I hope this run passes but if not, any suggestions about what is going on are most welcome!

[jni]

... Nope. 😭 Anyone got any ideas?

[jni]

@joshmoore 🎉 🎉 🎉 Probably should have done this a while back. 😂 I'll revert the pytest commits and then hopefully 🤞 this can go in?

[jni]

... Or maybe we want to keep the timeouts? They're kinda handy.

[joshmoore]

Latest changes & test status are looking good. Also bubbling up #725 (comment) in case there were any opinions:

For the curious, the if-statement adds 8µs to the indexing operation, which normally takes about 150µs at least (that's when everything is in-memory and we are grabbing a single value), for a ~5% worst-case slowdown. I hope that's acceptable.

[joshmoore]

Note from zulip: this fixed the weird timeout issue, though we still don't know why the deadlock was platform specific.

[joshmoore]

Merging now that 2.10.2 is released (fixing #840). I'll look into getting a 2.11rc1 released soon.

[jni]

🎉:rocket:!!!!