Uh oh!
There was an error while loading. Please reload this page.
Select minor duplicate indices - #35
Merged
Merged
Conversation
`arr[:, [1, 1, 3]]` returns zeros where column 1's second copy should be. No error, no warning -- just wrong values, in a shape that looks right. v[:, [1, 1, 3]] want got [[ 2 2 4] [[ 0 2 4] [ 6 6 8] [ 0 6 8] [10 10 12]] [ 0 10 12]] This is a regression. Before #23 the same expression fell through to `to_scipy()[...]`, and scipy fans duplicate fancy indices out correctly; verified against 45ce15d, where it returns the right answer. The cause is structural rather than an off-by-one: `_select_minor` builds a single old -> new lookup array, remap = np.full(self.n_minor, -1); remap[idx] = np.arange(len(idx)) and a repeated index writes that slot twice, so only its last destination survives. One array cannot express one-to-many, so the fix is to invert the selection instead: `fanout`/`offsets`/`positions` give, for each original minor index, every output position it maps to, and each stored element emits one entry per destination. Two passes (count, then fill) via `_ops.minor_select_counts`/`minor_select_fill`, so each surviving slot writes a disjoint range and the passes parallelize. Deriving each major slice's slot count with `np.searchsorted` on `major_ptr` also drops the `np.repeat(np.arange(n_unique), ...)` this used to build, which was `nnz`-sized. Peak allocation for a 2e6-nonzero selection falls from 62.27 MB to 4.78 MB, so this closes ISSUE-30 as well -- not a separate concern, just what the correct implementation happens not to need. All 1066 tests from main pass unchanged. New coverage coming from the fan-out semantics: five duplicate patterns, duplicates on either axis and both at once, 25 random selections with repeats compared against scipy directly, plus the selections that already worked (sorted, reversed, empty, negative, slices, boolean masks) so the fix can't silently narrow them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The fan-out explanation moves from a block comment into short notes inside the kernels. Duplicate-pattern sweep drops to three cases and the type/dtype assertions go, since construction guarantees them.
aarmey
approved these changes
Sep 4, 2026
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix silent data loss when a minor-axis selection repeats an index
What's wrong
arr[:, [1, 1, 3]]returns zeros where column 1's second copy should be. Just wrong values, in a shape that looks right.This is a regression introduced by #23. Before it, the same expression fell through to
to_scipy()[...], and scipy fans duplicate fancy indices out correctly. Verified directly against45ce15d, where it returns the right answer.Duplicate indices aren't exotic here: any resampling-with-replacement, any bootstrap over cells or genes, any join that maps several output columns onto one source column produces them. The failure is silent, so it surfaces as a downstream result being subtly wrong rather than as a crash.
Why it isn't a small patch
The cause is structural.
_select_minorbuilds one old → new lookup array:A repeated index writes that slot twice and only its last destination survives. A single array can't express one-to-many, so no amount of care around this formulation fixes it.
The fix inverts the selection instead.
fanout/offsets/positionsgive, for each original minor index, every output position it maps to; each stored element then emits one entry per destination.It also removes an nnz-sized temporary
Deriving each major slice's slot count with
np.searchsortedonmajor_ptrdrops thenp.repeat(np.arange(n_unique), ...)the old version needed, which wasnnz-sized. Peak allocation for a 2e6-nonzero selection: