Skip to content

Normalized view stats - #31

Merged
aarmey merged 6 commits into
mainfrom
fix/normalized-view-stats
Sep 4, 2026
Merged

Normalized view stats#31
aarmey merged 6 commits into
mainfrom
fix/normalized-view-stats

Conversation

@fishidaho

Copy link
Copy Markdown
Contributor

Make post-selection normalization explicit and reachable

What's wrong

A normalized view computes row_scale/gene_scale/col_mean once, at construction, over the whole array it wrapped
(NormalizedViewBase.__init__, src/vsparse/_norm_common.py). __getitem__ then applies those parent statistics to whatever sub-block is asked for. On a mixed population — two cell types differing in both read depth and marker-gene expression — indexing the view for one cell type differs from normalizing those cells by 52% relative
Frobenius norm
:

view-then-index vs normalize(subset): rel Frobenius = 52.2%
index-then-normalize vs normalize(subset): rel Frobenius = 1.90e-15

Read as "the normalized data for these cells", this silently fits downstream analysis (parafac2, PCA, anything) to different data than the caller believes.

What this changes

Rather than redefine what indexing returns, this keeps both operations and gives them names:

  • __getitem__ stays a window into this view's matrix — exactly toarray()[key] without materializing the full matrix — and now says so, including what it is not.
  • select(rows, cols) is new: it renormalizes the selected sub-array on its own terms, equivalent to arr[sel].normalized(). It returns a view, not a dense block, so it still composes with @ and toarray().

select also fills a real gap rather than just wrapping something that already worked: selecting rows of a VCSCArray is a minor-axis selection, which falls out of _VCSBase.__getitem__ as a scipy array with no .normalized() at all. The raw-array route isn't uniformly available; select is.

Verification

tests/test_norm_selection.py (new, 22 cases across both array types):

  • select() matches normalizing the selection directly to < 1e-12 relative error, on the mixed-population fixture;
  • select() == the arr[sel].normalized() route where that route exists;
  • select() returns a composable view — sub @ B matches the dense reference;
  • column-only and both-axes selection (the two-index-array case selects a sub-block, not a broadcast diagonal — that was a real bug in the first draft, caught here);
  • select() with no arguments == the whole view;
  • __getitem__ == toarray()[key], pinning the window contract;
  • the two disagree by > 10% on a realistic selection — so the distinction can't be quietly collapsed by a later change.

fishidahoand others added 3 commits September 2, 2026 17:04
A normalized view's statistics are computed once, at construction, over the
whole array it wrapped. `__getitem__` then applies those parent statistics
to whatever sub-block is asked for -- while its docstring described the
result as though the block were self-consistently normalized ("the
transform/centering formula is then applied to that small block directly").
It isn't, and the gap is not small. On a mixed population -- two cell types
differing in read depth and marker expression -- indexing the view for one
cell type differs from normalizing those cells by **52% relative Frobenius
norm**. Read as "the normalized data for these cells", that silently fits
downstream analysis to different data than the caller believes.
Rather than redefine what indexing returns, this keeps the two operations
and names them:
- `__getitem__` stays a window into this view's matrix -- exactly
`toarray()[key]` without materializing the full matrix -- and now says so,
including what it is *not*.
- `select(rows, cols)` is new: it renormalizes the selected sub-array on its
own terms, equivalent to `arr[sel].normalized()`, and returns a view so it
still composes with `@`/`toarray()`.
Recomputation was deliberately not made the default for indexing, because
it is not uniformly more correct: `row_scale` is a per-cell total over the
columns present, so recomputing after a *gene* selection re-derives read
depth from just those genes, which is usually wrong. Making the caller ask
for it keeps that choice explicit; `select`'s docstring spells the caveat
out. Choosing per-selection semantics properly is v0.4's selection-algebra
work.
`select` also fills a real gap: selecting rows of a VCSCArray is a
minor-axis selection, which drops out as scipy with no `.normalized()` at
all, so the raw-array route isn't uniformly available.
Tests pin both contracts against a dense reference, and assert the two
disagree by >10% on a realistic selection -- so the distinction can't be
quietly collapsed by a later change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#23 changed what a raw array's __getitem__ returns underneath this branch,
in two ways that matter here:
- A minor-axis selection now stays VCS-native instead of falling out as a
scipy array. `select`'s re-wrap is therefore no longer the normal path,
just a defensive one -- reworded to match the same defensive re-wrap in
`_anndata_class._subset_2d`, and the test comment claiming a VCSCArray row
selection "has no .normalized() at all" is simply no longer true.
- `arr[rows, cols]` with two index arrays now composes a major- and a
minor-axis selection, which is the sub-block semantics `select` wants. It
previously fell through to scipy, which broadcasts two index arrays
pointwise, and that's why this was written one axis at a time. That
workaround is now redundant, so it collapses to a single index operation.
Adds a test pinning the outer-vs-pointwise distinction directly, using row
and column selections of different lengths -- a pointwise broadcast would
fail outright rather than return something subtly wrong, which is what makes
this worth keeping a test on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@fishidaho
fishidaho marked this pull request as draft September 3, 2026 21:13
Docstrings state what the code does. The two both-axes selection tests fold
into one that uses different-length keys, which is what actually pins outer
semantics.
@fishidaho
fishidaho marked this pull request as ready for review September 4, 2026 00:25
@fishidahofishidaho self-assigned this Sep 4, 2026
@aarmey
aarmey merged commit 31fbbc6 into mainSep 4, 2026
3 checks passed
@aarmey
aarmey deleted the fix/normalized-view-stats branch September 4, 2026 01:51
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@fishidaho@aarmey