normalize_total(target_sum=None) picks a different target for CSR than for dense — and can zero out the data

[alexdobin]

Please make sure these conditions are met

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of scanpy.
• (optional) I have confirmed this bug exists on the main branch of scanpy.

What happened?

normalize_total(target_sum=None) documents its target as the "median of total counts for observations", but _normalize_total_helper computes it two different ways depending on the input layout (src/scanpy/preprocessing/_normalization.py, line numbers from main):

if isinstance(x, CSRBase):
    ...
    if target_sum is None:
        target_sum = np.median(counts_per_cell)            # L105 — median over ALL cells
else:
    counts_per_cell = stats.sum(x, axis=1)
    ...
    if target_sum is None:
        target_sum = _compute_nnz_median(counts_per_cell)  # L117 — median over POSITIVE cells

So when any cell has zero total counts, CSR and dense inputs normalize the same data to different targets. CSC is .tocsr()-converted first (L267–268), so it follows the CSR rule.

The branches disagree only when a zero-total cell shifts the median — ties between the middle order statistics hide it (row sums [0, 10, 10, 20] give 10 either way), which is probably why this has gone unnoticed.

The severe case: when at least half the cells are empty, the CSR branch's median is 0, and dividing by it wipes out the matrix. This is a silent data-destruction path — it returns all zeros, with only two RuntimeWarnings to show for it:

case 2: row sums [0, 0, 10] -> X after normalize_total
  dense: [0. 0.  0. 0.  4. 6.]     # correct: target = 10, populated cell untouched
    csr: [0. 0.  0. 0.  0. 0.]     # target = 0 -> the only populated cell is destroyed

Empty cells are common enough in practice (an unfiltered raw matrix, an aggressive gene subset, a .raw slice) that this is reachable without doing anything unusual. normalize_total already anticipates them — it warns "Some cells have zero counts" and takes care to divide only positive cells. It just doesn't exclude them when choosing the target on the CSR path.

Which rule looks like the intended one?

The positive-cell median, for three reasons:

1. It is the historical behaviour. In 1.11.1 and earlier there was a single code path and it excluded zeros on both branches — np.median(counts_greater_than_zero, axis=0) and np.median(counts_per_cell[cell_subset]), with cell_subset = counts_per_cell > 0. The split arrived with the numba CSR kernel in normalize_total with numba #3571 (merged 2025-05-19; backported to 1.11.x as Backport PR #3571 on branch 1.11.x (normalize_total with numba) #3636), which preserved the old rule for the non-CSR branch as _compute_nnz_median and gave the new CSR branch a plain np.median. That reads like an oversight rather than a decision — neither the PR nor its release note mentions a semantic change.
2. _compute_nnz_median's own docstring says "compute the median of the non-zero counts".
3. It cannot produce a zero target as long as one cell is non-empty.

Suggested fix — one line, restoring the pre-1.11.2 semantics:

     if target_sum is None:
-        target_sum = np.median(counts_per_cell)
+        target_sum = _compute_nnz_median(counts_per_cell)

Happy to open that PR with a regression test if you agree on the direction. Whichever rule you prefer, it would help to state it in the target_sum docstring — "median of total counts for observations" doesn't currently settle it.

Affected versions

Introduced in 1.11.2 (2025-05-28) and present in every release since, including the latest 1.12.3, plus current main. 1.11.1 and earlier are unaffected. Verified by reading _normalization.py at tags 1.11.0, 1.11.1, 1.11.2, 1.11.5, 1.12.1, 1.12.3 and main, and by running the script below against both 1.12.1 and main.

Minimal code sample

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "scanpy@git+https://github.com/scverse/scanpy.git@main",
# ]
# ///
#
# Runs against scanpy's development branch.
#
# `normalize_total(target_sum=None)` picks its target over ALL cells for CSR
# input but over POSITIVE cells only for dense input, so the same data
# normalizes to two different targets when a zero-total cell is present.

import anndata as ad
import numpy as np
import scanpy as sc
import scipy.sparse as sp

# Case 1 — row sums [0, 10, 20, 30]. The zero-total cell shifts the median.
dense = np.array([[0.0, 0.0], [4.0, 6.0], [8.0, 12.0], [12.0, 18.0]], dtype=np.float32)

print("case 1: row sums [0, 10, 20, 30] -> per-cell totals after normalize_total")
for label, X in (("dense", dense.copy()), ("csr", sp.csr_matrix(dense))):
    a = ad.AnnData(X=X)
    sc.pp.normalize_total(a, target_sum=None)
    print(f"  {label:>5}: {np.asarray(a.X.sum(axis=1)).ravel()}")
print("  expected: dense target = median(10, 20, 30)    = 20")
print("            csr   target = median(0, 10, 20, 30) = 15")

# Case 2 — row sums [0, 0, 10]. The CSR median is 0, so the divide wipes the
# matrix: the only populated cell is destroyed.
dense2 = np.array([[0.0, 0.0], [0.0, 0.0], [4.0, 6.0]], dtype=np.float32)

print("\ncase 2: row sums [0, 0, 10] -> X after normalize_total")
for label, X in (("dense", dense2.copy()), ("csr", sp.csr_matrix(dense2))):
    a = ad.AnnData(X=X)
    sc.pp.normalize_total(a, target_sum=None)
    out = a.X if isinstance(a.X, np.ndarray) else a.X.toarray()
    print(f"  {label:>5}: {out.ravel()}")
print("  expected: dense keeps the populated cell (target = 10)")
print("            csr   returns all zeros        (target = 0)")

Output on main:

case 1: row sums [0, 10, 20, 30] -> per-cell totals after normalize_total
  dense: [ 0. 20. 20. 20.]
    csr: [ 0. 15. 15. 15.]

case 2: row sums [0, 0, 10] -> X after normalize_total
  dense: [0. 0. 0. 0. 4. 6.]
    csr: [0. 0. 0. 0. 0. 0.]

Error output

# No exception — the destructive case is silent apart from these:
.../scanpy/preprocessing/_normalization.py:119: RuntimeWarning: divide by zero encountered in divide
  counts_per_cell = counts_per_cell / target_sum
.../scanpy/preprocessing/_normalization.py:119: RuntimeWarning: invalid value encountered in divide
  counts_per_cell = counts_per_cell / target_sum

Versions

Details

scanpy      1.12.1
anndata     0.12.19
numpy       2.4.6
scipy       1.18.0
pandas      2.3.3
h5py        3.16.0
numba       0.65.1
scikit-learn 1.8.0
Python      3.12.11 (main, Jun  4 2025, 17:36:43) [Clang 20.1.4]
OS          Linux-5.15.0-176-generic-x86_64-with-glibc2.35

Also reproduced on scanpy @ main (git), Python 3.13.
Source-verified on 1.11.2 - 1.12.3 and main; see "Affected versions".

[flying-sheep]

Strange, how were you able to create an issue without issue type? We don’t allow creating blank issues and the “Bug” template should have added the “Bug” issue type.

Thanks for the report and straightforward fix!