You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is somewhat load-bearing for #25's zero-read filtering gate
_rapid_load._cell_totals already proves the core trick works: a cell's total count is computable from major_ptr/values/value_ptr alone —
far fewer entries than nnz — without ever touching the packed indices stream. This generalizes: obs/total_counts and n_genes_by_counts-style per-cell/per-gene statistics can be derived the same way, and a two-predicate QC filter built on them touches X not at all.
Today that only works incidentally, when a particular file happens to carry scanpy-computed QC columns in obs/var — not because vsparse guarantees or maintains them. There's no first-class mechanism for #25 's selection algebra to rely on a sidecar being present, current, or complete.
Proposed scope
Define a sidecar schema (per-cell and per-gene statistics — at minimum total counts, n-genes/n-cells-by-counts, matching what QC filtering typically needs) as a designed, versioned part of the write path, not something derived ad hoc.
Writers populate it automatically on write_h5ad/write_zarr; readers can trust it's present and correct for files written by this version, and degrade gracefully (recompute) for older files that don't have it.
This is somewhat load-bearing for #25's zero-read filtering gate
_rapid_load._cell_totalsalready proves the core trick works: a cell's total count is computable frommajor_ptr/values/value_ptralone —far fewer entries than nnz — without ever touching the packed
indicesstream. This generalizes:obs/total_countsandn_genes_by_counts-style per-cell/per-gene statistics can be derived the same way, and a two-predicate QC filter built on them touchesXnot at all.Today that only works incidentally, when a particular file happens to carry scanpy-computed QC columns in
obs/var— not becausevsparseguarantees or maintains them. There's no first-class mechanism for #25 's selection algebra to rely on a sidecar being present, current, or complete.Proposed scope
write_h5ad/write_zarr; readers can trust it's present and correct for files written by this version, and degrade gracefully (recompute) for older files that don't have it.Selectionobject to serve obs/var-driven filters from the sidecar when available, entirely without a data read.Acceptance criteria
write_h5ad/write_zarrcarries the sidecar by default.Selectionfilter expressible purely in sidecar terms performs zero reads againstX.References
src/vsparse/_rapid_load.py(_cell_totals).