Skip to content

Create a statistics sidecar as an on-disk format feature #26

Description

@fishidaho

This is somewhat load-bearing for #25's zero-read filtering gate

_rapid_load._cell_totals already proves the core trick works: a cell's total count is computable from major_ptr/values/value_ptr alone —
far fewer entries than nnz — without ever touching the packed indices stream. This generalizes: obs/total_counts and n_genes_by_counts-style per-cell/per-gene statistics can be derived the same way, and a two-predicate QC filter built on them touches X not at all.

Today that only works incidentally, when a particular file happens to carry scanpy-computed QC columns in obs/var — not because vsparse guarantees or maintains them. There's no first-class mechanism for #25 's selection algebra to rely on a sidecar being present, current, or complete.

Proposed scope

  • Define a sidecar schema (per-cell and per-gene statistics — at minimum total counts, n-genes/n-cells-by-counts, matching what QC filtering typically needs) as a designed, versioned part of the write path, not something derived ad hoc.
  • Writers populate it automatically on write_h5ad/write_zarr; readers can trust it's present and correct for files written by this version, and degrade gracefully (recompute) for older files that don't have it.
  • Wire Design a composable lazy Selection object for VCSCAnnData #25's Selection object to serve obs/var-driven filters from the sidecar when available, entirely without a data read.

Acceptance criteria

  • Every file written by write_h5ad/write_zarr carries the sidecar by default.
  • Sidecar values match recomputation exactly.
  • A Selection filter expressible purely in sidecar terms performs zero reads against X.

References

  • src/vsparse/_rapid_load.py (_cell_totals).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions