Metadata compression

[fishidaho]

Categorical-encode obs/var strings on write, as anndata's writers do

Background

numeric_only_compression (src/vsparse/_compression.py) exists because Blosc2 crashes on variable-length-string HDF5 datasets, so every string column in obs/var is written uncompressed. That left the package with no size strategy for metadata at all, and metadata is a large fraction of a written file.

This started as an investigation into whether a different codec was safe for strings, but it turned up a plain bug instead.

The bug

anndata.AnnData.write_h5ad converts string columns to categoricals before writing — that's its own convert_strings_to_categoricals=True parameter. VCSCAnnData writes field-by-field rather than delegating to anndata's writer and so never did this.

A low-cardinality annotation — cell type, sample ID, batch — therefore landed as one variable-length string per row uncompressed, and unrecoverable by any later step. Measured on 200k cells × 2k genes with three such columns:

before (plain strings) file 41.72 MB
after (categorical) file 13.48 MB

3.1× off the whole file, from a parameter the parent class has had all along.

What this changes

write_h5ad and write_zarr gain convert_strings_to_categoricals: bool = True. Note it mutates obs/var in place, exactly as anndata's writers do. Pass False to opt out.

The codec question, answered

A 20k-element vlen string dataset written under each available filter, one subprocess per filter, against h5py 3.16.0 / HDF5 2.0.0 / hdf5plugin 7.0.0:

filter	result
none	fine
gzip	fine
lzf	fine
blosc2	dies with SIGFPE before returning