Feat/zarr store class - #1
Closed
selmanozleyen wants to merge 49 commits into
Closed
Conversation
selmanozleyen
marked this pull request as ready for review
April 17, 2026 10:53
selmanozleyenforce-pushed
the
feat/zarr-store-class
branch
from
April 17, 2026 12:07
f105f5f to
e6fba59Comparefor more information, see https://pre-commit.ci
This reverts commit 2c4a579.
Dask's task-graph shape changed in PR scverse#1006 ("unpinning dask", commit 53b9438): parquet reads are now Task objects wrapping ``_fragment_to_table`` with a ``FragmentWrapper`` in kwargs or inside fused-expression subgraphs, instead of the legacy dicts with a ``piece`` tuple. This broke ``_search_for_backing_files_recursively`` in two ways: 1. ``"piece" in v.args[0]`` raised ``TypeError: argument of type 'Task' is not iterable`` before the fallback branch ever ran -- affecting every test that writes+reads points (``test_points``, ``test_roundtrip[points]``, ``test_io_and_lazy_loading_points``). 2. Fused expressions use key prefix ``readparquetpyarrowfs-fused-values-*`` (not ``read_parquet-*``) with the FragmentWrapper nested inside lists of tuples inside a subgraph dict, so even after fixing (1) the parquet file was never discovered -- affecting ``test_self_contained``. Replace the ad-hoc ``v.args[0]["piece"]`` / ``v.args[0].values()`` logic with a uniformly recursive helper ``_extract_parquet_paths_from_task`` that walks Mappings, Sequences, ``.args`` and ``.kwargs``, detecting FragmentWrapper via the ``.fragment.path`` attribute chain (no private dask_expr import) and still validating the legacy ``piece`` tuple shape for backward compatibility. Broaden the outer key-prefix match to any key containing "parquet"; ``.endswith(".parquet")`` inside the extractor keeps false positives out. Validated: 130 passed / 1 failed on tests/io/test_readwrite.py + test_store.py + test_store_abstractions.py (up from 113 / 18 on baseline; the one remaining failure is the intentionally-exposed consolidated- metadata-on-read gap, unrelated to this change). Made-with: Cursor
updates: - [github.com/rbubley/mirrors-prettier: v3.8.1 → v3.8.3](rbubley/mirrors-prettier@v3.8.1...v3.8.3) - [github.com/pre-commit/mirrors-mypy: v1.19.1 → v1.20.2](pre-commit/mirrors-mypy@v1.19.1...v1.20.2) - [github.com/astral-sh/ruff-pre-commit: v0.15.2 → v0.15.12](astral-sh/ruff-pre-commit@v0.15.2...v0.15.12) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* improve CI matrix job names Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix zarr v3 URL-parsing truncating element names with special chars Pass Path() to zarr.open in shapes, points, and table readers so zarr v3 does not URL-parse the store path (which strips everything after '#'). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…#1115) * fix: workaround for KeyError 'ome' when writing multiscale with processes scheduler (scverse#1024) Re-open the zarr group after da.compute() in _write_raster_datatree so that the main process picks up metadata written by ome-zarr-py's delayed write_multiscales_metadata() task, which runs in a subprocess under the 'processes' scheduler and leaves the original in-memory GroupMetadata stale. The fresh group is returned to _write_raster so the subsequent spatialdata attrs write also uses the correct on-disk state. Upstream issue: ome/ome-zarr-py#580 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use use_consolidated=False when re-opening group after compute The re-open after da.compute() was inheriting stale consolidated metadata, which caused KeyError when newly written children (e.g. labels3d_multiscale_xarray) were not yet reflected in it. Passing use_consolidated=False forces zarr to read group membership directly from the store instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* reduce warnings and fix pandas CoW issues - Use zip(strict=True) when converting tuple chunks to dict in RasterSchema - Copy AnnData views before modifying .uns in TableModel.parse - Use explicit string obs index in AnnData construction to avoid implicit index warnings - Add .copy() after compute()/slice to avoid pandas SettingWithCopyWarning (CoW) - Remove unused docs/_templates autosummary files - Update tutorials submodule Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix pre-commit * fix: avoid coordinate column collision in test_points_model to suppress INFO noise Build the dataframe with only the coordinate columns actually needed for each branch: source columns (A/B/C) when a coordinates mapping is provided, target columns (x/y/z) otherwise. This prevents the rename-drop and z-ignored-in-2D INFO messages (240 occurrences) from firing during tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use pd.Categorical for feature_key column in test_points_model to suppress WARNING noise The target column was a plain str Series; PointsModel.parse() converted it to an unknown-categories dask Categorical, triggering a performance warning on every parametrized invocation. Using pd.Categorical with known categories up-front avoids the conversion path entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: suppress 'not stored in most current format' UserWarning in old-format tests Use `re.match`-compatible pattern (anchored at start) and add `@pytest.mark.filterwarnings` to standalone parametrized tests outside `TestReadWrite` that intentionally exercise the V01 container format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refactor: simplify old-format warning suppression using pytest.mark.filterwarnings Replace verbose autouse fixtures (catch_warnings + isinstance check) with a single @pytest.mark.filterwarnings decorator on each class/function. The isinstance guard was unnecessary because V02 tests never emit the warning, so the filter is a harmless no-op for them. Also removes now-unused `import warnings` from test_format.py and test_readwrite.py, and CurrentSpatialDataContainerFormat from test_attrs_io.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* perf: session-scope blobs fixtures to cut setup time by ~10s Cache `blobs(256, 300, 3)` and `BlobsDataset()._labels_blobs()` once per session via private session-scoped fixtures, then deepcopy into each function-scoped fixture. Cuts fixture setup from 44.8s to 35.0s (-9.8s) and total suite from 186s to 180s. All 1332 tests still pass. Benchmark CSVs committed for reference (pytest_*.csv). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: use blobs(length=128) in test_vectorize to cut to_circles cost to_circles() on labels scales linearly with pixel count. Dropping from 512×512 to 128×128 cuts test_labels_2d_to_circles from ~3.9s to ~1.0s per parametrized variant (−4.7s across the file). Updated hardcoded coordinate/radius assertions to match the new size. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: remove benchmark CSVs from repo * chore: remove benchmark stats csv * perf: eliminate double-validation in __setitem__ and use fast fixture deepcopy Two orthogonal wins: 1. _elements.py: `get_model()` already calls `schema.validate()` internally; the explicit second validate() + get_axes_names() call in every __setitem__ was redundant. Removing it halves the DataTree (_to_dataset_view) overhead per element insertion — directly speeds up fixture setup. 2. conftest.py: introduce `_fast_deepcopy_sdata` (copy.deepcopy + manual attrs restoration for DaskDataFrame/scverse#503 and GeoDataFrame/scverse#286) that is ~13x faster than sd_deepcopy (7ms vs 93ms for full_sdata). Session-scope full_sdata, images, labels and the 'full' sdata parametrized fixture; each test gets a fresh 7ms copy instead of an 87ms full reconstruction. Also switch sdata_blobs from sd_deepcopy to fast_deepcopy_sdata (2ms vs 25ms). Full suite: 186s → 163s (~12% reduction, ~23s saved). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: add validate=False option to get_model, clarify setitem validation comment - get_model() now accepts validate=False to skip schema.validate() when the caller only needs to infer the element type without re-running validation - add comment to Elements.__setitem__ noting that subclass overrides call get_model() which performs the validation - drop Python 3.13 CI matrix entries (superseded by 3.14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: skip re-validation when building SpatialData from already-valid elements Add skip_element_validation() context manager (backed by a ContextVar) that makes __setitem__ call get_model(validate=False) — type inference only, no schema.validate(). Use it in every code path that constructs a SpatialData from elements that originated from an existing SpatialData and were never externally mutated: bounding_box_query, polygon_query, query_by_coordinate_system, transform_to_coordinate_system, subset, and init_from_elements. test_query_spatial_data: 0.77s → 0.64s (the remaining time is the query work itself — filtering, shapely ops, raster cropping). Also inline a minimal 2-image SpatialData in test_transformations_between_coordinate_systems instead of relying on the full 8-element images fixture; the test only ever uses image2d and image2d_multiscale, so writing the other 6 to disk was pure waste. test_transformations_between_coordinate_systems: 0.61s → 0.44s. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "perf: skip re-validation when building SpatialData from already-valid elements" This reverts commit acb25b1. * perf: vectorize label centroid computation, 30x speedup Replace per-slice O(H+W) approach (512 dask compute() calls for a 256×256 array) with a single array materialization + np.bincount O(n_pixels) pass. This speeds up get_centroids() on labels from ~1.5s to ~50ms, cutting to_circles(labels) from ~1.6s to ~53ms. Affects test_validation dataloader variants (~2.5s → ~0.2s each, saving ~9s), test_labels_2d_to_circles, and any production call to get_centroids or to_circles on label arrays. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: session-scope blobs fixtures in concatenate and get_attrs tests Add blobs_factory fixture that returns fast deepcopies of the session-scoped blobs dataset, and update test_concatenate_* and test_no_shared_transformations to use it instead of calling blobs() per test. Also update test_get_attrs.py sdata_attrs fixture and test_empty_attrs to use sdata_blobs. The concatenate parametrized tests (6 variants) drop from ~0.55s to ~0.02s each (~3s total savings); test_get_attrs drops from ~0.28s to ~0.02s per test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: reduce categories and partitions in test_categories_on_partitioned_dataframe Drop from 200 to 20 for both N_PARTITIONS and the number of gene categories. The test's assertions (round-trip values, category-order mismatch between pandas and dask, dtype) are all preserved at the smaller scale; the 200-partition stress test was over-specified for what's actually being verified. ~1.56s → ~0.95s (39% faster) for that test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * perf: shrink test_categories_on_partitioned_dataframe to N=10, local seed Drop from 20 to 10 for both N_PARTITIONS and the number of gene categories, and replace the module-level RNG with a local default_rng(seed=0) so the test is independent of other tests' RNG consumption. With seed=0 and N=10, partition 0 is deterministically missing gene_4, which is sufficient to trigger the dask category-order mismatch being tested. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * ci: parallelize test suite with pytest-xdist worksteal Add pytest-xdist to test dependencies and pass -n auto --dist worksteal to pytest in CI. worksteal distributes tests dynamically across workers so slow IO tests don't block fast unit tests on a single worker. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: clean up _get_centroids_for_labels docstring and clarify indexing - Remove reference to the old per-slice implementation from the docstring - Add inline comment explaining why indexing="ij" is correct for any number of spatial dimensions (2D and 3D labels) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: minor code swap for improved clarity * perf: speed up test suite — subset sdata, disable numba JIT, promote fixtures to session scope - test_delete_element_from_disk: subset full_sdata to [element_name, points_0_1] before writing, cutting the initial write from 19 elements to 2 (2–13× speedup per parametrize case) - conftest: set NUMBA_DISABLE_JIT=1 to avoid ~1.4s JIT overhead per worker on first datashader/rasterize call - test_partial_read: promote all module-scoped fixtures to session scope so the corrupted zarr stores are built once per session instead of once per module - test_transform: use small_translation=True to avoid out-of-bounds raster operations that trigger large dask computations - test_spatialdata_operations: reduce target_width 1000→100 in test_transform_to_data_extent to shrink the output raster Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix+perf: faster test suite — subset fixtures, disable numba JIT, fix napari plugin Napari registers a pytest11 entry point that loads (and breaks) numba before conftest.py runs. Fix: block it in pyproject.toml addopts with -p no:napari. datashader's @jit(cache=True) raises "no locator available" on Python 3.13 + numba. With napari blocked, conftest.py now runs before any plugin imports numba, so NUMBA_DISABLE_JIT=1 consistently disables all @jit decorators (datashader, xrspatial) and avoids the mixed JIT/non-JIT crash. Promote all module-scoped fixtures in test_partial_read to session scope so corrupted zarr stores are built once per session. For tests that write a full SpatialData object but only need a single element (or a small subset) to trigger the condition under test, subset the fixture before writing. The initial full write dominates test time (19 elements including 3-D multiscale rasters), so even subsetting to 1-6 elements gives 2-13× speedups per parametrize case: - test_delete_element_from_disk: subset to [element_name, points_0_1] - test_incremental_io_on_disk: subset to the 7 elements the loop accesses - test_overwrite_fails_when_zarr_store_present: use empty SpatialData() - test_element_already_on_disk_different_type: subset to [element_name] - test_self_contained: subset to [image2d, labels2d, points_0, circles] - test_change_path_of_subset: subset to the 5 elements + points_0_1 (needed so only_on_disk > 0 assertion passes) - test_validate_can_write_metadata_on_element: subset to [element_name] - test_save_transformations_incremental: subset to [element_name, image2d] (image2d anchors the non-self-contained assertion for the circles case) - test_consolidated_metadata: subset to one element per type - test_channel_names_raster_images_v1_to_v2_to_v3: subset to [image2d, image2d_multiscale] Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: update comments and refine test_consolidated_metadata subset Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: prevent numba JIT errors during test collection on runner envs Two issues caused collection-time failures when running tests in runner environments (e.g. the ome_sdc conda env where Airflow is
used): 1. os.environ.setdefault("NUMBA_DISABLE_JIT", "1") is a no-op if the environment already exports NUMBA_DISABLE_JIT=0. Switched to a force-set assignment, and added a sys.modules patch for the case where a pytest plugin (fast-array-utils, npe2,
napari-plugin-engine) has already imported numba before conftest.py runs — numba reads its config once at import time, so the env var alone is too late. 2. Module-level blobs() calls in four test files executed during pytest collection, before conftest.py had taken effect. Replaced
each with a @pytest.fixture(scope="module"), which fires during test execution. No performance change — blobs() is still called once
per file.
refactor: consolidate numba JIT disable into root conftest only The sub-conftest files in core/operations/ and dataloader/ duplicated the same NUMBA_DISABLE_JIT logic already present in the root tests/conftest.py. Removed the duplicates.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>* Fix write validation not catching empty (rather than None) transformations
After remove_transformation(element, remove_all=True) the transformations
dict is set to {} rather than None, bypassing the is-None guard in all three
IO writers. Changed the check to `not transformations` so both None and empty
dicts are caught, and added a parametrized regression test covering images,
multiscale images, labels, multiscale labels, shapes, and points.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Move empty-transformation validation into model validate() methods
Instead of guarding in each IO writer, call get_model(element) (which
already dispatches to the right schema and runs validate()) at the start
of _write_element() for all non-table spatial elements.
Also fix the is-None guards in all three validate() methods to use
`not transformations` / `not data.attrs.get(key)` so that an empty
dict {} is caught in addition to None.
The IO-level guards added in the previous commit are removed since they
are now superseded by the model-level check; assert statements are kept
to narrow the type for mypy.
The regression test is updated to reflect the correct production scenario:
element is already inside a SpatialData object when its transformations are
removed in-place, so the error fires during write() not at construction.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Move empty-transformation validation into model validate() methods
Instead of guarding in each IO writer, call validate_element(element)
(a new public helper in spatialdata.models that delegates to get_model)
at the start of _write_element() for all non-table spatial elements.
Validation changes in models.py:
- RasterSchema._check_transforms_present: two explicit checks —
one for None (key absent) and one for empty dict, with separate messages
- ShapesModel.validate / PointsModel.validate: same split into two checks
- asserts in IO files are kept solely for mypy type-narrowing, each
annotated with a comment explaining that validate_element() guarantees
the invariant at runtime
New public API:
- spatialdata.models.validate_element(e) raises ValueError if the element
fails schema validation; documented in docs/api/models_utils.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>* Fix adata layers None after anndata X unification * Handle layers[None] (X) correctly in sanitize_table and add test coverage Skip None when building new_keys so it is never passed to sanitize_name, and preserve it verbatim in new_dict so the layers setter round-trips X correctly on anndata >= 0.13 where X lives at layers[None]. Add test_sanitize_table_layers_preserves_x to verify X is intact after sanitization; the None-in-layers assertion is guarded by a pre-check so the test is valid on both anndata 0.12.x and >= 0.13. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* wip add pixi tasks * wip add pixi tasks * Add profiling skills and pixi profiling environment tasks Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…rse#946) * init * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix mypy linterrors * update the location and the design * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update docs * make coverage 100/100 because why not * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed type annotation * dont compute eagerly use. delete other instance key for consistency * update the tests and make sure we use match_element_to_table * wip rewrite tests using existing APIs * test passing without using subset_sdata_by_table_mask() * Remove _filter_by_instance_ids and _get_scale_factors; refactor tests to use existing API - Remove _get_scale_factors (duplicated logic already in transformations/_utils.py) - Remove _filter_by_instance_ids and subset_sdata_by_table_mask (superseded by match_sdata_to_table / filter_by_table_query) - Parametrize test_subset_sdata_by_table_mask over both API functions - Replace test_filter_2d_labels_by_instance_ids with test_filter_out_instances, parametrized over both API functions and element types (2D / multiscale labels) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add filter_label_pixels flag to match_sdata_to_table and filter_by_table_query Threads a filter_label_pixels: bool = False parameter through the full join stack (filter_by_table_query → match_sdata_to_table → join_spatialelement_table → _call_join → _right/_inner_join_spatialelement_table). When True, label pixels for removed instances are zeroed via a new _filter_labels_element helper (handles both DataArray and multiscale DataTree). When False (default), the existing warning is preserved but now also hints at the new flag. Tests no longer need manual _set_instance_ids_in_labels_to_zero calls or warnings suppression. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Change filter_label_pixels default to None; False silences the warning - None (default): warn that label pixels are not filtered, hint at the flag - True: filter label pixels (set removed instance pixels to zero) - False: skip silently, no warning Updated docstrings in join_spatialelement_table, match_sdata_to_table, and filter_by_table_query to document all three states. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Move and consolidate label-filtering tests into test_relational_query_match_sdata_to_table - Replace test_match_sdata_to_table_match_labels_error with test_filter_out_instances: parametrized over both API functions and element types; tests all three filter_label_pixels states (None→warn, False→nullcontext noop, True→pixels filtered) - Add test_subset_sdata_by_table_mask for mixed-element subsetting - Delete test_relational_query_subset_sdata_by_table_mask.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add 3D labels guard to _filter_labels_element; fix annsel list predicate in tests - Raise NotImplementedError in _filter_labels_element when element is Labels3DModel - Add test_filter_out_instances_3d_labels_not_supported parametrized over both API functions - Use an.col().is_in() instead of == [list] in 3D test (narwhals does not support nested literals) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Luca Marconato <m.lucalmer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
updates: - [github.com/pre-commit/mirrors-mypy: v1.20.2 → v2.0.0](pre-commit/mirrors-mypy@v1.20.2...v2.0.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Removed unnecessary compute() call * Using the R-tree for spatial querying of shapes * Cleanup * Cleanup * Cleanup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Optimize transformed point bounding box queries and add multi-box coverage * add comments * Used spatial indexing also for polygon query * sped up querying for scaling transformation and removed warning about performance issues * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ruff pre-commit violations in spatial_query.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * code review: remove points parameter in bounding box internal function; add extra tests for querying points; remove unnecessary copy() in polygon query of points * Fix bounding_box_query: restore negative-scale interval swap, use axes_adjusted in identity path, restore npartitions - Restore np.minimum/np.maximum swap so axis-flip transformations (negative scale) no longer raise ValueError; add regression test - Use axes_adjusted/min_coordinate_adjusted consistently in the identity path - Revert npartitions=1 back to points.npartitions in result construction - Add test for general affine transform (rotate 45° + translate) in aligned space Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dschaub95 <schaub.darius@gmail.com> Co-authored-by: LucaMarconato <2664412+LucaMarconato@users.noreply.github.com> Co-authored-by: Luca Marconato <m.lucalmer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…uet (scverse#1127) fix _search_for_backing_files_recursively() to support multifile parquet
* fix _search_for_backing_files_recursively() to support multifile parquet * Fix repr showing <Delayed> instead of row count for backed points (scverse#1084) Replace broken dask graph introspection (which only worked for single-task graphs with a HighLevelGraph layer API that no longer exists) with get_dask_backing_files() + pyarrow footer metadata reads. This handles all graph shapes including the list-of-piece-dicts case produced by aggregate_files=True. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix transform points with duplicate indices * Fix spatialdata_attrs lost after transform on points with duplicate index dd.from_delayed starts with empty attrs, so spatialdata_attrs (feature_key, instance_key) was silently dropped. Copy all non-transform attrs from the original element to restore the previous behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix test_filter_by_coordinate_system_also_table from @giovp * remove unused _filter_table_in_coordinate_systems(); replace assert * Refactor _filter_table_by_elements to delegate to join_spatialelement_table - Replace the bespoke numpy implementation in _filter_table_by_elements with a call to join_spatialelement_table(how="left"), removing the unused match_rows parameter and consolidating to a single code path. - Fix join functions (_left_join, _inner_join, _right_exclusive_join) to update spatialdata_attrs region metadata after filtering. - Fix _right_exclusive_join: restore groupby pattern (consistent with other joins) and use reset_index so integer positions are used as the group index, avoiding IndexError when obs names are duplicated and fixing a latent bug where pd.concat of per-group masks produced a partial-length boolean mask. - Simplify _get_filtered_or_unfiltered_tables to use _filter_table_by_elements. - Replace SpatialData.init_from_elements with get_model() lookup when sdata=None in join_spatialelement_table, removing an expensive importlib call. - Add test_join_updates_spatialdata_attrs covering metadata update for all join types; update test_filter_by_table_query_edge_cases to match new behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Remove _filter_table_by_element_names and replace with _filter_table_by_elements The old function filtered only by region name, ignoring instance IDs. Replace its only caller (filter_tables by coordinate system) with _filter_table_by_elements, which correctly filters by both region and instance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
update tutorials; add R and JS links (@wolfganghuber)
* implement selection * update * vectorize adjust_bounding_box_to_real_axes * update * replace append with insert * add comment * vectorize * update to handle multiple boxes * vectorize with numba * fix corner len * update * fix validation * refactor * refactor * add test for query with multiple bounding boxes * fix typing * vectorize bounding box query on polygons * add test to cover no polygon overlap (None) * vectorize bounding box query on points and tests * fix type * Fix rasterize path and bugs in PR scverse#687 dataloader; add benchmark **Bugs fixed in datasets.py:** - rasterize=True path was broken: __getitem__ always called image.sel() regardless of rasterize flag, bypassing rasterize_fn entirely. Fixed by storing self._rasterize and branching in __getitem__. - ad.concat(*tables_l) unpacked the list as positional args, failing with >1 region. Fixed to ad.concat(tables_l). - Vectorized selection pre-computation was always run even for rasterize=True where it is unused. Fixed by guarding with `if not rasterize`. - Removed stale commented-out pandas.apply fallback code. **Fixes in _utils.py:** - Removed redundant nopython=True from @nb.njit (njit implies nopython=True, and the argument caused a RuntimeWarning). - Replaced invalid nb.types.Array[nb.float64, nb.float64] annotations with np.ndarray. **Fixes in spatial_query.py:** - Restored BoundingBoxRequest validation that was commented out. The validator's __post_init__ already handles both 1-D (single box) and 2-D (multi-box) arrays. **Benchmark (benchmark_dataloader.py):** Synthetic 2048x2048 image, 500 circle regions (32 px radius), 3-channel. Phase main PR (fixed) speedup init ~162 ms ~20 ms ~8x fetch 500 ~618 ms ~118 ms ~5x per-tile ~1237 us ~235 us ~5x Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * add asv benchmark for dataloader performance --------- Co-authored-by: Luca Marconato <m.lucalmer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
updates: - [github.com/pre-commit/mirrors-mypy: v2.0.0 → v2.1.0](pre-commit/mirrors-mypy@v2.0.0...v2.1.0) - [github.com/astral-sh/ruff-pre-commit: v0.15.12 → v0.15.13](astral-sh/ruff-pre-commit@v0.15.12...v0.15.13) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* add raster_compressor argument for writing raster data compressed to disk * add roundtrip tests for writing raster data in compressed manner --------- Co-authored-by: Luca Marconato <m.lucalmer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.15.13 → v0.15.14](astral-sh/ruff-pre-commit@v0.15.13...v0.15.14) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
for more information, see https://pre-commit.ci
…yen/spatialdata into feat/zarr-store-class
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.