You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR introduces the use of GeoParquet to save shapes and replaces the old ragged array-based storage.
I also fixes the issue that only the geometry column was saved, now all the columns of the dataframe are saved. Closes#311
Notes:
No performance change is observed for writing (they are both binary formats). But GeoParquet improves interoperability.
The PR is backward compatible: all previous datasets can be read, and the user can choose if writing the shapes with the latest format or with a previous format. This choice can be performed per-element.
Tests are added to ensure the compatibility between the shapes "v01" and "v02"
These last two points required a overhaul of format.py and related code, code that now is more capable and robust. A few notes on this:
For raster types we didn't introduce a "spatialdata format" because we were optimistic that transformations etc would have landed in NGFF faster; since they are not there yet, I believe that we need versioning also for raster types (which I now implemented).
To avoid confusion between the version property, that always gives 0.4 and the spatialdata version, I renamed the version property to spatialdata_format_version. When writing points, shapes and tables, we need just the second one (because NGFF doesn't support them); instead, while writing labels and images we need both: ome-zarr-py needs to know that we are working with 0.4, and spatialdata needs to know which spatial data format for raster we are using to augment the NGFF storage.
I have introduced a function _parse_version() that retrieves the spatialdata zarr version for a given element from its Zarr store.
The function _parse_version() less clean that I hoped, this because of the versioning was done in a heterogenous way for the elements:
Shapes and points saved and still save the version under .attrs['spatialdata_attrs']['version']
Tables saved the version under .attrs['version'], they still save the version there for backward compatibility even if switching to .attrs['spatialdata_attrs']['version'] would be cleaner.
Images and labels didn't save the version at all, now they save it under .attrs['spatialdata_attrs']['version']. If _parse_version() can't find the version for raster types, it is assumed that the spatialdata RasterFormatV01 is used.
* test read write on disk
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* improved tests for workarounds for incremental io
* fixed tests
* improved comment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Luca Marconato <m.lucalmer@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces the use of GeoParquet to save shapes and replaces the old ragged array-based storage.
I also fixes the issue that only the geometry column was saved, now all the columns of the dataframe are saved.
Closes#311
Notes:
These last two points required a overhaul of format.py and related code, code that now is more capable and robust. A few notes on this:
versionproperty, that always gives 0.4 and the spatialdata version, I renamed theversionproperty tospatialdata_format_version. When writing points, shapes and tables, we need just the second one (because NGFF doesn't support them); instead, while writing labels and images we need both:ome-zarr-pyneeds to know that we are working with0.4, and spatialdata needs to know which spatial data format for raster we are using to augment the NGFF storage._parse_version()that retrieves the spatialdata zarr version for a given element from its Zarr store._parse_version()less clean that I hoped, this because of the versioning was done in a heterogenous way for the elements:.attrs['spatialdata_attrs']['version'].attrs['version'], they still save the version there for backward compatibility even if switching to.attrs['spatialdata_attrs']['version']would be cleaner..attrs['spatialdata_attrs']['version']. If_parse_version()can't find the version for raster types, it is assumed that the spatialdataRasterFormatV01is used.