Partial reading

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.

[aeisenbarth]

This PR implements modes for read_zarr that allow to continue reading SpatialData elements even if some elements caused read errors, for example due to corrupted files (crash during write, a race condition or copy errors). This is useful, because with growing datasets (by number of elements), a corruption of a single file can make all other elements inaccessible. The ability to read at least the unaffected data helps for data recovery.

This is implemented by wrapping element-specific reader calls into a context managerhandle_read_errors. If enabled with on_bad_files="warn", it converts exceptions into warnings, and continues execution. Since execution is not aborted, all subsequent code within an iteration must be wrapped in the context manager to avoid that it is executed after a handled error. In order to identify which element failed to be read, the warning message is prepended with its location, e.g.:

shapes/blobs_polygons: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

At the end, a SpatialData object is always returned, containing only the successfully read elements.

The current implementation is at the granularity level of whole elements, that means if anything in an element causes an error, the whole element is skipped. In principle it could be made more fine-granular, e.g. returning an element with just invalid attributes skipped, or returning a table with unreadable columns skipped. Especially for large single tables with annotations for all elements, this would be very beneficial (although they are column-based, so that a corrupted file would destroy a column's values for all annotations). However, the relevant code is not contained in this repository, but requires changes in anndata.read_zarr. To avoid losing annotations due to file corruption, I would recommend not to use a single table, but one table per SpatialData element.

For tests, see test_read_zarr_with_error and test_read_zarr_with_warnings.

Closes#457

Release notes

- Added option `on_bad_files="warn"` to `read_zarr` for partially reading corrupted datasets #457

[codecov]

Codecov Report

Attention: Patch coverage is 89.89899% with 10 lines in your changes missing coverage. Please review.

Project coverage is 92.00%. Comparing base (a7e9e99) to head (9b1c7a3).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/io_zarr.py	87.71%	7 Missing ⚠️
src/spatialdata/_io/io_table.py	86.36%	3 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #765 +/- ##
==========================================
+ Coverage 91.95% 92.00% +0.05% 
==========================================
Files 47 47 Lines 7293 7331 +38 ==========================================
+ Hits 6706 6745 +39 + Misses 587 586 -1 

Files with missing lines	Coverage Δ
src/spatialdata/_io/_utils.py	88.44% <100.00%> (+1.21%)	⬆️
src/spatialdata/_io/io_table.py	94.44% <86.36%> (+0.56%)	⬆️
src/spatialdata/_io/io_zarr.py	88.46% <87.71%> (+0.68%)	⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LucaMarconato]

Thanks @aeisenbarth, great PR and very well structured tests! I made some very minor adjustments here: I made a test more strict and replaced shutil.rmtree() with another approach; with the other devs we opted to minimize/avoid calls to rmtree(), for possible safety concerns.