Incremental io

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.

[ArneDefauw]

PR for illustration of possible workflow for adding elements to spatialdata object inspired by

spatialdata/tests/io/test_readwrite.py

Line 136 in 582622f

deftest_incremental_io_on_disk(

and related to discussion #520.

Currently only implemented for image and labels, but similar approaches would work for points, shapes and tables.

See unit test for example usage.
https://github.com/ArneDefauw/spatialdata/blob/2e71e63ca9ccbd849879efe1f26c7221cdd60b34/tests/io/test_readwrite.py#L662

Approach also allows for writing to multiscale, and avoids recomputation via writing to intermediate slot.

Some things are not covered yet, such as automatic reload, and proper passing of transformations to spatialdata.models.Image2Dmodel.parse

I am still unsure to what extent spatialdata should implement such workflows (e.g. overwriting elements on disk), or should merely give suggestions about possible workflows (via notebooks/documentation)

[LucaMarconato]

Notes from the in-person discussion.

• we want to give more control to the user for in-memory vs non-memory objects, by giving control of how each element is stored Refactoring the io and in-memory lazy vs non-lazy representation #293 We should consider also the option to make the objects lazy but persistent (.persist().
• we identified a possible bug in ome-zarr-py that could lead up to ~10x slower writing performance when dealing with large multiscale images; Optimize writing performance for MultiscaleSpatialImage #577.
• the overwrite behavior got tested by @ArneDefauw on large data pipelines and no problems were highlighted; we should enable it inside write() and write_element() in a follow-up PR.

Thanks again for the demo of this PR and for bringing attention to the above topics.