Add cells tutorial

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!

[timtreis]

This notebook describes the creation of the cells dataset which I hope ends up being a more biologically looking dataset that we can use for user-facing testing and small tutorials. Ideally it'd be distributed via SpatialData, if out-of-scope it'll land via Squidpy.

I'm not sure where in the documentation it'd belong, but I feel like it should be public?

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[LucaMarconato]

The notebook looks very good to me, thanks! I have some comments for minor fixes, let's merge soon then.

Comments:

• The notebook needs to be tested in spatialdata-integration-testing. If you want to give it a try please go ahead, if instead you prefer not to have to deal with spatialdata-integration-testing, we can do it at some point later (it's a Claude one-shot task once one has the pipeline running (also easy to setup)).
• The rectangle coordinates are different from the bounding box, I'd make them the same
• We'll implement a helper function for this at some point

  ny, nx = morph_top.sizes["y"], morph_top.sizes["x"]
  px_per_um_x = nx / (xmax - xmin)
  px_per_um_y = ny / (ymax - ymin)
  

• typo here (extra space): Lazy, dask-backed points ( length)
• here we'll provide as cells via SpatialData.datasets., I guess you mean either lowercase spatialdata or squidpy.
• when talking about non-trivial transformations, I'm not sure what it is meant with the word "baked".

[LucaMarconato]

Pre-approving

[timtreis]

Tried it but then it doesn't serve its purpose anymore because it fully covers the little tissue blob I'm targeting

[ajkswamy]

Thanks @timtreis for this nice notebook. I went through it's current state and it's a very valuable example that illustrates how to visualize, subset and save a xenium 3.0.0 dataset using spatialdata framework.

I downloaded the raw data needed (~38GB) and tried to run it on my linux laptop, but the kernel crashed as the OS ran out of RAM (16GB + 20GB Swap). Here are my observations:

1. Please link the associated PR on spatialdata.
2. The cell with imports fails with ModuleNotFoundError: No module named 'spatialdata_io'. Could we add it to the dependencies of this repo?
3. I think this notebook it too heavy for an average user to run it locally, as it needs the user to download a large amount of raw data (~38GB) and use more than 36GB of RAM to run the notebook (my kernel crashed at the bounding-box query step). One idea is to split it into download.py, to_zarr.py, etc as has been done for other datasets in spatialdata-sandbox to save the subsetted SpatialData object to zarr. This notebook could then start by downloading that zarr from our datasets list, loading it etc. However, I am not very sure about this. @LucaMarconato what do you think here? How would this concern affect testing this notebook on spatialdata-integration-testing?

[timtreis]

I think this notebook it too heavy for an average user to run it locally,

I'm loading a very normal Xenium dataset though, it's not even that large 😅 I think typically people just run these things on their HPC and not private laptops. I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

The notebook was more to document how it was created and where it came from.

[LucaMarconato]

Thanks @timtreis and @ajkswamy for the code and review!

Please link the associated PR on spatialdata.

Good point, but luckily no need anymore now that scverse/spatialdata#1149 is merged.

we'll provide as cells via SpatialData.datasets.

I changed the title to spatialdata.datasets since now we ship it also there; I kept both spatialdata.datasets and squidpy.datasets in the test description.

I generally don't think that this is a notebook a user will realistically want to run or even should run, especially since the cells dataset will be distributed via SpatialData eventually.

I agree.

One idea is to split it into download.py

Still, this is very important if we want to ensure that the notebook is tested by us via spatialdata-integration-testing and the cells datasets that we host in S3 is up-to-date (when we will bump the on-disk format). I will keep the current notebook as is, but I will add a comment that the data can also be downloaded via a download.py file available in spatialdata-sandbox (I'm adding it here giovp/spatialdata-sandbox#63). At some point we will hook it up to spatialdata-integration-testing.

[LucaMarconato]

Downloaded the data via the new spatialdata-sandbox downloader. The notebook runs on my machine. I will finish reviewing and we can merge soon.

[LucaMarconato]

@timtreis are you fine with he_aligned and he_image having different resolutions or would you prefer them to have the same?

Currently we have:

├── Images
│ ├── 'he_aligned': DataTree[cyx] (3, 430, 540), (3, 215, 270)
│ ├── 'he_image': DataTree[cyx] (3, 423, 339), (3, 211, 169)
│ └── 'morphology_focus': DataTree[cyx] (4, 430, 540), (4, 215, 270)

Edit: ah ok it's done on purpose "Non-trivial transformations: he_image still carries its affine+translation Sequence as a lazy coordinate transformation, next to he_aligned, the same image after that transformation has been applied (rasterized) into pixel space, so both coordinate-system and rasterize/alignment code paths are represented.", great!

[LucaMarconato]

Finished reviewing the notebook, looks great to me, ready to merge!