dummy_spatialdata/generate_datasets used in a separate package

[Artur-man]

Although generate_datasets was used for creating arbitrary spatialdata objects, perhaps it is better to split to a separate package since BioC data packages are mostly used to access data sources only.

SpatialData.data/R/generate_dataset.R

Lines 50 to 63 in 47ca584

	generate_dataset <- function(file = tempfile(fileext = ".zarr"),
	sd_version = getOption("sd_version"),
	images = NULL,
	labels = NULL,
	shapes = NULL,
	points = NULL,
	tables = NULL,
	coordinate_systems = NULL,
	seed = 42L) {
	proc <- basilisk::basiliskStart(
	.get_basilisk_env(sd_version)
	)
	on.exit(basilisk::basiliskStop(proc))
	basilisk::basiliskRun(proc, function(file) {

[HelenaLC]

Just to clarify, from the code above, this writes a Zarr store and requires Python? Or would this eventually be R-native and include in-memory generation? Depending on the overall extent and complexity, I would favor keeping it in SpatialData as opposed to a separate package, because 1) changes to SD might requires changes to the class and 2) it could help with unit testing.

[HelenaLC]

If, on the other hand, it is just calling a Python module, we might not even need a package. One could use plain reticulate to call dummy_spatialdata from Python or no? I worry about long-term maintenance and cross-package synchronization if we overkill factorization too early. Can always do it later on, if sensible. E.g., we had a single package (not three) for nearly a year, I think.

[Artur-man]

If, on the other hand, it is just calling a Python module, we might not even need a package.

Yes it is just a python package, whether we seperate as a package or not, I was wondering if it should be a part of a BioC.data package in essence.

My initial goal with it was to easily generate datasets with different spatialdata versions, thus you do not need to create environments yourself for versions, the package handles it and also wraps the module. Perhaps that was what you were referring to with changes to SD ?

I am asking because perhaps we do not need versions anymore ... both spatialdata v0.3.0 and v0.5.2 uses outdated zarr v2s. At some point we have to make a decision to support which versions, or starting from which version. CC @Bisaloo.

I would favor keeping it in SpatialData as opposed to a separate package.

We can do this too. Currently the package has three functionalities:

• Download .zarr object or data bundles (from providers) from various buckets
• Facilitate interface to spatialdata-io to read/import data bundles, thus an R wrapper for https://github.com/scverse/spatialdata-io (very important, Roche was asking this)
• Facilitate interface to https://github.com/BIMSBbioinfo/dummy-spatialdata

We can discuss what to keep, what to split/take-out .... I am ok with any

[Bisaloo]

Facilitate interface to spatialdata-io to read/import data bundles, thus an R wrapper for scverse/spatialdata-io (very important, Roche was asking this)

Isn't the natural workflow here to read the data with spatialdata-io in Python, save it on-disk as an actual SpatialData object and then read it in R with spatialdataR?

I would just document this in the spatialdataR vignette.

[Artur-man]

No, most datasets we download are already processed by spatialdata-io.
https://spatialdata.scverse.org/en/latest/tutorials/notebooks/datasets/README.html

[Artur-man]

I would just document this in the spatialdataR vignette.

What are you suggesting? We initially thought SpatialData.data can be a separate package (aside from downloading sd stores from dedicated s3 buckets) to lift the heavy load of reticulate/basilisk workflows since thats not something we will do natively in R, and do not wanna do directly in spatialdataR.

[HelenaLC]

We are talking about "dummy" data, no? Not downloading existing datasets. .data remains for "real" datasets as is. My comments/suggests were only pertaining synthetic data: If it is only calling a wrapper, we might not need a package and can document in the vignette. If this can be done R-natively in the future, we can still separate into a new package later.

[Artur-man]

Ok, lets keep it then! I will leave the issue open for now though.