Skip to content

Add very minimal xarray plugin for engine="rasterio" - #281

Merged
snowman2 merged 28 commits into
corteva:masterfrom
bopen:xarray-plugin
Apr 7, 2021
Merged

Add very minimal xarray plugin for engine="rasterio"#281
snowman2 merged 28 commits into
corteva:masterfrom
bopen:xarray-plugin

Conversation

@alexamici

@alexamicialexamici commented Apr 5, 2021

Copy link
Copy Markdown
Contributor

Use xarray plugin infrastructure to add support for:

>>>importxarrayasxr>>>xr.open_dataset("myfile.jp2", engine="rasterio")
<xarray.Dataset>Dimensions: (x: 19087, y: 3932)
Coordinates:
*band (band) int641*y (y) float64 ...
*x (x) float64 ...
spatial_refint640Datavariables:
band_data (band, y, x) float32 ...

also add automatic engine selection for files with .tif and .geotif extensions.

>>>xr.open_dataset("myfile.tif")
<xarray.Dataset>Dimensions: (x: 19087, y: 3932)
Coordinates:
*band (band) int641*y (y) float64 ...
*x (x) float64 ...
spatial_refint640Datavariables:
band_data (band, y, x) float32 ...

@codecov

codecovBot commented Apr 5, 2021

Copy link
Copy Markdown

Codecov Report

Merging #281 (511d99e) into master (74c2c0f) will decrease coverage by 1.08%.
The diff coverage is 0.00%.

❗ Current head 511d99e differs from pull request most recent head 18d7bd8. Consider uploading reports for the commit 18d7bd8 to get more accurate results
Impacted file tree graph

@@ Coverage Diff @@## master #281 +/- ##
==========================================
- Coverage 93.68% 92.59% -1.09% 
==========================================
Files 12 13 +1 Lines 1362 1378 +16 ==========================================
Hits 1276 1276 - Misses 86 102 +16 
Impacted FilesCoverage Δ
rioxarray/xarray_plugin.py0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74c2c0f...18d7bd8. Read the comment docs.

@alexamici
alexamici marked this pull request as draft April 5, 2021 10:49
Comment threadrioxarray/xarray_plugin.py Outdated
@snowman2

Copy link
Copy Markdown
Member

Use xarray plugin infrastructure to add support for:

>>>importxarrayasxr>>>xr.open_dataset("myfile.jp2", engine="gdal")
<xarray.Dataset>Dimensions: (x: 19087, y: 3932)
Coordinates:
*y (y) float64 ...
*x (x) float64 ...
spatial_refint640Datavariables:
band1 (y, x) float32 ...
Attributes:
scale_factor: 1.0add_offset: 0.0grid_mapping: spatial_ref

also add automatic engine selection for files with .tif and .geotif extensinos.

>>>xr.open_dataset("mygile.tif")
<xarray.Dataset>Dimensions: (x: 19087, y: 3932)
Coordinates:
*y (y) float64 ...
*x (x) float64 ...
spatial_refint640Datavariables:
band1 (y, x) float32 ...
Attributes:
scale_factor: 1.0add_offset: 0.0grid_mapping: spatial_ref

One thing I am noticing in these examples is that the attributes for the band data variables are on the Dataset. If you go this route, it would be important to make sure that those attributes are pushed to the DataArray.

Comment threadrioxarray/xarray_plugin.py Outdated
@alexamici

Copy link
Copy Markdown
ContributorAuthor

One thing I am noticing in these examples is that the attributes for the band data variables are on the Dataset. If you go this route, it would be important to make sure that those attributes are pushed to the DataArray.

You are right, I used .to_netcdf and started looking for fix-ups, but I missed the fact that array attributes where promoted to global attributes.

Anyway the whole PR is work in progress. If you like the idea we can discuss the direction to go here, or in the main issue.

Comment threadrioxarray/xarray_plugin.py Outdated
@alexamici

alexamici commented Apr 6, 2021

Copy link
Copy Markdown
ContributorAuthor

@snowman2 The current implementation is just to see where I'd be aiming following pydata/xarray#2844

>>>importxarrayasxr>>>xr.open_dataset("test/test_data/input/cog.tif", decode_coords="coordinates") # or decode_coords=True<xarray.Dataset>Dimensions: (band: 1, x: 500, y: 500)
Coordinates:
*band (band) int641*y (y) float642.715e+062.714e+06 ... 2.565e+062.565e+06*x (x) float641.635e+061.635e+06 ... 1.784e+061.784e+06Datavariables:
spatial_refint64 ...
band_data (band, y, x) int16 ...
>>>xr.open_dataset("test/test_data/input/cog.tif", decode_coords="all")
<xarray.Dataset>Dimensions: (band: 1, x: 500, y: 500)
Coordinates:
*band (band) int641*y (y) float642.715e+062.714e+06 ... 2.565e+062.565e+06*x (x) float641.635e+061.635e+06 ... 1.784e+061.784e+06spatial_refint64 ...
Datavariables:
band_data (band, y, x) int16 ...

I didn't manage to add the "grid_mapping": "spatial_ref" to band_data.encoding yet, but once we do it the resulting netCDF will be the same in both cases.

The best course of action would be to implement decode_coords="all" in open_rasterio though.

@alexamici

Copy link
Copy Markdown
ContributorAuthor

@snowman2 I'll close the PR for now, waiting for a resolution of #282. Thanks for your attention!

@alexamicialexamici reopened this Apr 6, 2021
@alexamici
alexamici marked this pull request as ready for review April 6, 2021 19:43
@alexamicialexamici changed the title WIP: add minimal xarray plugin for engine="gdal"Add very minimal xarray plugin for engine="rasterio"Apr 6, 2021
Comment threadrioxarray/xarray_plugin.py
Comment threadrioxarray/xarray_plugin.py
Comment threadtest/integration/test_integration_xarray_plugin.py
Comment threaddocs/getting_started/getting_started.rst Outdated
Comment threadrioxarray/xarray_plugin.py Outdated
@snowman2

Copy link
Copy Markdown
Member

@alexamici are you aware of the target release date for xarray 0.18?

@snowman2

Copy link
Copy Markdown
Member

@alexamici am I good to squash and merge? (rasterio 1.2.2 was just released and looks like it is breaking things, so I am not worried about test failures related to this PR).

@alexamici

alexamici commented Apr 7, 2021

Copy link
Copy Markdown
ContributorAuthor

Release should be "soonish", but we have no target date to my knowledge.

Anyway the changes are totally backward compatible, since che xarray_plugin.py is not imported in older version of xarray and will only be imported by upcoming releases.

@alexamici

Copy link
Copy Markdown
ContributorAuthor

You can merge now for me.

@snowman2
snowman2 merged commit 7e61ace into corteva:masterApr 7, 2021
@snowman2

Copy link
Copy Markdown
Member

Thanks @alexamici 👍

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENH: Add xarray entrypoint

3 participants

@alexamici@snowman2@justingruca