Implement gmt xarray BackendEntrypoint by weiji14 · Pull Request #3919 · GenericMappingTools/pygmt

weiji14 · 2025-04-17T02:14:01Z

Description of proposed changes

Allow 'gmt' to be used as an engine in xarray.open_dataarray and xarray.open_dataset for decoding raster NetCDF (grid) or GeoTIFF (image) files!

TODO:

Initial implementation supporting kind="grid"
Let pygmt.io.load_dataarray() use engine="gmt" by default
Support raster_kind="image"
Documentation improvements

References:

Implements idea in #3673 (comment)

Preview: https://pygmt-dev--3919.org.readthedocs.build/en/3919/api/generated/pygmt.GMTBackendEntrypoint.html

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash command is:

/format: automatically format and lint the code

Initial implementation of a 'gmtread' xarray BackendEntrypoint for decoding NetCDF files! Following instructions at https://docs.xarray.dev/en/v2025.03.1/internals/how-to-add-new-backend.html on registering a backend. Only a minimal implementation for now to read kind=grid data.

So that GMTDataArray accessor knows the path to the original NetCDF file to retrieve correct metadata.

Use gmtread as default engine instead of netcdf4.

Default is still kind='grid', by allow use of kind='image' too.

Users will need to set the 'kind' parameter explicitly to 'grid' or 'image'.

Also set `kind='grid'` argument in _load_earth_relief_holes

Need to set a default value of None for 'decode_kind' it seems, from reading https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#open-dataset

Because xr.core.types.ReadBuffer is not available until xarray 2024.11.0, xref pydata/xarray#9787

Partially reverts b9e3eb4

weiji14 · 2025-04-17T06:39:00Z

Probably best to split this into two PRs so that we have two changelog entries:

(feature) Implement gmtread xarray BackendEntrypoint
- Changes to pyproject.toml, xarray_backend.py and tests/test_xarray_backend.py
**Breaking** Set load_dataarray's default engine to engine="gmtread", and require decode_kind parameter to be set (e.g. to decode_kind='grid')
- Changes to io.py, helpers/testing.py, datasets/samples.py, etc

But please take a look at the current implementation, and suggest comments first.

seisman · 2025-04-17T06:57:15Z

Probably best to split this into two PRs so that we have two changelog entries:
(feature) Implement gmtread xarray BackendEntrypoint
Changes to pyproject.toml, xarray_backend.py and tests/test_xarray_backend.py
Breaking Set load_dataarray's default engine to engine="gmtread", and require decode_kind parameter to be set (e.g. to decode_kind='grid')
Changes to io.py, helpers/testing.py, datasets/samples.py, etc

Sounds good to me.

But please take a look at the current implementation, and suggest comments first.

Here are my questions:

Why not just use "gmt" as the engine name?
The load_dataarray function is almost the same as the xr.load_dataarray function, except that load_dataarray also get the GMT accessor information. With the "gmtread" engine added, xr.load_dataarray("xxx.grd", engine="gmtread", decode_kind="grid") works exactly the same as load_dataarray("xxx.grd", decode_kind="grid"). So maybe we can deprecate load_dataarray?

weiji14 · 2025-04-17T07:16:51Z

Why not just use "gmt" as the engine name?

Yes, we could do that actually, and engine="gmt" will make more sense if we want to implement writing to a NetCDF driver later via xr.DataArray.to_netcdf.

The load_dataarray function is almost the same as the xr.load_dataarray function, except that load_dataarray also get the GMT accessor information. With the "gmtread" engine added, xr.load_dataarray("xxx.grd", engine="gmtread", decode_kind="grid") works exactly the same as load_dataarray("xxx.grd", decode_kind="grid"). So maybe we can deprecate load_dataarray?

True, the GMT accessor loading is already in the BackendEntrypoint, so we might as well deprecate load_dataarray. Ok to do it in 2 releases (remove in v0.18.0) or 4 releases (remove in v0.20.0)?

seisman · 2025-04-17T08:01:43Z

True, the GMT accessor loading is already in the BackendEntrypoint, so we might as well deprecate load_dataarray. Ok to do it in 2 releases (remove in v0.18.0) or 4 releases (remove in v0.20.0)?

4 releases sounds good to me.

seisman · 2025-04-17T08:15:55Z

+ ext = Path(filename_or_obj).suffix
+ except TypeError:
+ return False
+ return ext in {".grd", ".nc", ".tif"}


What about .tiff?

Suggested change
returnextin {".grd", ".nc", ".tif"}
returnextin {".grd", ".nc", ".tif", ".tiff"}

Ok, added .tiff. Any other known extensions to add? E.g. based on https://docs.generic-mapping-tools.org/6.5/reference/file-formats.html#grid-files?

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

seisman · 2025-04-18T00:20:23Z

 PyGMT input/output (I/O) utilities.
 """

+from typing import Literal


Since we're going to deprecate load_dataarray(xref: #3919 (comment)), we should revert the changes in this file.

Do we want to change the default engine in load_dataarray to engine="gmt" or no?

Changing the default engine to engine="gmt" means that users have to set decode_kind, so it's a breaking change. It makes little sense to introduce a breaking change to a deprecated function, so my answer is no.

Ok, I've reverted changes to io.py in commit f722b47. So keeping default engine behaviour as before for the load_dataarray function that will be deprecated in #3922.

seisman · 2025-04-18T00:21:17Z

 """
 fname = which("@static_earth_relief.nc", download="c")
- return load_dataarray(fname)
+ return load_dataarray(fname, decode_kind="grid")


Again, with load_dataarray deprecated, we should use xr.load_dataarray(fname, engine="gmt", decode_kind="grid") instead, but it can be done in a separate PR.

Started draft PR at #3922, will cherrypick things across.

seisman · 2025-04-18T00:32:49Z

 .. autosummary::
 :toctree: generated

+ GMTBackendEntrypoint


I'm wondering if it makes more sense to have a separate category like "Xarray Integration" and put both GMTBackendEntrypoint and GMTDataArrayAccessor in the category. Also, if #3854 is implemented, GMTDataArrayAccessor no longer makes sense in the "Metadata" category.

Yeah, I was wondering about this too. I can put GMTBackendEntrypoint in a new "Xarray Integration" section as you suggested, and then we'll remove the I/O section in a separate PR.

Move GMTDataArrayAccessor to the "Xarray Integration" section in this PR?

Move GMTDataArrayAccessor to the "Xarray Integration" section in this PR?
I think do that in #3854. Hopefully this PR can be done before that.

seisman · 2025-04-23T02:26:28Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):

weiji14 · 2025-04-23T03:21:14Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):

Appears to be a sphinx thing. I found mention of a napoleon_use_ivar setting set here (added in #383) that converts the .. attribute:: directive, and because the docstring output has Attributes:, sphinx converts it to ivar 😅

seisman · 2025-04-23T06:20:49Z

This PR almost looks good, except that the doctest doesn't look good (what's ivar?):
Appears to be a sphinx thing. I found mention of a napoleon_use_ivar setting set here (added in #383) that converts the .. attribute:: directive, and because the docstring output has Attributes:, sphinx converts it to ivar 😅

The xarray documentation also uses napoleon, but the xarray repr looks correct (e.g., https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html#xarray.DataArray), but I don't find any special settings in the conf.py file (https://github.com/pydata/xarray/blob/main/doc/conf.py).

seisman · 2025-04-23T06:23:42Z

+ Attributes:
+ Conventions: CF-1.7
+ title: Produced by grdcut
+ history: grdcut @earth_relief_01d_p -R-55/-47/-24/-10 -Gstatic_eart...
+ description: Reduced by Gaussian Cartesian filtering (111.2 km fullwidt...
+ actual_range: [190. 981.]
+ long_name: elevation (m)


A workaround for issue #3919 (comment)
Suggested change
Attributes:
Conventions: CF-1.7
title: Producedbygrdcut
history: grdcut @earth_relief_01d_p-R-55/-47/-24/-10-Gstatic_eart...
description: ReducedbyGaussianCartesianfiltering (111.2kmfullwidt...
actual_range: [190.981.]
long_name: elevation (m)
...

I put the ellipisis after the colon in 'Attributes:' line, and it seemed to remove the :ivar from showing up (see commit f2446fc):
Only downside is the ellipsis shows as Attributes:..., but should be acceptable.

seisman · 2025-04-23T14:35:55Z

+ # `chunks` and `cache` DO NOT go here, they are handled by xarray
+ ) -> xr.Dataset:
+ """
+ Backend open_dataset method used by Xarray in :py:func:`~xarray.open_dataset`.


With the fix in #3927, sphinx-autogen will generate a stub file for this method. The method documentation is not ideal, at least one "Parameters" section is needed.

weiji14 · 2025-04-24T00:59:26Z

Considering moving this file to pygmt/xarray/backend.py, and then potentially in #3854, we can move pygmt/accessors.py to pygmt/xarray/accessors.py? And if we do pandas accessors (xref #3854 (comment)) in the future, we could put it under pygmt/pandas/accessor.py?

Sounds good

Xref #2620

Also cross-referencing to https://docs.generic-mapping-tools.org/6.5/reference/features.html#grid-file-format

seisman · 2025-04-24T01:53:07Z

+
+ Internally, GMT uses the netCDF C library to read netCDF files, and GDAL for GeoTIFF
+ and other raster formats. See also
+ :gmt-docs:`reference/features.html#grid-file-format`.


Maybe add a few sentences explaining why users may want to read netCDF/GeoTIFF files via GMT, rather than netcdf4/rasterio, something like:
Compared to the "netcdf4"/"rasterio" engines, the "gmt" engine can read GMT remote files (file names starting with @) directly and provides the GMTDataArray accessor .gmt for easy access to GMT-specific features and metadata.

Ok, updated the description in commit acab8b2, please review. I didn't mention the netcdf4/rasterio engines directly, but just mentioned that the GMT engine works better with GMT remote files and features in general.

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

seisman · 2025-04-24T02:56:53Z

Looks pretty good to me.

weiji14 · 2025-04-24T04:32:07Z

Will leave this up for review until end of the week-ish.

Xref 6a4e1a4 in #3919

weiji14 added the feature Brand new feature label Apr 17, 2025

Add source encoding to raster xarray.DataArray
2b98a65
So that GMTDataArray accessor knows the path to the original NetCDF file to retrieve correct metadata.

seisman mentioned this pull request Apr 17, 2025
Add pygmt.gmtread to read a dataset/grid/image into pandas.DataFrame/xarray.DataArray #3673
Draft
7 tasks

weiji14 added 11 commits April 17, 2025 14:30

Set load_dataarray's default engine to gmtread
f41a812
Use gmtread as default engine instead of netcdf4.

Support reading GeoTIFF images via the kind='image' argument
b71dcf5
Default is still kind='grid', by allow use of kind='image' too.

[skip ci] Satisfy type checking
60c6726

Make 'kind' a required kwarg with no default value
bf09a39
Users will need to set the 'kind' parameter explicitly to 'grid' or 'image'.

Add .grd to set of supported extensions in guess_can_open
e622878
Also set `kind='grid'` argument in _load_earth_relief_holes

Rename 'kind' parameter to 'decode_kind' and mypy fix
b9e3eb4
Need to set a default value of None for 'decode_kind' it seems, from reading https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#open-dataset

Rename parameter kind to decode_kind in load_dataarray
3a109ac

Specify decode_kind="grid" in load_ functions
d764f78

[skip ci] format
98910f8

Use type: ignore[override]
126560e
Because xr.core.types.ReadBuffer is not available until xarray 2024.11.0, xref pydata/xarray#9787

Don't set default decode_kind in open_dataset
436f0fc
Partially reverts b9e3eb4

weiji14 added the needs review This PR has higher priority and needs review. label Apr 17, 2025

Rename engine="gmtread" to engine="gmt"
328b6ab

weiji14 changed the title ~~Implement gmtread xarray BackendEntrypoint~~Implement gmt xarray BackendEntrypointApr 17, 2025

Add GMTBackendEntrypoint to doc/api/index.rst
06aa39d

seisman reviewed Apr 17, 2025
View reviewed changes

Apply suggestions from code review
c79e50a
Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

seisman reviewed Apr 18, 2025
View reviewed changes

weiji14 added 4 commits April 20, 2025 08:26

Add test for xr.load_dataarray with a GRD grid
c0ccd4b

Update GMTBackendEntrypoint description to say NetCDF C library is used
c5ebf9f

lint
17fd9d7

Move GMTBackendEntrypoint to a new "Xarray Integration" API section
1c8af3a

[skip ci] lint
2e5fb6b

seisman reviewed Apr 23, 2025
View reviewed changes

seisman mentioned this pull request Apr 23, 2025
Add 'toctree' option to autosummary directive to generate stub pages #3927
Merged

seisman reviewed Apr 23, 2025
View reviewed changes

Merge branch 'main' into backendentrypoint/gmtread
b136e26

weiji14 commented Apr 24, 2025
View reviewed changes

weiji14 added 5 commits April 24, 2025 13:08

NetCDF to netCDF
19c5252
Xref #2620

Add Parameters section to docs of open_dataset
3bb1f44
Also cross-referencing to https://docs.generic-mapping-tools.org/6.5/reference/features.html#grid-file-format

Move pygmt/xarray_backend.py to pygmt/xarray/backend.py
8984c1c

Workaround sphinx Attributes ivar using ellipsis
f2446fc

Remove extra blankspace
52c0b7e

seisman reviewed Apr 24, 2025
View reviewed changes

weiji14and others added 2 commits April 24, 2025 14:08

Apply suggestions from code review
143a375
Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

Describe more on why one should choose the gmt engine
acab8b2

weiji14 marked this pull request as ready for review April 24, 2025 04:18

weiji14 added final review call This PR requires final review and approval from a second reviewer and removed needs review This PR has higher priority and needs review. labels Apr 24, 2025

seisman approved these changes Apr 24, 2025
View reviewed changes

seisman merged commit ecb933c into mainApr 26, 2025

seisman deleted the backendentrypoint/gmtread branch April 26, 2025 01:21

seisman removed the final review call This PR requires final review and approval from a second reviewer label Apr 26, 2025

This was referenced Apr 26, 2025
Reorganize GMTDataArrayAccessor to the pygmt.xarray module #3928
Merged
Remove the dependency of netCDF4 #3643
Merged

weiji14 added a commit that referenced this pull request Apr 27, 2025
Rename decode_kind to raster_kind
6430bb2
Xref 6a4e1a4 in #3919

weiji14 mentioned this pull request Apr 29, 2025
Allow passing region to GMTBackendEntrypoint.open_dataset #3932
Merged
6 tasks

	returnextin {".grd", ".nc", ".tif"}
	returnextin {".grd", ".nc", ".tif", ".tiff"}

-Attributes:
-Conventions: CF-1.7
-title: Producedbygrdcut
-history: grdcut @earth_relief_01d_p-R-55/-47/-24/-10-Gstatic_eart...
-description: ReducedbyGaussianCartesianfiltering (111.2kmfullwidt...
-actual_range: [190.981.]
-long_name: elevation (m)
+ ...

Uh oh!

Conversation

weiji14 commented Apr 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weiji14 commented Apr 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seisman commented Apr 17, 2025

Uh oh!

weiji14 commented Apr 17, 2025

Uh oh!

seisman commented Apr 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14Apr 17, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14Apr 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seismanApr 20, 2025• edited by weiji14 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seisman commented Apr 23, 2025

Uh oh!

weiji14 commented Apr 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seisman commented Apr 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weiji14Apr 24, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

seismanApr 23, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seisman commented Apr 24, 2025

Uh oh!

weiji14 commented Apr 17, 2025•
edited
Loading

weiji14 commented Apr 17, 2025•
edited
Loading

weiji14Apr 17, 2025•
edited
Loading

weiji14Apr 19, 2025•
edited
Loading

seismanApr 20, 2025•
edited by weiji14
Loading

weiji14 commented Apr 23, 2025•
edited
Loading

weiji14Apr 24, 2025•
edited
Loading

seismanApr 23, 2025•
edited
Loading