implement Gradient

[fujiisoup]

• ClosesShape preserving diff via new keywords #1332
• Tests added
• Tests passed
• Fully documented, including whats-new.rst for all changes and api.rst for new API

Added xr.gradient, xr.DataArray.gradient, and xr.Dataset.gradient according to #1332.

Altough this API is similar to numpy's counterpart, I'm wondering whether we really need to support this. The return value is a tuple of DataArrays, which I feel inconsistent to other xarray functions.

When I wrote my own wrapper for gradient of a DataArray, I returned a dataset with each variable being a gradient along one dimension, but even that isn't very elegant ¯_(ツ)_/¯

It would be another option, but in order to store them to a dataset, we will need to define their names very heuristically...

Maybe we can drop this api as this does not speed up the computation and it's easy to do the same thing manually.

I agree. I don't think it's particularly useful to compute to compute independent gradients with respect to multiple axes at the same time, e.g., du/dx, du/dy, etc. If anything, I would be more excited about computing higher order derivatives, e.g., \frac{d^2 u}{dx dy}

Maybe this should be implemented in the upstream...

Do you mean like this? Requires Dask 0.17.3+ though. So IDK if that works for you or not.

Thanks @jakirkahm.
I need to compute the gradient with a non uniform coordinate, but dask Array.gradient only supports uniformly spacing coordinate.

I think this can be implemented using overlap as I do here. Is it in the scope of the dask development?

Ah sorry. Should have read more of the code.

Yeah I think that is in scope. Just not currently supported yet.

Could you please open an issue or perhaps PR over on the Dask repo?

E501 line too long (80 > 79 characters)

E501 line too long (82 > 79 characters)

E501 line too long (81 > 79 characters)

E501 line too long (82 > 79 characters)

E501 line too long (80 > 79 characters)

E501 line too long (82 > 79 characters)

E501 line too long (82 > 79 characters)

E226 missing whitespace around arithmetic operator

E501 line too long (83 > 79 characters)

E226 missing whitespace around arithmetic operator

I think this will be very welcome functionality!

I wonder if we should consider calling this xarray.differentiate instead of xarray.gradient. I think the NumPy function is poorly named for differentiating along a single axis at once.

F841 local variable 'ag' is assigned to but never used

F841 local variable 'n_chunk' is assigned to but never used

W293 blank line contains whitespace

E126 continuation line over-indented for hanging indent

I wonder if we should consider calling this xarray.differentiate instead of xarray.gradient. I think the NumPy function is poorly named for differentiating along a single axis at once.

Agreed. Differentiate is nicer.

Some other api questions arised during the implementation

• Do we support differentiate for Dataset? In that case, what should we do for the variables that are independent from the target coordinate?
  I thought 'keep them as is' is intuitive (and I implemented so), but mathematically, they should be zero.

• Do we need to sort the array before computing differentiate? np.gradient implicitly assumes the array is sorted (but do nothing about this).

E303 too many blank lines (2)

F821 undefined name 'AxisError'
E501 line too long (80 > 79 characters)

E303 too many blank lines (2)

E303 too many blank lines (2)

F401 '.npcompat' imported but unused

any thoughts for this?

Why would you sort the array? Aren't you taking differences of values and dividing by differences between the matching coordinates?

Thanks, @dopplershift .

Aren't you taking differences of values and dividing by differences between the matching coordinates?

Yes, correct. But if we have closer data points, then the estimate of the gradient becomes more precise. Sorting the array according to the coordinate provides the closest points, resulting in the most precise estimate of the gradient.

But I also think users can do it manually before taking the gradient.

I'm a little confused here. You wrote an implementation for Dataset.differentiate, too, and here is a duplicate version of DataArray.differentiate.

Agreed. Maybe the top level function is not necessary here and DataArray.differentiate and Dataset.differentiate would be sufficient.

👍 I don't think we need the separate function.

W293 blank line contains whitespace

E131 continuation line unaligned for hanging indent

needs time_unit here

Use tuple unpacking here, e.g., time_unit, _ = np.datetime_data(coord_data.dtype)

In the long term, hopefully we'll solve this by getting real unit support working, but a time_unit argument seems like a good choice for now

I think it's ready :)

@fujiisoup I'm really looking forward to having this built in to xarray! Thanks for the extra work in making things dask-compatible as well.

Eventually it would be nice if this worked on DataArrays with cftime.datetime coordinates; I think it would be relatively straightforward to modify to_numeric to enable it (we could probably enable it for interp at the same time), but I can take care of that later if you'd like.

I think xarray.differentiate is no longer added.

It looks like everywhere to_numeric is called, this check is already made. Is it redundant to check again here (or conversely are the checks before the function is called redundant)?

This example raises an error when you run it -- I think you need to supply two dimension names.

"finite central differences" -> "centered finite differences"

This is so great! This is going to simplify my classes about derivatives on the globe even more: my only concern is that students will soon forget what a numerical derivative actually is, and that it's not a trivial to implement ;-)

Speaking of derivatives on the globe, we might want to include an option for periodic boundary conditions (and possibly for other functions like interp as well). But we can save that for another PR :).

But we can save that for another PR :)

See also #1288 : integrate is the next on my list ;) - I can try to give it a go if @fujiisoup doesn't want to do it himself

@shoyer, have you seen da.pad?

@shoyer, have you seen da.pad?

Yes, pad solves the hard part of periodic boundary conditions -- we should just need to do a bit of book-keeping on the xarray side.

I agree that the features implemented here are broadly useful and belong in xarray. The fundamental question is: how far does xarray itself want to go in supporting vector calculus in curvilinear coordinates (i.e. on the sphere).

There is a fair bit of overlap between this new functionality and some of the things that we are trying to do in xgcm: https://xgcm.readthedocs.io/en/latest/grids.html. Xgcm supports periodic boundary conditions, as well as more complex topological connections between array edges. It allows users to reproduce precisely the sort of operations used to take gradients in finite-volume staggered-grid models.

@rabernat my inclination would be to draw the line at regular grids (with or without periodic boundary conditions), which are pretty commonly encountered in data analysis for any continuous physical systems. We would leave non-regular cases like staggered-grids to add-ons like xgcm -- these grids tend to be more application/PDE specific.

We should definitely mention specialized tools like xgcm as alternatives in the docstring for these xarray methods.

Thanks all. Updated.

Thanks, @spencerkclark

Eventually it would be nice if this worked on DataArrays with cftime.datetime coordinates; I think it would be relatively straightforward to modify to_numeric to enable it (we could probably enable it for interp at the same time), but I can take care of that later if you'd like.

Thanks. I added this function for something like this extension, though I do not yet fully follow your cftime update. It would be super nice if you could take care of this after merge.

@shoyer ,

we might want to include an option for periodic boundary conditions

Agreed. This option is nice not only differentiate but also interp and rolling.
I think we can add a common logic to take care of them.

@fmaussion

See also #1288 : integrate is the next on my list ;) - I can try to give it a go if @fujiisoup doesn't want to do it himself

Thanks.
Actually, I am now moving to another nation and would not have enough time in a few weeks.
I will appreciate if you could take care of this.

I added this function for something like this extension, though I do not yet fully follow your cftime update. It would be super nice if you could take care of this after merge.

I totally understand; cftime objects probably seem fairly esoteric to non climate scientists. I'll be happy to take care of this after this gets merged. Indeed having the relevant logic already contained in to_numeric is very convenient.

I'm going to merge this after the Appveyor tests pass, unless there are any further objections.

@rabernat let me know if this makes sense to you / if you agree.

Sorry I'm late to the party on reviewing this. It looks like a great feature, and I know @fujiisoup has put a huge amount of work into it! 👏

My only concern is that the first thing most geoscience users will try with this function is to take a field with lat, lon dimensions and call differentiate on it, expecting to obtain the zonal and meridional gradients.

I feel that some mention is needed in the documentation that this feature is limited to simple cartesian geometry, in which the physical locations of the variables are accurately described by the dimension coordinates.

This is a place where we could mention the limitations of differentiate and gradient for non-cartesian geometry.

and here

and here

Thanks, @rabernat for the review.

Added the limitation of this method to docs.

👍