Uh oh!
There was an error while loading. Please reload this page.
Expose apply_ufunc as public API and add documentation - #1619
Conversation
TODO: turn the description from GH1517 into an auto-gallery example, and link to it from `computation.rst` and `dask.rst`.
jhamman
left a comment
There was a problem hiding this comment.
This looks pretty good. I understand this much better now myself. Ping me again once you have linked to the doc/example/recipe.
| written to work on NumPy arrays to support labels on xarray objects. | ||
| ``apply_ufunc`` also support automatic parallelization for many functions | ||
| with dask. See :ref:`computation.wrapping-custom` for details. | ||
There was a problem hiding this comment.
Don't forget:
(:issue:`XXXX`). By `Stephan Hoyer <https://github.com/shoyer>`_. | many functions when using dask by setting ``dask='parallelized'``. This is | ||
| illustrated in a separate example. | ||
| .. TODO: add link! |
There was a problem hiding this comment.
I'm thinking the recipes would be a good place for this: http://xarray.pydata.org/en/stable/auto_gallery/index.html
There was a problem hiding this comment.
Yes, if I can come up with a figure!
benbovy
left a comment
There was a problem hiding this comment.
Great! Just found two small typos while reading the new doc.
| Unfortunately, a limitation of the current version of numpy means that we | ||
| cannot override ufuncs for datasets, because datasets cannot be written as | ||
| a single array [1]_. :py:meth:`~xarray.Dataset.apply` works around this | ||
| Unfortunately, we current do not support NUmPy ufuncs for datasets [1]_. |
| squared_error = lambda x, y: (x - y) ** 2 | ||
| arr1 = xr.DataArray([0, 1, 2, 3], dims='x') | ||
| xr.apply_func(squared_error, arr1, 1) |
shoyer
commented
Oct 10, 2017
jhamman
left a comment
There was a problem hiding this comment.
doc updates look good. Just one small comment.
| Another option is to use xarray's :py:func:`~xarray.apply_ufunc`, which can | ||
| automatically parallelize functions written for processing NumPy arrays when | ||
| applied to dask arrays. It works similarly to :py:func:`dask.array.map_blocks` | ||
| and :py:func:`dask.array.atop`, but without requiring an immediate layer of |
| It doesn't always make sense to do computation directly with xarray objects: | ||
| - When working with small arrays (less than ~1e7 elements), applying an |
There was a problem hiding this comment.
Is the point on speed is a distraction? If an array is that small, the absolute difference in speed is still very small, so it really only makes a difference if you're doing those operations in a loop.
There was a problem hiding this comment.
Agreed. I had "in the inner loop" in mind when I wrote this, but I see now that that never made it into the text. Let me know if this latest update seems more reasonable to you, or if I'm still pushing too hard on performance considerations.
| xarray functionality: | ||
| Calculate the vector magnitude of two arguments: | ||
| Calculate the vector magnitude of two arguments:: |
There was a problem hiding this comment.
When you write ::, Sphinx formats the block below as code in the generated API docs.
shoyer
commented
Oct 19, 2017
Any additional comments? If not, I'll merge this in about 24 hours. |
apply_ufunc()does not meet all our needs for wrapping unlabeled array routines with xarray (see #1618 for a proposal forapply_raw()), but it should be useful for many advanced users and isn't doing much in its current state as non-public API. So I'd like to make it public for the xarray 0.10 release.git diff upstream/master | flake8 --diffwhats-new.rstfor all changes andapi.rstfor new APIcomputation.rstanddask.rst.@MaximilianR@rabernat@jhamman Review from any of you would be appreciated here!