Skip to content

Optimize polyfit - #9766

Merged
dcherian merged 8 commits into
pydata:mainfrom
dcherian:polyfit-blockwise-reshape
Nov 13, 2024
Merged

Optimize polyfit#9766
dcherian merged 8 commits into
pydata:mainfrom
dcherian:polyfit-blockwise-reshape

Conversation

@dcherian

@dcheriandcherian commented Nov 11, 2024

Copy link
Copy Markdown
Contributor

Closes#5629

  1. Use Variable instead of DataArray
  2. Use reshape_blockwise when possible following Polyfit performance on large datasets - Suboptimal dask task graph #5629 (comment)
  • User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @phofl

Comment threadxarray/tests/test_dataset.py
Closespydata#5629
1. Use Variable instead of DataArray
2. Use `reshape_blockwise` when possible following pydata#5629 (comment)
@dcherian
dcherianforce-pushed the polyfit-blockwise-reshape branch from 9295f17 to 3ec786cCompareNovember 11, 2024 17:31
Comment threaddoc/whats-new.rst Outdated
@dcherian

Copy link
Copy Markdown
ContributorAuthor

@aulemahal can you check that xclim's test suite passes with this please

@phofl

Copy link
Copy Markdown
Contributor

Nice!

Comment threaddoc/whats-new.rst Outdated
@aulemahal

Copy link
Copy Markdown
Contributor

@dcherian I can confirm the test suite of xclim passes successfully when tested against this branch!

Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>
@dcheriandcherian added the plan to merge Final call for comments label Nov 12, 2024
@dcherian
dcherian merged commit 5a9ff0b into pydata:mainNov 13, 2024
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 16, 2024
* main:
fix cf decoding of grid_mapping (pydata#9765)
Allow wrapping `np.ndarray` subclasses (pydata#9760)
Optimize polyfit (pydata#9766)
Use `map_overlap` for rolling reductions with Dask (pydata#9770)
fix html repr indexes section (pydata#9768)
@dcherian
dcherian deleted the polyfit-blockwise-reshape branch November 17, 2024 16:15
dcherian added a commit that referenced this pull request Nov 19, 2024
* main: (24 commits)
Bump minimum versions (#9796)
Namespace-aware `xarray.ufuncs` (#9776)
Add prettier and pygrep hooks to pre-commit hooks (#9644)
`rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (#9720)
Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (#9793)
Buffer types (#9787)
Add download stats badges (#9786)
Fix open_mfdataset for list of fsspec files (#9785)
add 'User-Agent'-header to pooch.retrieve (#9782)
Optimize `ffill`, `bfill` with dask when `limit` is specified (#9771)
fix cf decoding of grid_mapping (#9765)
Allow wrapping `np.ndarray` subclasses (#9760)
Optimize polyfit (#9766)
Use `map_overlap` for rolling reductions with Dask (#9770)
fix html repr indexes section (#9768)
Bump pypa/gh-action-pypi-publish from 1.11.0 to 1.12.2 in the actions group (#9763)
unpin array-api-strict, as issues are resolved upstream (#9762)
rewrite the `min_deps_check` script (#9754)
CI runs ruff instead of pep8speaks (#9759)
Specify copyright holders in main license file (#9756)
...
dcherian added a commit to dcherian/xarray that referenced this pull request Nov 19, 2024
* main:
Bump minimum versions (pydata#9796)
Namespace-aware `xarray.ufuncs` (pydata#9776)
Add prettier and pygrep hooks to pre-commit hooks (pydata#9644)
`rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (pydata#9720)
Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (pydata#9793)
Buffer types (pydata#9787)
Add download stats badges (pydata#9786)
Fix open_mfdataset for list of fsspec files (pydata#9785)
add 'User-Agent'-header to pooch.retrieve (pydata#9782)
Optimize `ffill`, `bfill` with dask when `limit` is specified (pydata#9771)
fix cf decoding of grid_mapping (pydata#9765)
Allow wrapping `np.ndarray` subclasses (pydata#9760)
Optimize polyfit (pydata#9766)
Use `map_overlap` for rolling reductions with Dask (pydata#9770)
@dcheriandcherian mentioned this pull request Nov 19, 2024
6 tasks
wavebyrd pushed a commit to wavebyrd/xarray that referenced this pull request Mar 13, 2026
* main: (24 commits)
Bump minimum versions (pydata#9796)
Namespace-aware `xarray.ufuncs` (pydata#9776)
Add prettier and pygrep hooks to pre-commit hooks (pydata#9644)
`rolling.construct`: Add `sliding_window_kwargs` to pipe arguments down to `sliding_window_view` (pydata#9720)
Bump codecov/codecov-action from 4.6.0 to 5.0.2 in the actions group (pydata#9793)
Buffer types (pydata#9787)
Add download stats badges (pydata#9786)
Fix open_mfdataset for list of fsspec files (pydata#9785)
add 'User-Agent'-header to pooch.retrieve (pydata#9782)
Optimize `ffill`, `bfill` with dask when `limit` is specified (pydata#9771)
fix cf decoding of grid_mapping (pydata#9765)
Allow wrapping `np.ndarray` subclasses (pydata#9760)
Optimize polyfit (pydata#9766)
Use `map_overlap` for rolling reductions with Dask (pydata#9770)
fix html repr indexes section (pydata#9768)
Bump pypa/gh-action-pypi-publish from 1.11.0 to 1.12.2 in the actions group (pydata#9763)
unpin array-api-strict, as issues are resolved upstream (pydata#9762)
rewrite the `min_deps_check` script (pydata#9754)
CI runs ruff instead of pep8speaks (pydata#9759)
Specify copyright holders in main license file (pydata#9756)
...
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

plan to mergeFinal call for commentstopic-dask

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Polyfit performance on large datasets - Suboptimal dask task graph

4 participants

@dcherian@phofl@aulemahal@max-sixty