Add blogpost on flox heuristics by dcherian · Pull Request #695 · xarray-contrib/xarray.dev

dcherian · 2024-07-29T22:18:38Z

https://xarray-dev-git-flox-smart-xarray.vercel.app/blog/flox-smart

vercel · 2024-07-29T22:18:40Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
xarray-dev	✅ Ready (Inspect)	Visit Preview	✅ 7 resolved	Nov 12, 2024 9:42pm

Co-authored-by: Anderson Banihirwe <13301940+andersy005@users.noreply.github.com>

dcherian · 2024-07-29T22:36:18Z

@@ -0,0 +1,157 @@
+---
+title: 'flox: GroupBy, now with smarts!'
+date: '2024-05-31'


Suggested change
date: '2024-05-31'
date: '2024-08-05'

or whenever we decide to merge.

dcherian · 2024-07-30T05:19:26Z

+
+Unlike [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index), _containment_ [isn't skewed](http://ekzhu.com/datasketch/lshensemble.html) when one of the sets is much larger than the other.
+
+The steps are as follows:


Is this bit clear and useful? Should I take it out? Or expand it?

for more information, see https://pre-commit.ci

TomNicholas

Nice! Really incredible work you have done here @dcherian

I left a few comments: some nits, some minor suggestions on structure/content.

TomNicholas · 2024-08-06T22:12:06Z

+Notice how much cleaner the graph is in this image:
+![map-reduce](https://flox.readthedocs.io/en/latest/_images/new-map-reduce-reindex-True-annotated.svg)
+See our [previous blog post](https://xarray.dev/blog/flox) or the [docs](https://flox.readthedocs.io/en/latest/implementation.html#method-map-reduce) for more.


It's a little confusing that these two images are not for identical problems - the inputs are different.

TomNicholas · 2024-08-06T22:18:02Z

+
+_Note that the improvements here are strongly dependent on the details
+of the grouping variable, the chunksize, and even dask's scheduler. In fact, writing this post
+prompted the discovery of a bug in dask's scheduler that should substantially improve the "map-reduce"


TomNicholas · 2024-08-06T22:28:26Z

+
+Importantly this inference is fast — [~250ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where approximately 3000 groups are distributed over 2500 chunks; and ~1.25s for grouping by US watersheds ~87000 groups across 640 chunks.
+
+## What's next?


Your conclusion is a bit rambly, and I think some of the content (especially call-outs to any other groupby nerds who can help) could be in a footnote.
I think you should end with a succinct one-line conclusion that emphasises that most of this complexity can now be safely ignored by users.

TomNicholas · 2024-08-06T22:30:33Z

+Using the algorithm described below, flox will **automatically** set
+`method="cohorts"` for this dataset unless specified, yielding a 5X decrease in
+memory used and a 2X increase in runtime. Read on to figure out how!
+Read on to figure out how!


A significant number of readers will think it's cool that this is now automatic, but not really care about the specifics of how you did it. That section below is also very dense. I think if there is any other information that you want readers to take away from this point onwards (e.g. that Groupers exist) you should telegraph it, and explicitly give them the option to skip forwards to a different subheading guilt-free.

dcherian · 2024-08-06T22:36:36Z

Thanks for the review @TomNicholas . I'm waiting for some tests on the new dask release before revising: https://docs.dask.org/en/stable/changelog.html#improve-scheduling-efficiency-for-xarray-groupby-reduce-patterns

* main: Link to second cubed blog post (#702) Bump webpack from 5.92.1 to 5.93.0 (#700) Bump @fontsource-variable/inter from 5.0.18 to 5.0.20 (#698) Bump typescript from 5.5.3 to 5.5.4 (#699) Bump the dependencies group with 7 updates (#697)

* upstream/main: Fix hydration warning due to nested links (#725) Bump typescript from 5.6.2 to 5.6.3 (#724) [pre-commit.ci] pre-commit autoupdate (#720) Bump webpack from 5.95.0 to 5.96.1 (#723) Bump the dev-dependencies group across 1 directory with 4 updates (#719) Bump webpack from 5.94.0 to 5.95.0 (#718) Bump @fontsource-variable/inter from 5.0.20 to 5.1.0 (#717) Bump typescript from 5.5.4 to 5.6.2 (#716) Bump the dependencies group with 9 updates (#714) Bump next from 14.2.7 to 14.2.10 (#711) Turn off survey banner (#710) Update opengraph thumbnails (#709) GroupBy(multiple variables) (#703) Bump webpack from 5.93.0 to 5.94.0 (#704) Bump micromatch from 4.0.7 to 4.0.8 (#707) Bump the dependencies group with 7 updates (#705) Bump the dev-dependencies group with 4 updates (#706)

dcherian · 2024-11-12T21:38:42Z

+
+we run
+
+```python


TODO: this doesn't reproduce any more, will need a different problem

Add blogpost on flox heuristics
1b261ab

vercelBothad a problem deploying to Preview July 29, 2024 22:18 Failure

Comment out images for now
e04df47

vercelBothad a problem deploying to Preview July 29, 2024 22:20 Failure

andersy005 reviewed Jul 29, 2024
View reviewed changes

Comment threadsrc/posts/flox-smart/index.md Outdated

dcherianand others added 3 commits July 29, 2024 16:35

Update src/posts/flox-smart/index.md
503a36b
Co-authored-by: Anderson Banihirwe <13301940+andersy005@users.noreply.github.com>

edits
093a75e

Merge branch 'main' into flox-smart
d7eb21d

dcherian commented Jul 29, 2024
View reviewed changes

vercelBotdeployed to Preview July 29, 2024 22:38 View deployment

more edits
44a8f02

dcherian force-pushed the flox-smart branch from 0c6e212 to 44a8f02Compare July 29, 2024 22:44

vercelBotdeployed to Preview July 29, 2024 22:46 View deployment

vercelBotdeployed to Preview July 30, 2024 02:39 View deployment

more edits
105c22d

dcherian force-pushed the flox-smart branch from eb64606 to 105c22dCompare July 30, 2024 02:40

vercelBotdeployed to Preview July 30, 2024 02:41 View deployment

dcherian commented Jul 30, 2024
View reviewed changes

Comment threadsrc/posts/flox-smart/index.md Outdated

dcherian commented Jul 30, 2024
View reviewed changes

Comment threadsrc/posts/flox-smart/index.md

dcherian commented Jul 30, 2024
View reviewed changes

Comment threadsrc/posts/flox-smart/index.md

dcherian commented Jul 30, 2024
View reviewed changes

more edits
010f01d

vercelBotdeployed to Preview July 30, 2024 22:52 View deployment

more edits
1136f29

vercelBotdeployed to Preview July 31, 2024 04:01 View deployment

more edits
1383ad8

vercelBotdeployed to Preview July 31, 2024 04:02 View deployment

edit
5a2bb65

vercelBotdeployed to Preview July 31, 2024 04:04 View deployment

more edits
53e1c3a

dcherian force-pushed the flox-smart branch from 3507c99 to 53e1c3aCompare July 31, 2024 04:18

vercelBotdeployed to Preview July 31, 2024 04:20 View deployment

tweak
ff8ccff

vercelBotdeployed to Preview July 31, 2024 04:23 View deployment

dcherianand others added 3 commits August 1, 2024 09:57

Update
1c19f7f

Add demo.
5da9295

[pre-commit.ci] auto fixes from pre-commit.com hooks
ad8caea
for more information, see https://pre-commit.ci

vercelBotdeployed to Preview August 3, 2024 03:24 View deployment

Fix
2203a34

vercelBotdeployed to Preview August 3, 2024 03:29 View deployment

Add assets
ebbd007

vercelBotdeployed to Preview August 3, 2024 03:31 View deployment

fix
9448fb2

vercelBotdeployed to Preview August 3, 2024 03:34 View deployment

edit
ab715b9

vercelBotdeployed to Preview August 3, 2024 03:38 View deployment

dcherian force-pushed the flox-smart branch from 433afa6 to ab715b9Compare August 3, 2024 03:38

vercelBotdeployed to Preview August 3, 2024 03:39 View deployment

Updates
dc559a7

vercelBotdeployed to Preview August 5, 2024 19:30 View deployment

TomNicholas reviewed Aug 6, 2024
View reviewed changes

Merge branch 'main' into flox-smart
5a0764d
* main: Link to second cubed blog post (#702) Bump webpack from 5.92.1 to 5.93.0 (#700) Bump @fontsource-variable/inter from 5.0.18 to 5.0.20 (#698) Bump typescript from 5.5.3 to 5.5.4 (#699) Bump the dependencies group with 7 updates (#697)

TomNicholas added the blog Blog post label Sep 13, 2024

vercelBotdeployed to Preview November 12, 2024 21:21 View deployment

dcherian commented Nov 12, 2024
View reviewed changes

edits
8b39d32

vercelBotdeployed to Preview November 12, 2024 21:42 View deployment


		Unlike [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index), _containment_ [isn't skewed](http://ekzhu.com/datasketch/lshensemble.html) when one of the sets is much larger than the other.

		The steps are as follows:


		Importantly this inference is fast — [~250ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where approximately 3000 groups are distributed over 2500 chunks; and ~1.25s for grouping by US watersheds ~87000 groups across 640 chunks.

		## What's next?


		we run

		```python

Uh oh!

Conversation

dcherian commented Jul 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vercelBot commented Jul 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dcherianJul 29, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcherianJul 29, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcherianJul 30, 2024

Choose a reason for hiding this comment

Uh oh!

TomNicholas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TomNicholasAug 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TomNicholasAug 6, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TomNicholasAug 6, 2024

Choose a reason for hiding this comment

Uh oh!

TomNicholasAug 6, 2024

Choose a reason for hiding this comment

Uh oh!

dcherian commented Aug 6, 2024

Uh oh!

dcherianNov 12, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dcherian commented Jul 29, 2024•
edited
Loading

vercelBot commented Jul 29, 2024•
edited
Loading

dcherianJul 29, 2024•
edited
Loading