Skip to content

Add blogpost on flox heuristics #695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 25 commits into
base: main
Choose a base branch
from
Draft

Add blogpost on flox heuristics #695

Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1b261ab
Add blogpost on flox heuristics
dcherian Jul 29, 2024
e04df47
Comment out images for now
dcherian Jul 29, 2024
503a36b
Update src/posts/flox-smart/index.md
dcherian Jul 29, 2024
093a75e
edits
dcherian Jul 29, 2024
d7eb21d
Merge branch 'main' into flox-smart
dcherian Jul 29, 2024
44a8f02
more edits
dcherian Jul 29, 2024
105c22d
more edits
dcherian Jul 30, 2024
010f01d
more edits
dcherian Jul 30, 2024
1136f29
more edits
dcherian Jul 31, 2024
1383ad8
more edits
dcherian Jul 31, 2024
5a2bb65
edit
dcherian Jul 31, 2024
1cc7eb5
edits
dcherian Jul 31, 2024
53e1c3a
more edits
dcherian Jul 31, 2024
ff8ccff
tweak
dcherian Jul 31, 2024
1c19f7f
Update
dcherian Aug 1, 2024
5da9295
Add demo.
dcherian Aug 3, 2024
ad8caea
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 3, 2024
2203a34
Fix
dcherian Aug 3, 2024
ebbd007
Add assets
dcherian Aug 3, 2024
9448fb2
fix
dcherian Aug 3, 2024
ab715b9
edit
dcherian Aug 3, 2024
dc559a7
Updates
dcherian Aug 5, 2024
5a0764d
Merge branch 'main' into flox-smart
dcherian Aug 28, 2024
ec9f274
Merge remote-tracking branch 'upstream/main' into flox-smart
dcherian Nov 12, 2024
8b39d32
edits
dcherian Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update
  • Loading branch information
@dcherian
dcherian committed Aug 1, 2024
commit 1c19f7fd486b67ffe120f73e799bcb07a4e068a4
3 changes: 1 addition & 2 deletions src/posts/flox-smart/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,7 @@ Given the above `C`, flox will choose `"cohorts"` for chunk sizes (1, 2, 3, 4, 6

Cool, isn't it?!

Importantly this inference is fast — [400ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where the group labels are a 2GB 2D array!
But we have not tried with bigger problems (example: GroupBy(100,000 watersheds) in the US).
Importantly this inference is fast — [~250ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where approximately 3000 groups are distributed over 2500 chunks; and ~1.25s for grouping by US watersheds ~87000 groups across 640 chunks.

## What's next?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your conclusion is a bit rambly, and I think some of the content (especially call-outs to any other groupby nerds who can help) could be in a footnote.

I think you should end with a succinct one-line conclusion that emphasises that most of this complexity can now be safely ignored by users.


Expand Down