Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ooc-sort issues #14959

Merged
merged 1 commit into from
Mar 10, 2024
Merged

fix: ooc-sort issues #14959

merged 1 commit into from
Mar 10, 2024

Conversation

ritchie46
Copy link
Member

Ensures the binview garbage collects after partitioning otherwise we multiply data by the factor of paritititons, which can be several orders of magnitude. :/

Also make the source phase make more use of available memory so that we get better parallelism. Fixes #14948

That query is now bound by the sink_parquet, which is still slow.

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Mar 10, 2024
@ritchie46 ritchie46 merged commit f29e3a6 into main Mar 10, 2024
19 checks passed
@ritchie46 ritchie46 deleted the ooc branch March 10, 2024 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sink parquet make the disk full after spilling, even for small data
1 participant