Skip to content

chore: Upgrade rand crate and some other minor crates #14967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

chore: Upgrade rand crate and some other minor crates #14967

wants to merge 15 commits into from
+969 −856

Conversation

comphead
Copy link
Contributor

@comphead comphead commented Mar 3, 2025

Which issue does this PR close?

  • Closes #.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate common Related to common crate execution Related to the execution crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate labels Mar 3, 2025
@comphead comphead marked this pull request as draft March 3, 2025 15:52
@comphead
Copy link
Contributor Author

comphead commented Mar 3, 2025

Looks like some tests relying on random data generation may fail if the rand version changed

@comphead comphead force-pushed the dev branch 4 times, most recently from b6ef148 to 6f3b6c5 Compare March 12, 2025 15:37
@comphead
Copy link
Contributor Author

Depends on apache/arrow-rs#7293

@comphead
Copy link
Contributor Author

Depends on apache/arrow-rs#7126.

@comphead
Copy link
Contributor Author

wasm pack fails because getrandom 0.2.x attached to rand 0.8.x.
wasm requires js feature to be enabled, but the feature renamed from js to wasm-js in getrandom 0.3 and this incosistency in versions feature names seems cannot let wasm to compile.

@comphead
Copy link
Contributor Author

@mbrobbel cc as you working on the migration of arrow-rs

comphead
comphead commented Mar 14, 2025
View reviewed changes
datafusion/core/tests/parquet/filter_pushdown.rs
#[tokio::test]
// The result can be flaky if `rand` crate updated.
//
// Reason: Data generated by
Copy link
Contributor Author

@comphead comphead Mar 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These fuzzy tests based random data gen become non relevant because of rand version upgrade and seems the seed now returns different random values, different from what tests expect in terms of data distribution, column data, etc making the test useless. Considering the Filter Pushdown is important test and conditions in the tests are carefully chosen to cover use cases, we probably materialize the parquet file and get rid of random data gen to avoid test to be stale when rand crate upgraded.

@mbrobbel WDYT?

Copy link
Member

@mbrobbel mbrobbel Mar 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed the same (in #14447) for some different tests, which I fixed by explicitly using rand_chacha: e.g. https://github.com/apache/datafusion/pull/14447/files#diff-4a599584dfc900ec21169f4f820a1b1db46b004b77533dab83a6178d5d3a467eR4253 following https://rust-random.github.io/book/crate-reprod.html.

@comphead
Copy link
Contributor Author

Depends on apache/arrow-rs#7084

@comphead comphead mentioned this pull request Mar 28, 2025
Draft
@comphead
Copy link
Contributor Author

comphead commented Apr 6, 2025

this wasm PR might be related https://github.com/apache/datafusion/pull/15595/files

@github-actions github-actions bot added the catalog Related to the catalog crate label Apr 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mbrobbel mbrobbel mbrobbel left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
catalog Related to the catalog crate common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate execution Related to the execution crate functions Changes to functions implementation physical-expr Changes to the physical-expr crates proto Related to proto crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@comphead @mbrobbel