chore: upgrade to DataFusion 50.0.0, Arrow 56.1.0, Parquet 56.0.0 among others #2286
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2286 +/- ##
============================================
+ Coverage 56.12% 57.51% +1.39%
- Complexity 976 1295 +319
============================================
Files 119 147 +28
Lines 11743 13469 +1726
Branches 2251 2352 +101
============================================
+ Hits 6591 7747 +1156
- Misses 4012 4457 +445
- Partials 1140 1265 +125 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@mbutrovich DataFusion 50 has now been released |
Yep I'm testing this morning and will update the PR. |
mbutrovich
changed the title
mbutrovich
changed the title
|
Note that there's discussion of DF 50.0.1, but I want to unblock a few features and fixes that are waiting on 50, so let's get 50.0.0 in first. |
# Conflicts: # native/Cargo.lock
| comet_hour, | ||
| args, | ||
| field_ref, | ||
| Arc::new(ConfigOptions::default()), |
There was a problem hiding this comment.
Instead of possibly instantiating multiple default ConfigOptions, in the future we stash one somewhere. This would have the benefits of:
- A custom config would propagate throughout
- Reduced memory overhead
There was a problem hiding this comment.
Instead of possibly instantiating multiple default
ConfigOptions, in the future we stash one somewhere. This would have the benefits of:
- A custom config would propagate throughout
- Reduced memory overhead
That would be in the ExecutionContext perhaps?
| fn hash<H: std::hash::Hasher>(&self, state: &mut H) { | ||
| self.name.hash(state); | ||
| self.signature.hash(state); | ||
| (self.stats_type as u8).hash(state); |
There was a problem hiding this comment.
StatsType does not implement Hash, so here we do a cast. std::mem::discriminant is the solution if the enum gets any more complex.
|
|
||
| impl DynEq for RLike { | ||
| fn dyn_eq(&self, other: &dyn Any) -> bool { | ||
| if let Some(other) = other.downcast_ref::<Self>() { |
There was a problem hiding this comment.
The old DynEq implementation did not look at the child. Is that expected?
| } | ||
|
|
||
| impl Hash for RLike { | ||
| fn hash<H: Hasher>(&self, state: &mut H) { |
There was a problem hiding this comment.
The old hash implementation did not hash the child. Is that expected?
|
|
||
| impl Hash for RLike { | ||
| fn hash<H: Hasher>(&self, state: &mut H) { | ||
| self.child.hash(state); |
There was a problem hiding this comment.
The new implementation hashes the child.
mbutrovich
changed the title
mbutrovich
changed the title
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
|
For scalar functions partialEqs the DF relies on builtin mechanism |
mbutrovich
changed the title
mbutrovich
changed the title
mbutrovich
changed the title
5 participants
Which issue does this PR close?
Closes #2343
Rationale for this change
What changes are included in this PR?
cargo updateHow are these changes tested?
Existing tests.