latest #2
Merged
latest #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a new Optimiser rule PruneApplySourceColumns. This rule, and a project-off rule PruneApplyColumns, togrther provide the same column-pruning capability as PruneUnreferencedOutputs#visitApply().
Changed Domain#simplify to avoid copying internal structures to get size of Ranges or DiscreteValues
Things declared in Hive connector are for hive, so `@ForHive` does not add more information.
Previously the code assumed that every conjunct of `newJoinPredicate` is a `ComparisonExpression`. It's no longer the case.
It allows extending with new group providers by mounting different group-provider.properties files.
This is a fix for #2730. When merging small reads, if the first range and second range are more than 2 GB apart, mergeAdjacentDiskRanges() throw sn ArithmeticException because merging those two ranges is too big to fit in a DiskRange. The correct behavior is to not merge those ranges because this implies the ranges are farther apart than maxReadSizeBytes.
File.getPath() is the canonical way to convert File to its String representation suitable for passing to new File().
Some connectors need better control over various parts of the query.
Deprecate the overload that doesn't take `dynamicFilter` parameter.
`TupleDomain` is not none, so `Domain` is not none too.
Handle none `TupleDomain` on the calling side.
The original condition is equivalent to `isAll`, handled above.
Adds utility classes that enable explicit initialization and management of antlr parser and lexer ATN caches. Without them, these fields are static constants that can grow to retain multiple GB of heap space depending on input query strings.
`PredicatePushDown` may determine a join needs to be inner because of dynamic filter function call. Since dynamic filters pushdown is delayed after join reordering, when this happens the distribution type may be fixed already. For INNER joins we are able to completely remove join condition.
Nested loops operator for cross join does not support column pruning. Before this change, there was a check in the JoinNode constructor that did not allow symbol pruning in the case of a cross join. This change removes the constraint from the JoinNode constructor and adds a sanity check in ValidateDependenciesChecker to make sure that output symbols of cross join contain all input symbols. Delaying the check until post-optimization gives the convenience of creating a column-pruning cross join on intermediate steps of optimization. It simplifies changing JoinNode's parameters such as filter, criteria and type. This change will be completed by set of optimizer rules to ensure that the optimized plan does not contain pruning cross join nodes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.