Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -884,12 +884,16 @@ config_namespace! {
/// into the file scan phase.
pub enable_join_dynamic_filter_pushdown: bool, default = true

/// When set to true attempts to push down dynamic filters generated by operators (topk & join) into the file scan phase.
/// When set to true, the optimizer will attempt to push down Aggregate dynamic filters
/// into the file scan phase.
pub enable_aggregate_dynamic_filter_pushdown: bool, default = true

/// When set to true attempts to push down dynamic filters generated by operators (TopK, Join & Aggregate) into the file scan phase.
/// For example, for a query such as `SELECT * FROM t ORDER BY timestamp DESC LIMIT 10`, the optimizer
/// will attempt to push down the current top 10 timestamps that the TopK operator references into the file scans.
/// This means that if we already have 10 timestamps in the year 2025
/// any files that only have timestamps in the year 2024 can be skipped / pruned at various stages in the scan.
/// The config will suppress `enable_join_dynamic_filter_pushdown` & `enable_topk_dynamic_filter_pushdown`
/// The config will suppress `enable_join_dynamic_filter_pushdown`, `enable_topk_dynamic_filter_pushdown` & `enable_aggregate_dynamic_filter_pushdown`
/// So if you disable `enable_topk_dynamic_filter_pushdown`, then enable `enable_dynamic_filter_pushdown`, the `enable_topk_dynamic_filter_pushdown` will be overridden.
pub enable_dynamic_filter_pushdown: bool, default = true

Expand Down Expand Up @@ -1201,6 +1205,7 @@ impl ConfigOptions {
self.optimizer.enable_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_topk_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_join_dynamic_filter_pushdown = bool_value;
self.optimizer.enable_aggregate_dynamic_filter_pushdown = bool_value;
}
return Ok(());
}
Expand Down
Loading