Skip to content

Commit 6ed2e84

Browse files
tshauckalamb
andauthored
docs: add section on supports_filters_pushdown (#7680)
* docs: add section on supports_filters_pushdown * fix: fixup code * Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * fix: apply prettier --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
1 parent fcd94fb commit 6ed2e84

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

docs/source/library-user-guide/custom-table-providers.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,22 @@ impl TableProvider for CustomDataSource {
121121

122122
With this, and the implementation of the omitted methods, we can now use the `CustomDataSource` as a `TableProvider` in DataFusion.
123123

124+
##### Additional `TableProvider` Methods
125+
126+
`scan` has no default implementation, so it needed to be written. There are other methods on the `TableProvider` that have default implementations, but can be overridden if needed to provide additional functionality.
127+
128+
###### `supports_filters_pushdown`
129+
130+
The `supports_filters_pushdown` method can be overridden to indicate which filter expressions support being pushed down to the data source and within that the specificity of the pushdown.
131+
132+
This returns a `Vec` of `TableProviderFilterPushDown` enums where each enum represents a filter that can be pushed down. The `TableProviderFilterPushDown` enum has three variants:
133+
134+
- `TableProviderFilterPushDown::Unsupported` - the filter cannot be pushed down
135+
- `TableProviderFilterPushDown::Exact` - the filter can be pushed down and the data source can guarantee that the filter will be applied completely to all rows. This is the highest performance option.
136+
- `TableProviderFilterPushDown::Inexact` - the filter can be pushed down, but the data source cannot guarantee that the filter will be applied to all rows. DataFusion will apply `Inexact` filters again after the scan to ensure correctness.
137+
138+
For filters that can be pushed down, they'll be passed to the `scan` method as the `filters` parameter and they can be made use of there.
139+
124140
## Using the Custom Table Provider
125141

126142
In order to use the custom table provider, we need to register it with DataFusion. This is done by creating a `TableProvider` and registering it with the `ExecutionContext`.

0 commit comments

Comments
 (0)