[SPARK-51883][DOCS][PYTHON] Python Data Source user guide for filter pushdown #50684
Conversation
wengh
changed the title
|
|
||
| Other methods such as DataSource.schema() and DataSourceStreamReader.latestOffset() can be stateful. Changes to the object state made in these methods are visible to future invocations. | ||
|
|
||
| Refer to the documentation of each method for more details. |
There was a problem hiding this comment.
Can we also link to the documentation here?
There was a problem hiding this comment.
the method documentations don't actually mention whether it can change state :(
There was a problem hiding this comment.
most methods can change state so I guess we can just list the exceptions here
There was a problem hiding this comment.
The following methods should not mutate internal state. Changes to the object state made in these methods are not guaranteed to be visible or invisible to future calls.
- DataSourceReader.partitions()
- DataSourceReader.read()
- DataSourceStreamReader.read()
- SimpleDataSourceStreamReader.readBetweenOffsets()
- All writer methods
All other methods such as DataSource.schema() and DataSourceStreamReader.latestOffset() can be stateful. Changes to the object state made in these methods are visible to future calls.
|
|
||
| from pyspark.sql.datasource import EqualTo, Filter, GreaterThan, LessThan | ||
|
|
||
| def pushFilters(self, filters: List[Filter]) -> Iterable[Filter]: |
There was a problem hiding this comment.
Can we add a complete example here so that people can copy paste and try it out?
There was a problem hiding this comment.
Changed to an example source that returns prime numbers sequentially
There was a problem hiding this comment.
@wengh this looks good! Could you also attach a screenshot for the updated user guide to the PR description? Thanks!
done 🤗 |
|
Would you rebase master changes? The CI test failure should have been fixed |
What changes were proposed in this pull request?
Update
python_data_source.rstto add filter pushdown docs.Why are the changes needed?
Feature was added but documentation was still missing.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Verified locally
Was this patch authored or co-authored using generative AI tooling?
Yes. Initial draft was generated using AI then manually edited.
Generated-by: GitHub Copilot with Claude 3.7 Sonnet