Skip to content

Conversation

@timsaucer
Copy link
Member

This is follow on work to #12147 .

This adds explicit tests to see what data come back from our push down filters.

The difference between this and the e2e tests is that the e2e tests will check the result of the full output from datafusion collection. That will go through the FilterExec after our table provider, so the data filtered could be done by datafusion rather than our table provider.

We cannot mark our table provider as Exact because it will send back chunks that have more data than was requested.

This test will get record batches directly from our table provider without going through any of the other mechanisms of DataFusion. We can verify we get exactly the chunks we expect from the server and no more.

In addition to these checks I am adding a client util. Since the TableProvider requires a ConnectionClient we need a little more than just the server hook.

Also I updated one method in the TableProvider that removes a block_on call that makes integration with testing difficult.

@timsaucer timsaucer added the do-not-merge Do not merge this PR label Dec 19, 2025
@github-actions
Copy link

github-actions bot commented Dec 19, 2025

Web viewer built successfully.

Result Commit Link Manifest
2c85318 https://rerun.io/viewer/pr/12285 +nightly +main

View image diff on kitdiff.

Note: This comment is updated whenever you push a commit.

@timsaucer timsaucer force-pushed the tsaucer/pushdown-redap-test branch from 2b4d400 to a720fde Compare January 5, 2026 16:36
@timsaucer timsaucer added 🔨 testing testing and benchmarks exclude from changelog PRs with this won't show up in CHANGELOG.md and removed do-not-merge Do not merge this PR labels Jan 5, 2026
@timsaucer timsaucer force-pushed the tsaucer/pushdown-redap-test branch from c792a8a to 2c85318 Compare January 6, 2026 17:10
@timsaucer
Copy link
Member Author

timsaucer commented Jan 6, 2026

Since I ended up with a fair amount of scope creep on this PR, I'm going to open another one with just the updates to the OSS server and existing tests. Then this PR will rebase with only the new tests.

#12328

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exclude from changelog PRs with this won't show up in CHANGELOG.md OSS-server 🔨 testing testing and benchmarks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants