Skip to content

Comments

Expose table scan progress to DuckDB#6596

Merged
myrrc merged 1 commit intodevelopfrom
myrrc/duckdb-progress
Feb 20, 2026
Merged

Expose table scan progress to DuckDB#6596
myrrc merged 1 commit intodevelopfrom
myrrc/duckdb-progress

Conversation

@myrrc
Copy link
Contributor

@myrrc myrrc commented Feb 19, 2026

Add table_scan_progress in duckdb's table function vtable. Return sum of read
(exported) chunks' length divided by sum of all chunks' length.

Unrelated changes:

  • panic in duckdb-bench if no output formats were provided
  • add a note how to set up duckdb-vortex's full testing with vortex-duckdb.

Testing:

  • Generate .vortex files: cargo run --bin duckdb-bench -- tpch -i1 --opt scale-factor=10.0 --formats vortex
  1. Build duckdb vortex extension from this PR
  2. Verify these queries show progress bar

PRAGMA enable_progress_bar;
SET progress_bar_time = 10;
select * from read_vortex("/path/to/vortex/vortex-bench/data/tpch/10.0/vortex-file-compressed/l*.vortex");

@0ax1 0ax1 self-requested a review February 19, 2026 10:40
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from 1dc73bc to 989e43b Compare February 19, 2026 10:48
@myrrc myrrc added the changelog/feature A new feature label Feb 19, 2026
@myrrc myrrc changed the title Duckdb progress Expose table scan progress to DuckDB Feb 19, 2026
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from e9bc935 to d7e5c3b Compare February 19, 2026 17:27
@myrrc myrrc marked this pull request as ready for review February 19, 2026 17:27
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from d7e5c3b to 575e591 Compare February 19, 2026 17:27
@myrrc myrrc requested review from 0ax1 and joseph-isaacs February 19, 2026 17:27
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from 575e591 to 1547002 Compare February 19, 2026 17:33
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from 1547002 to c69c86b Compare February 20, 2026 10:33
@myrrc
Copy link
Contributor Author

myrrc commented Feb 20, 2026 via email

@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from c69c86b to 2303af8 Compare February 20, 2026 10:39
@myrrc myrrc enabled auto-merge (squash) February 20, 2026 10:40
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from 2303af8 to 5ae24d7 Compare February 20, 2026 10:48
Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc myrrc force-pushed the myrrc/duckdb-progress branch from 5ae24d7 to 4fd7dca Compare February 20, 2026 11:09
@myrrc myrrc merged commit 5f8bbe8 into develop Feb 20, 2026
49 checks passed
@myrrc myrrc deleted the myrrc/duckdb-progress branch February 20, 2026 11:23
let has_more_data = exporter.export(chunk)?;
global_state
.bytes_read
.fetch_add(chunk.len(), Ordering::Relaxed);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is wrong right? The variable is called bytes_read, but we're adding the row count of the chunk.

We're also adding this after pulling the chunk? So presumably the progress just hovers around or just below 100% the entire time?

I think we need to figure out a better measure of progress using the Scan API after #6652

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants