Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add parquet source to python ffi #749

Merged
merged 19 commits into from
Sep 17, 2023
Merged

feat: add parquet source to python ffi #749

Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
cb577ec
Pass the object registry through
jordanrfrazier Sep 7, 2023
ca7032c
Wrap file sets in arc to mutate it in table.rs and data context
jordanrfrazier Sep 7, 2023
d039f54
pass source through table initialization
jordanrfrazier Sep 8, 2023
ac2b087
Add concurrent file sets
jordanrfrazier Sep 11, 2023
c5c8289
Cleanup a bit and fix add_parquet ffi function
jordanrfrazier Sep 12, 2023
5951561
Use full path in python side of add_parquet
jordanrfrazier Sep 13, 2023
cdf03d3
update scan to use file sets, and append to the tableinfo's concurren…
jordanrfrazier Sep 13, 2023
2d0388c
update key hash inverse for parquet files in the ffi table
jordanrfrazier Sep 14, 2023
48bded6
cleanup and use path instead of string
jordanrfrazier Sep 14, 2023
58f18ed
lint
jordanrfrazier Sep 14, 2023
34dcc55
pydoc style
jordanrfrazier Sep 14, 2023
d19f4dc
Fix test
jordanrfrazier Sep 14, 2023
b2cc9d1
format
jordanrfrazier Sep 14, 2023
436ddec
Rename to file sets
jordanrfrazier Sep 14, 2023
6ba0b53
Move arc to wrap the filesets
jordanrfrazier Sep 15, 2023
9c70f18
Attempt to use parking_lot for the mutex guard, but it's not send+sync
jordanrfrazier Sep 15, 2023
5397cde
fix guard
bjchambers Sep 15, 2023
26c2098
remove prepare version from path
jordanrfrazier Sep 17, 2023
438a3e7
gs
jordanrfrazier Sep 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix guard
  • Loading branch information
@bjchambers
bjchambers committed Sep 15, 2023
commit 5397cde3a5980bdb9048ec493f03840619c043d8
32 changes: 17 additions & 15 deletions crates/sparrow-runtime/src/execute/operation/scan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,22 +234,24 @@ impl ScanOperation {
//
// Ideally, we don't support both paths.
if table_source.is_none() && !table_info.file_sets.is_empty() {
let prepared_files = table_info
.file_sets
.prepared_files(&None)
.into_report()
.change_context(Error::Internal("failed to get prepared files"))?;
{
let prepared_files = table_info
.file_sets
.prepared_files(&None)
.into_report()
.change_context(Error::Internal("failed to get prepared files"))?;

// Send initial progress information.
let total_num_rows = prepared_files
.iter()
.map(|file| file.num_rows as usize)
.sum();
context
.progress_updates_tx
.try_send(ProgressUpdate::InputMetadata { total_num_rows })
.into_report()
.change_context(Error::internal())?;
// Send initial progress information.
let total_num_rows = prepared_files
.iter()
.map(|file| file.num_rows as usize)
.sum();
context
.progress_updates_tx
.try_send(ProgressUpdate::InputMetadata { total_num_rows })
.into_report()
.change_context(Error::internal())?;
}

let input_stream = table_reader(
&context.object_stores,
Expand Down