feat: Support directory paths in scans for Parquet, IPC and CSV #17017

nameexhaustion · 2024-06-17T11:50:08Z

This PR enables passing directory paths to scan_(parquet|ipc|csv), which will be recursively traversed to load all files.

There is also some refactoring around hive partition logic to prepare for future behavior changes.

codspeed-hq · 2024-06-17T12:06:55Z

CodSpeed Performance Report

Merging #17017 will not alter performance

_{Comparing nameexhaustion:scan-dir (8786193) with main (915eb08)}

Summary

✅ 37 untouched benchmarks

codecov · 2024-06-18T06:21:34Z

Codecov Report

Attention: Patch coverage is 88.79668% with 27 lines in your changes missing coverage. Please review.

Project coverage is 80.92%. Comparing base (7e19e04) to head (666648e).
Report is 3 commits behind head on main.

❗ Current head 666648e differs from pull request most recent head 8786193

Please upload reports for the commit 8786193 to get more accurate results.

Files	Patch %	Lines
crates/polars-lazy/src/scan/file_list_reader.rs	92.30%	9 Missing ⚠️
crates/polars-plan/src/plans/hive.rs	77.77%	6 Missing ⚠️
crates/polars-lazy/src/scan/csv.rs	72.72%	3 Missing ⚠️
crates/polars-lazy/src/scan/ipc.rs	77.77%	2 Missing ⚠️
crates/polars-lazy/src/scan/parquet.rs	84.61%	2 Missing ⚠️
...es/polars-mem-engine/src/executors/scan/parquet.rs	92.30%	2 Missing ⚠️
crates/polars-plan/src/plans/ir/dot.rs	0.00%	1 Missing ⚠️
py-polars/polars/io/parquet/functions.py	66.66%	1 Missing ⚠️
py-polars/src/lazyframe/visitor/nodes.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17017      +/-   ##
==========================================
- Coverage   80.93%   80.92%   -0.01%     
==========================================
  Files        1448     1448              
  Lines      190704   190710       +6     
  Branches     2723     2723              
==========================================
+ Hits       154338   154340       +2     
- Misses      35862    35866       +4     
  Partials      504      504

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nameexhaustion · 2024-06-18T07:40:31Z

crates/polars-lazy/src/scan/file_list_reader.rs

@@ -104,19 +205,9 @@ pub trait LazyFileListReader: Clone {
        true
    }

-    /// Path of the scanned file.
-    /// It can be potentially a glob pattern.
-    fn path(&self) -> &Path;


Removed the path function from the trait - we just use a length-1 slice in paths for single paths

nameexhaustion · 2024-06-18T07:45:54Z

py-polars/tests/unit/io/test_scan.py

@@ -191,9 +192,6 @@ def data_file(
 def test_scan(
    capfd: Any, monkeypatch: pytest.MonkeyPatch, data_file: _DataFile, force_async: bool
 ) -> None:
-    if data_file.path.suffix == ".csv" and force_async:


drive-by - enable these parametric tests for CSV now it supports async

nameexhaustion · 2024-06-18T07:49:42Z

crates/polars-lazy/src/scan/file_list_reader.rs

    }
+
+    // Todo:
+    // This maintains existing behavior - will remove very soon.


will change this and add tests in a follow-up PR

nameexhaustion · 2024-06-18T07:51:43Z

crates/polars-plan/src/logical_plan/mod.rs

@@ -80,6 +81,7 @@ pub enum DslPlan {
        paths: Arc<[PathBuf]>,
        // Option as this is mostly materialized on the IR phase.
        file_info: Option<FileInfo>,
+        hive_parts: Option<Vec<Arc<HivePartitions>>>,


Moved this out from file_info - we store hive parts for every path in a Vec that we can resolve up-front - this replaces the current approach of using update_hive_partitions.

ritchie46 · 2024-06-18T10:38:27Z

Can you do a rebase? 🙈 The physical engine is moved to its own crate.

nameexhaustion · 2024-06-18T12:07:31Z

rebased 👍

ritchie46

Cool stuff. Left some questions.

crates/polars-lazy/src/scan/csv.rs

crates/polars-lazy/src/scan/file_list_reader.rs

crates/polars-plan/src/plans/hive.rs

…-rs#17017)

github-actions bot added the internal An internal refactor or improvement label Jun 17, 2024

nameexhaustion force-pushed the scan-dir branch 2 times, most recently from 7487f00 to 0d4526b Compare June 18, 2024 05:12

nameexhaustion force-pushed the scan-dir branch from 24c0a7e to 2f061c0 Compare June 18, 2024 07:33

nameexhaustion changed the title ~~test directory scans~~ feat: Support directory paths in scans for Parquet, IPC and CSV Jun 18, 2024

github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Jun 18, 2024

nameexhaustion commented Jun 18, 2024

View reviewed changes

nameexhaustion marked this pull request as ready for review June 18, 2024 07:53

nameexhaustion requested review from ritchie46, stinodego, c-peters, alexander-beedie, MarcoGorelli, reswqa and orlp as code owners June 18, 2024 07:53

c

72c0a2b

nameexhaustion force-pushed the scan-dir branch from 5e1ca5c to 72c0a2b Compare June 18, 2024 11:40

c

666648e

ritchie46 reviewed Jun 18, 2024

View reviewed changes

crates/polars-lazy/src/scan/csv.rs Show resolved Hide resolved

crates/polars-lazy/src/scan/file_list_reader.rs Show resolved Hide resolved

crates/polars-lazy/src/scan/file_list_reader.rs Outdated Show resolved Hide resolved

ritchie46 reviewed Jun 18, 2024

View reviewed changes

crates/polars-plan/src/plans/hive.rs Show resolved Hide resolved

add comments

8786193

ritchie46 merged commit 306a918 into pola-rs:main Jun 18, 2024
26 checks passed

nameexhaustion deleted the scan-dir branch June 19, 2024 01:33

nameexhaustion mentioned this pull request Jun 19, 2024

[Python] Scan_parquet() should be able to accept directory as input #14342

Closed

Wouittone pushed a commit to Wouittone/polars that referenced this pull request Jun 22, 2024

feat: Support directory paths in scans for Parquet, IPC and CSV (pola…

f0b3421

…-rs#17017)

nameexhaustion self-assigned this Jun 24, 2024

c-peters added the accepted Ready for implementation label Jun 24, 2024

stinodego mentioned this pull request Jul 2, 2024

Error when passing directory path of hive-partitioned parquet dataset to scan_parquet #17362

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support directory paths in scans for Parquet, IPC and CSV #17017

feat: Support directory paths in scans for Parquet, IPC and CSV #17017

nameexhaustion commented Jun 17, 2024 •

edited

Loading

codspeed-hq bot commented Jun 17, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading

nameexhaustion Jun 18, 2024

nameexhaustion Jun 18, 2024 •

edited

Loading

nameexhaustion Jun 18, 2024

nameexhaustion Jun 18, 2024

ritchie46 commented Jun 18, 2024

nameexhaustion commented Jun 18, 2024

ritchie46 left a comment

feat: Support directory paths in scans for Parquet, IPC and CSV #17017

feat: Support directory paths in scans for Parquet, IPC and CSV #17017

Conversation

nameexhaustion commented Jun 17, 2024 • edited Loading

codspeed-hq bot commented Jun 17, 2024 • edited Loading

CodSpeed Performance Report

Merging #17017 will not alter performance

Summary

codecov bot commented Jun 18, 2024 • edited Loading

Codecov Report

nameexhaustion Jun 18, 2024

Choose a reason for hiding this comment

nameexhaustion Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

nameexhaustion Jun 18, 2024

Choose a reason for hiding this comment

nameexhaustion Jun 18, 2024

Choose a reason for hiding this comment

ritchie46 commented Jun 18, 2024

nameexhaustion commented Jun 18, 2024

ritchie46 left a comment

Choose a reason for hiding this comment

nameexhaustion commented Jun 17, 2024 •

edited

Loading

codspeed-hq bot commented Jun 17, 2024 •

edited

Loading

codecov bot commented Jun 18, 2024 •

edited

Loading

nameexhaustion Jun 18, 2024 •

edited

Loading