perf: Adding support for LatestBaseFilesPathFilter to Spark File Index by suryaprasanna · Pull Request #18136 · apache/hudi

suryaprasanna · 2026-02-08T23:27:45Z

Describe the issue this Pull Request addresses

This PR adds an opt-in path filtering mechanism during file listing to prevent Spark driver OOM errors when querying large Hudi datasets with multiple file versions per partition.

Problem: When file listing is performed without filtering, all file versions (including older ones) are loaded into driver memory, causing OOM on large tables.

Solution: Added a new config hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter (default: false) that enables HoodieROPathFilter during file listing to exclude older file versions.

Summary and Changelog

Users can now enable path filtering during file listing to avoid loading multiple file versions into memory on the driver. This is controlled by the new config hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter.

Changes:

New Config: FILE_INDEX_LIST_FILE_STATUSES_USING_RO_PATH_FILTER (default: false)
- Enables path filtering during file listing to reduce driver memory pressure
- Filters out older file versions, keeping only the latest files needed for queries
API Extensions:
- Extended HoodieTableMetadata, BaseTableMetadata, and FileSystemBackedTableMetadata to accept optional StoragePathFilter parameter
- Added FSUtils.getAllDataFilesInPartition overload with path filter support
- Created HoodieROTableStoragePathFilter wrapper to adapt Hadoop PathFilter to Hudi's StoragePathFilter interface
Spark Integration:
- Updated BaseHoodieTableFileIndex to use path filter when enabled
- Modified SparkHoodieTableFileIndex to apply HoodieROTablePathFilter during partition listing

Impact

Config Changes:

New config: hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter (default: false)
- When enabled, applies HoodieROTablePathFilter during file listing to filter out older file versions
- Helps prevent OOM issues on driver for large tables with multiple file versions

Performance:

Reduces memory pressure on Spark driver for datasets with multiple file versions per partition
Enables successful queries on very large tables that previously failed with OOM errors

Risk Level

Low - The feature is behind a config flag (default: false) and does not change existing behavior unless explicitly enabled.

Verification:

Existing unit tests pass
Tested on large production datasets with OOM issues - queries now succeed with filter enabled

Documentation Update

Config documentation is included in the withDocumentation method of the new config property.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

suryaprasanna · 2026-02-09T16:53:18Z

@nsivabalan seems like the checks are not running on the PR. Can you please chek?

nsivabalan · 2026-02-09T23:54:30Z

can we consider both table types, and all query types and ensure we wire in the config only wherever applicable.

suryaprasanna · 2026-02-09T23:54:19Z

@@ -143,6 +146,7 @@ public abstract class BaseHoodieTableFileIndex implements AutoCloseable {
   * @param configProperties             unifying configuration (in the form of generic properties)
   * @param queryType                    target query type
   * @param queryPaths                   target DFS paths being queried


Add MOR condition.

Added MOR table type condition.

nsivabalan · 2026-02-11T03:18:03Z

        .fromProperties(configProperties)
        .enable(configProperties.getBoolean(ENABLE.key(), DEFAULT_METADATA_ENABLE_FOR_READERS)
-            && HoodieTableMetadataUtil.isFilesPartitionAvailable(metaClient))
+            && HoodieTableMetadataUtil.isFilesPartitionAvailable(metaClient)


hey @yihua : lets chat about this as well tomorrow.

nsivabalan · 2026-02-11T03:29:26Z

+
+    if (useROPathFilterForListing && !shouldIncludePendingCommits) {
+      // Group files by partition path, then by file group ID
+      Map<String, PartitionPath> partitionsMap = new HashMap<>();


can we move this to a private method.
generatePartitionFileSlicesPostROTablePathFilter

nsivabalan · 2026-02-11T03:30:44Z

   * By passing metaClient and completedTimeline, we can sync the view seen from this class against HoodieFileIndex class
   */
-  public HoodieROTablePathFilter(Configuration conf,
+  public HoodieROTablePathFilter(StorageConfiguration conf,


hey @yihua : can you review the changes in this patch

nsivabalan · 2026-02-11T03:31:41Z

        " them (if possible).")

+  val FILE_INDEX_LIST_FILE_STATUSES_USING_RO_PATH_FILTER: ConfigProperty[Boolean] =
+    ConfigProperty.key("hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter")


hoodie.datasource.read.file.index.optimize.listing.using.path.filter

Made the change.

nsivabalan · 2026-02-11T03:32:30Z

      properties.setProperty(DataSourceReadOptions.FILE_INDEX_LISTING_MODE_OVERRIDE.key, listingModeOverride)
    }

+    var hoodieROTablePathFilterBasedFileListingEnabled = getConfigValue(options, sqlConf,


once we fix the config key, lets fix these vars as well

Yes, updated.

nsivabalan · 2026-02-11T03:35:51Z

+        val result = spark.sql(s"select id, name, price, ts from $tableName order by id").collect()
+        // Should have deleted records where id % 3 = 0 (3, 6, 9)
+        // Should have doubled price for even ids (2, 4, 8, 10)
+        assert(result.length == 7) // 10 - 3 deleted = 7


do you think below assertion would work.

we can rename one of the earlier versions of a file slice so that HoodieBaseFile parsing will fail.
so, if RO table path filter works as intended, listing files from a given partition should not fail, since we won't even try to parse the file.

but if RO table path filter did not work, it would fail.

@suryaprasanna : did you get a chance to address this?

HoodieROTablePathFilter also creates FSV, so renaming the older file slice will also fail the latestBaseFiles API right?

gotcha. sg.

yihua

Nice work on adding an opt-in path filter to avoid driver OOM during file listing — the feature is well-motivated and the config is cleanly gated behind a default-off flag. The main concerns are correctness issues in the new loadFileSlicesForPartitions fast path: the partition path passed to FileSlice appears to be absolute rather than relative, and the partition map lookup can NPE if the key doesn't match. It's also worth clarifying MOR table compatibility and fixing the shared Hadoop config mutation in getPartitionPathFilter before merging.

yihua · 2026-02-10T00:01:20Z

+      Map<String, PartitionPath> partitionsMap = new HashMap<>();
+      partitions.forEach(p -> partitionsMap.put(p.path, p));
+      Map<PartitionPath, List<FileSlice>> partitionToFileSlices = new HashMap<>();
+


The partitionPathStr here is the absolute path (pathInfo.getPath().getParent().toString()), but FileSlice expects a relative partition path. The existing code path via HoodieTableFileSystemView always uses relative paths. This would cause mismatches downstream wherever FileSlice.getPartitionPath() is used. Should this be relPartitionPath instead?

@yihua PartitionPath object stores only relative partition path str, not absolute paths. Which code path are you referring it?

yihua · 2026-02-10T00:01:20Z

+        // Create FileSlice obj from StoragePathInfo.
+        String partitionPathStr = pathInfo.getPath().getParent().toString();
+        String relPartitionPath = FSUtils.getRelativePartitionPath(basePath, pathInfo.getPath().getParent());
+        HoodieBaseFile baseFile = new HoodieBaseFile(pathInfo);


If relPartitionPath doesn't exactly match a key in partitionsMap, partitionPathObj will be null and the computeIfAbsent call below will throw NPE. This could happen with path normalization differences (trailing slashes, scheme differences). Could you add a null check or use getRelativePartitionPath consistently with how PartitionPath.path was originally set?

yihua · 2026-02-10T00:01:20Z

+    List<StoragePathInfo> allFiles = listPartitionPathFiles(partitions, activeTimeline);
+    log.info("On {} with query instant as {}, it took {}ms to list all files {} Hudi partitions",
+        metaClient.getTableConfig().getTableName(), queryInstant.map(instant -> instant).orElse("N/A"),
+        timer.endTimer(), partitions.size());


Have you considered what happens with MOR tables here? The HoodieROTablePathFilter only returns base files (it calls fsView.getLatestBaseFiles()), so this path constructs FileSlices without log files. The !shouldIncludePendingCommits guard doesn't prevent MOR tables from reaching this code. It might be worth adding a table-type check (COW only) or documenting this limitation.

Added table type equals COW condition.

yihua · 2026-02-10T00:01:20Z

@@ -146,6 +147,12 @@ public List<StoragePathInfo> getAllFilesInPartition(StoragePath partitionPath) t
  }

  @Override


It looks like the @Override annotation that belonged to getAllFilesInPartitions(Collection<String>) has been absorbed by the new method insertion. In the diff, the @Override on line 148 now applies to the new two-arg overload, while the original single-arg method (which is the actual interface abstract method) loses its @Override. Could you add @Override back to the original method?

Yes my bad, it is a refactoring mistake, fixed it now.

yihua · 2026-02-11T22:38:39Z

+    return getAllFilesInPartitions(partitions);
+  }
+
  public Map<String, List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitions)


nit: Let this call getAllFilesInPartitions(partitions, Option.empty()) to be easier to read?

Yeah, makes sense. Refactored the code accordingly.

yihua · 2026-02-11T22:40:13Z

  Map<String, List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitionPaths)
      throws IOException;

+  default Map<String, List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitionPaths,
+                                                                      Option<StoragePathFilter> pathFilterOption)
+      throws IOException {
+    return getAllFilesInPartitions(partitionPaths);
+  }


Make Map<String, List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitionPaths) to have default implementation of getAllFilesInPartitions(partitionPaths, Option.empty()) so subclasses can avoid the repeating code? Then getAllFilesInPartitions(Collection<String> partitionPaths, Option<StoragePathFilter> pathFilterOption) becomes an abstract method.

Good idea, made the suggested code changes.

yihua · 2026-02-11T22:42:27Z

+    HoodieTimer timer = HoodieTimer.start();
+    List<StoragePathInfo> allFiles = listPartitionPathFiles(partitions, activeTimeline);
+    log.info("On {} with query instant as {}, it took {}ms to list all files {} Hudi partitions",
+        metaClient.getTableConfig().getTableName(), queryInstant.map(instant -> instant).orElse("N/A"),


nit: queryInstant.map(instant -> instant).orElse("N/A") — the .map(instant -> instant) is a no-op. You can simplify to queryInstant.orElse("N/A").

Made the suggested change.

wasn't the suggestion to go w/
queryInstant.orElse("N/A")

My bad, fixed it now, please check.

yihua · 2026-02-11T22:52:30Z


  public HoodieROTablePathFilter() {
-    this(new Configuration());
+    this(HadoopFSUtils.getStorageConf());


HoodieROTablePathFilter and BaseFileOnlyRelation should no longer be used based on the latest master; instead, HoodieCopyOnWriteSnapshotHadoopFsRelationFactory is used.

+1

lets TAL at all implementations extending from HoodieBaseHadoopFsRelationFactory and we write them in

@yihua HoodieCopyOnWriteSnapshotHadoopFsRelationFactory uses HoodieFileIndex. To optimize the file system view calls in HoodieFileIndex we are using HoodieROPathFilter. I think originally HoodieROPathFilter is used at a Relation level, now we downgraded to PathFilter level. So, with the current setup it should be fine right?

yihua

A better and general approach would be adding a file system view based on the latest snapshot only to limit the size of file slices in memory, which is used by the file index. That should solve the problem with better layering.

nsivabalan · 2026-02-11T23:49:20Z

+    return getAllDataFilesInPartition(storage, partitionPath, Option.empty());
+  }
+
+  public static List<StoragePathInfo> getAllDataFilesInPartition(HoodieStorage storage,


we might need to change the naming, now that its not all files.

Renamed it to getAllDataFilesInPartitionByPathFilter

nsivabalan

lets add tests for time travel query as well

nsivabalan · 2026-02-12T19:23:14Z

+        // Add the FileSlice to partitionToFileSlices
+        PartitionPath partitionPathObj = partitionsMap.get(relPartitionPath);
+        List<FileSlice> fileSlices = partitionToFileSlices.computeIfAbsent(partitionPathObj, k -> new ArrayList<>());
+        fileSlices.add(fileSlice);


can we avoid this special handling.
lets route all the files into FSV.
so that we maintain one flow for all cases.

just that the the input files could have already been filtered (if path filter is applied), or could be referring to all files(if no path filter).
much simpler from maintainability standpoint.

Creating a file system view is costly, we have actually created fsv via ropathfilter.
Adding fsv again again here means we are doing it twice so we should avoid it here.

Yes, I see that, but building FSV here is not invoking distributed spark context and there is no metadata table involved (NoOpTableMetadata is used)). its happening in driver and we have just 1 file slice per file group. So, constructing should not add much overhead.

On the plus side, we dont' need to maintain additional custom code for, when path filter is enabled.

I think this is a tradeoff between simplicity and performance. Also, I feel creating FSV on a bunch of files itself is very error prone.
We should skip creating another FSV that way later if any more logic added to it, we dont need to go through all that logic.

nsivabalan · 2026-02-12T19:24:41Z


  public HoodieROTablePathFilter() {
-    this(new Configuration());
+    this(HadoopFSUtils.getStorageConf());


+1

lets TAL at all implementations extending from HoodieBaseHadoopFsRelationFactory and we write them in

nsivabalan · 2026-02-12T19:28:27Z

A better and general approach would be adding a file system view based on the latest snapshot only to limit the size of file slices in memory, which is used by the file index. That should solve the problem with better layering.

Hey @yihua : based on latest state of the patch, I feel it nicely sits w/n HoodieTableMetadata and so, we can leverage this w/ any of FSV.
Just that I see some special handling of FileIndex after filtering which we can avoid (shared feedback above).
otherwise, current layering seems ok to me.

nsivabalan · 2026-02-24T19:56:49Z

-  private final boolean shouldIncludePendingCommits;
+  protected final boolean shouldIncludePendingCommits;
  private final boolean shouldValidateInstant;
+  protected final boolean useROPathFilterForListing;


useLatestBasePathFilterForListing

Sure, renamed the variable.

nsivabalan · 2026-02-24T20:00:10Z

+    HoodieTimer timer = HoodieTimer.start();
+    List<StoragePathInfo> allFiles = listPartitionPathFiles(partitions, activeTimeline);
+    log.info("On {} with query instant as {}, it took {}ms to list all files {} Hudi partitions",
+        metaClient.getTableConfig().getTableName(), queryInstant.map(instant -> instant).orElse("N/A"),


wasn't the suggestion to go w/
queryInstant.orElse("N/A")

nsivabalan · 2026-02-24T20:00:54Z

+    // For MOR tables, we need log files which are not returned by HoodieROTablePathFilter
+    if (useROPathFilterForListing
+        && !shouldIncludePendingCommits
+        && metaClient.getTableConfig().getTableType() == HoodieTableType.COPY_ON_WRITE) {


for MOR, if query type is RO, we could also afford to add the filtering right?

Sure, also adding MOR with RO as well.

nsivabalan · 2026-02-24T20:03:15Z

+        // Add the FileSlice to partitionToFileSlices
+        PartitionPath partitionPathObj = partitionsMap.get(relPartitionPath);
+        List<FileSlice> fileSlices = partitionToFileSlices.computeIfAbsent(partitionPathObj, k -> new ArrayList<>());
+        fileSlices.add(fileSlice);


Yes, I see that, but building FSV here is not invoking distributed spark context and there is no metadata table involved (NoOpTableMetadata is used)). its happening in driver and we have just 1 file slice per file group. So, constructing should not add much overhead.

On the plus side, we dont' need to maintain additional custom code for, when path filter is enabled.

nsivabalan · 2026-02-24T20:07:50Z

+    } else {
+      // Also allow passing in the path filter config via Spark session conf for convenience
+      pathFilterOptimizedListingEnabled = getConfigValue(options, sqlConf,
+        "spark." + DataSourceReadOptions.FILE_INDEX_LIST_FILE_STATUSES_USING_RO_PATH_FILTER.key, null)


shouldn't your other patch fix this already. we should fix all config key prefix stripping at some root level and not litter across the code base.
#18205

Yes, we also need to clean other configs as well. Let me take that as a followup. I would like to get this landed soon so the other PRs can be unblocked.

nsivabalan · 2026-02-24T20:09:05Z

+        val result = spark.sql(s"select id, name, price, ts from $tableName order by id").collect()
+        // Should have deleted records where id % 3 = 0 (3, 6, 9)
+        // Should have doubled price for even ids (2, 4, 8, 10)
+        assert(result.length == 7) // 10 - 3 deleted = 7


@suryaprasanna : did you get a chance to address this?

nsivabalan · 2026-02-24T20:09:34Z

+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.storage.StoragePathFilter;
+
+public class HoodieROTableStoragePathFilter implements StoragePathFilter {


how about HoodieLatestBaseFilePathFilter

Sure, renaming the file.

…on the driver Reviewers: O955 Project Hoodie Project Reviewer: Add blocking reviewers, pwason, jingli, meenalb, singh.sumit Reviewed By: O955 Project Hoodie Project Reviewer: Add blocking reviewers, pwason Tags: #has_java JIRA Issues: HUDI-6646 Differential Revision: https://code.uberinternal.com/D17441111 Fix build failures Fix checkstyle Refactor code Create unit tests

hudi-bot · 2026-02-25T08:18:36Z

CI report:

8d3be59 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan · 2026-02-25T21:58:57Z

+        val result = spark.sql(s"select id, name, price, ts from $tableName order by id").collect()
+        // Should have deleted records where id % 3 = 0 (3, 6, 9)
+        // Should have doubled price for even ids (2, 4, 8, 10)
+        assert(result.length == 7) // 10 - 3 deleted = 7


gotcha. sg.

nsivabalan · 2026-02-25T22:00:15Z

hey @suryaprasanna : can you fix the title and PR desc based on latest.
Essentially, Adding support for LatestBaseFilesPathFilter to Spark File Index

nsivabalan · 2026-02-26T16:47:41Z

oh.
the PR desc is not fixed for the right config key
hoodie.datasource.read.file.index.optimize.listing.using.path.filter

apache#18136) This PR adds an opt-in path filtering mechanism during file listing to prevent Spark driver OOM errors when querying large Hudi datasets with multiple file versions per partition. Problem: When file listing is performed without filtering, all file versions (including older ones) are loaded into driver memory, causing OOM on large tables. Solution: Added a new config hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter (default: false) that enables HoodieROPathFilter during file listing to exclude older file versions. Summary and Changelog Users can now enable path filtering during file listing to avoid loading multiple file versions into memory on the driver. This is controlled by the new config hoodie.datasource.read.file.index.list.file.statuses.using.ro.path.filter. Changes: New Config: FILE_INDEX_LIST_FILE_STATUSES_USING_RO_PATH_FILTER (default: false) Enables path filtering during file listing to reduce driver memory pressure Filters out older file versions, keeping only the latest files needed for queries API Extensions: Extended HoodieTableMetadata, BaseTableMetadata, and FileSystemBackedTableMetadata to accept optional StoragePathFilter parameter Added FSUtils.getAllDataFilesInPartition overload with path filter support Created HoodieROTableStoragePathFilter wrapper to adapt Hadoop PathFilter to Hudi's StoragePathFilter interface Spark Integration: Updated BaseHoodieTableFileIndex to use path filter when enabled Modified SparkHoodieTableFileIndex to apply HoodieROTablePathFilter during partition listing

codecov-commenter · 2026-03-28T09:40:38Z

Codecov Report

❌ Patch coverage is 40.54054% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.28%. Comparing base (7f4dfda) to head (8d3be59).
⚠️ Report is 77 commits behind head on master.

Files with missing lines	Patch %	Lines
...java/org/apache/hudi/BaseHoodieTableFileIndex.java	36.36%	19 Missing and 2 partials ⚠️
...e/hudi/hadoop/HoodieLatestBaseFilesPathFilter.java	0.00%	7 Missing ⚠️
...c/main/scala/org/apache/hudi/HoodieFileIndex.scala	40.00%	4 Missing and 2 partials ⚠️
...la/org/apache/hudi/SparkHoodieTableFileIndex.scala	33.33%	5 Missing and 1 partial ⚠️
...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java	50.00%	2 Missing ⚠️
...c/main/java/org/apache/hudi/common/fs/FSUtils.java	66.66%	0 Missing and 1 partial ⚠️
...ache/hudi/common/table/view/NoOpTableMetadata.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18136      +/-   ##
============================================
- Coverage     57.29%   57.28%   -0.02%     
- Complexity    18555    18561       +6     
============================================
  Files          1945     1946       +1     
  Lines        106199   106262      +63     
  Branches      13128    13135       +7     
============================================
+ Hits          60852    60869      +17     
- Misses        39621    39662      +41     
- Partials       5726     5731       +5

Flag	Coverage Δ
hadoop-mr-java-client	`45.41% <36.00%> (-0.02%)`	⬇️
spark-java-tests	`47.41% <37.83%> (-0.02%)`	⬇️
spark-scala-tests	`45.52% <36.48%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...va/org/apache/hudi/metadata/BaseTableMetadata.java	`80.74% <ø> (ø)`
...e/hudi/metadata/FileSystemBackedTableMetadata.java	`81.88% <100.00%> (ø)`
.../org/apache/hudi/metadata/HoodieTableMetadata.java	`68.42% <100.00%> (+1.75%)`	⬆️
...g/apache/hudi/hadoop/HiveHoodieTableFileIndex.java	`100.00% <ø> (ø)`
...main/scala/org/apache/hudi/DataSourceOptions.scala	`95.29% <100.00%> (+0.04%)`	⬆️
...c/main/java/org/apache/hudi/common/fs/FSUtils.java	`65.29% <66.66%> (-0.11%)`	⬇️
...ache/hudi/common/table/view/NoOpTableMetadata.java	`12.50% <0.00%> (-0.55%)`	⬇️
...rg/apache/hudi/hadoop/HoodieROTablePathFilter.java	`81.25% <50.00%> (-0.86%)`	⬇️
...c/main/scala/org/apache/hudi/HoodieFileIndex.scala	`82.35% <40.00%> (-1.73%)`	⬇️
...la/org/apache/hudi/SparkHoodieTableFileIndex.scala	`72.22% <33.33%> (-1.21%)`	⬇️
... and 2 more

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The metadata config `BaseHoodieTableFileIndex` builds is gated on a three-way conjunction, and nothing in the repo asserted how it resolves. Both of the added conjuncts came from behavior fixes: `isFilesPartitionAvailable` from HUDI-5403 (apache#7488, a Trino listing regression) and `useLatestBaseFilesPathFilterForListing` from apache#18136. The reflective test removed earlier in this PR planted the field and asserted the getter returned it, so it discriminated nothing. Parameterize the existing `TestLocalIndex` harness, which already builds a real index over a real meta client, with the RO-path-filter flag, and assert the full truth table through it. Each row was checked against a mutant: dropping any one of the three conjuncts from the production expression makes exactly one row fail.

github-actions Bot added the size:M PR with lines of changes in (100, 300] label Feb 8, 2026

suryaprasanna force-pushed the path-filter-during-listing branch 2 times, most recently from 95fda80 to 2dabf5d Compare February 9, 2026 16:16

suryaprasanna force-pushed the path-filter-during-listing branch from 2dabf5d to 9b8395e Compare February 9, 2026 18:23

github-actions Bot added size:L PR with lines of changes in (300, 1000] and removed size:M PR with lines of changes in (100, 300] labels Feb 9, 2026

apache deleted a comment from hudi-bot Feb 10, 2026

suryaprasanna commented Feb 10, 2026

View reviewed changes

nsivabalan reviewed Feb 11, 2026

View reviewed changes

yihua reviewed Feb 11, 2026

View reviewed changes

nsivabalan reviewed Feb 11, 2026

View reviewed changes

suryaprasanna commented Feb 11, 2026

View reviewed changes

Comment thread ...urce/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestROPathFilterOnRead.scala

nsivabalan reviewed Feb 12, 2026

View reviewed changes

suryaprasanna force-pushed the path-filter-during-listing branch from 9b8395e to de29445 Compare February 23, 2026 23:38

nsivabalan reviewed Feb 24, 2026

View reviewed changes

suryaprasanna added 7 commits February 25, 2026 06:56

Remove extra unit test to avoid higher unit test runtimes

2d7a6be

Update documentation

0dc9842

Address review comments

85ae0c4

Add time travel unit test

c501757

Address review comments

958491d

Fix failed unit test

8d3be59

suryaprasanna force-pushed the path-filter-during-listing branch from d34823b to 8d3be59 Compare February 25, 2026 06:56

nsivabalan approved these changes Feb 25, 2026

View reviewed changes

suryaprasanna changed the title ~~perf: Add HoodieROPathFilter support during file listing to prevent driver OOM~~ perf: Adding support for LatestBaseFilesPathFilter to Spark File Index Feb 26, 2026

nsivabalan merged commit 181af01 into apache:master Feb 26, 2026
87 of 88 checks passed

This was referenced Apr 29, 2026

NPE in BaseHoodieTableFileIndex.getInputFileSlices when RO path filter is enabled and a queried partition has no base files #18638

Closed

[MINOR] Fix NPE in getInputFileSlices when RO path filter returns empty partition #18639

Merged

voonhous mentioned this pull request Sep 7, 2026

refactor(spark): collapse HoodieTableState #19849

Merged

2 of 3 tasks

		@@ -146,6 +147,12 @@ public List<StoragePathInfo> getAllFilesInPartition(StoragePath partitionPath) t
		}

		@Override

Conversation

suryaprasanna commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

suryaprasanna commented Feb 9, 2026

Uh oh!

nsivabalan commented Feb 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

suryaprasanna commented Feb 8, 2026 •

edited

Loading