Commit f1ac35b
committed
[SPARK-51049][CORE] Increase S3A Vector IO threshold for range merge
### What changes were proposed in this pull request?
This PR aims to increase S3A Vector IO threshold for range merge.
### Why are the changes needed?
Apache Spark 4.0.0 supported Hadoop Vectored IO via ORC and Parquet.
As a part of [HADOOP-18855 VectorIO API tuning/stabilization](https://issues.apache.org/jira/browse/HADOOP-18855), Apache Hadoop 3.4.2 will have new threshold default values. We had better follow these update in advance until Apache Hadoop 3.4.2 is released.
- apache/hadoop#7281
### Does this PR introduce _any_ user-facing change?
No, Hadoop Vectored IO features are new in Apache Spark 4.0.0 .
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49748 from dongjoon-hyun/SPARK-51049.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit b62c3f4)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>1 parent 0df77e6 commit f1ac35b
1 file changed
+4
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
423 | 423 | | |
424 | 424 | | |
425 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
426 | 430 | | |
427 | 431 | | |
428 | 432 | | |
| |||
0 commit comments