Skip to content

Commit d075b33

Browse files
Ngone51cloud-fan
authored andcommitted
[SPARK-28366][CORE][FOLLOW-UP] Improve the conf IO_WARNING_LARGEFILETHRESHOLD
### What changes were proposed in this pull request? Improve conf `IO_WARNING_LARGEFILETHRESHOLD` (a.k.a `spark.io.warning.largeFileThreshold`): * reword documentation * change type from `long` to `bytes` ### Why are the changes needed? Improvements according to #25134 (comment) & #25134 (comment). ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass Jenkins. Closes #26691 from Ngone51/SPARK-28366-followup. Authored-by: wuyi <ngone_5451@163.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 7c0ce28 commit d075b33

File tree

1 file changed

+3
-3
lines changed
  • core/src/main/scala/org/apache/spark/internal/config

1 file changed

+3
-3
lines changed

core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1325,9 +1325,9 @@ package object config {
13251325
private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
13261326
ConfigBuilder("spark.io.warning.largeFileThreshold")
13271327
.internal()
1328-
.doc("When spark loading one single large file, if file size exceed this " +
1329-
"threshold, then log warning with possible reasons.")
1330-
.longConf
1328+
.doc("If the size in bytes of a file loaded by Spark exceeds this threshold, " +
1329+
"a warning is logged with the possible reasons.")
1330+
.bytesConf(ByteUnit.BYTE)
13311331
.createWithDefault(1024 * 1024 * 1024)
13321332

13331333
private[spark] val EVENT_LOG_COMPRESSION_CODEC =

0 commit comments

Comments
 (0)