Skip to content

Commit b0b9714

Browse files
committed
Review fix
1 parent 7f14b55 commit b0b9714

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

docs/structured-streaming-programming-guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -553,7 +553,7 @@ Here are the details of all the sources in Spark.
553553
NOTE: Both archiving (via moving) or deleting completed files will introduce overhead (slow down, even if it's happening in separate thread) in each micro-batch, so you need to understand the cost for each operation in your file system before enabling this option. On the other hand, enabling this option will reduce the cost to list source files which can be an expensive operation.<br/>
554554
Number of threads used in completed file cleaner can be configured with<code>spark.sql.streaming.fileSource.cleaner.numThreads</code>.<br/>
555555
NOTE 2: The source path should not be used from multiple sources or queries when enabling this option.<br/>
556-
NOTE 3: Both delete and move actions are best effort. Failing to delete or move files will not fail the streaming query.
556+
NOTE 3: Both delete and move actions are best effort. Failing to delete or move files will not fail the streaming query. Spark may not clean up some source files in some circumstances - e.g. the application doesn't shut down gracefully, too many files are queued to clean up.
557557
<br/><br/>
558558
For file-format-specific options, see the related methods in <code>DataStreamReader</code>
559559
(<a href="api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">Scala</a>/<a href="api/java/org/apache/spark/sql/streaming/DataStreamReader.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.streaming.DataStreamReader">Python</a>/<a

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,7 @@ object FileStreamSource {
350350
SQLConf.get.getConf(SQLConf.FILE_SOURCE_CLEANER_NUM_THREADS)
351351
)
352352

353-
def stop(): Unit = cleanThreadPool.shutdown()
353+
def stop(): Unit = ThreadUtils.shutdown(cleanThreadPool)
354354

355355
def clean(entry: FileEntry): Unit
356356
}

0 commit comments

Comments
 (0)