Skip to content

Commit 9ba89ca

Browse files
author
Christophe Préaud
committed
Ensure that files are fetched atomically
tempFile is created in the same directory than targetFile, so that the move from tempFile to targetFile is always atomic
1 parent 54419ae commit 9ba89ca

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

core/src/main/scala/org/apache/spark/util/Utils.scala

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -425,8 +425,7 @@ private[spark] object Utils extends Logging {
425425
conf: SparkConf,
426426
securityMgr: SecurityManager,
427427
hadoopConf: Configuration) {
428-
val tempDir = getLocalDir(conf)
429-
val tempFile = File.createTempFile("fetchFileTemp", null, new File(tempDir))
428+
val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath))
430429
val targetFile = new File(targetDir, filename)
431430
val uri = new URI(url)
432431
val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false)

0 commit comments

Comments
 (0)