Fix deletion of files in current working directory by clearFiles() #345
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes an issue where Spark could delete original files in the current working directory that were added to the job using addFile(). There was also the potential for addFile() to overwrite local files, which is addressed by changing Utils.fetchFile() to log a warning instead of overwriting a file with new contents. This is a short-term fix; a better long-term solution would be to remove the dependence on storing files in the current working directory, since we can't change the cwd from Java.
| Utils.copyStream(in, out, true) | ||
| if (targetFile.exists && !Files.equal(tempFile, targetFile)) { | ||
| logWarning("File " + targetFile + " exists and does not match contents of " + url + | ||
| "; using existing version") |
There was a problem hiding this comment.
Turn this into an error and throw an exception here; it's too risky to continue running IMO.
There was a problem hiding this comment.
I agree; I pushed a commit to change this to SparkException.
mateiz
added a commit
that referenced
this pull request
Dec 29, 2012
Fix deletion of files in current working directory by clearFiles()
|
Thanks Josh! |
ash211
pushed a commit
to ash211/mesos-spark
that referenced
this pull request
Jan 10, 2014
support distributing extra files to worker for yarn client mode So that user doesn't need to package all dependency into one assemble jar as spark app jar
harveyfeng
pushed a commit
to harveyfeng/spark-mesos
that referenced
this pull request
Apr 11, 2014
This is a patch to address @mateiz 's comment in apache/spark#245 MLUtils#loadLibSVMData uses an anonymous function for the label parser. Java users won't like it. So I make a trait for LabelParser and provide two implementations: binary and multiclass. Author: Xiangrui Meng <meng@databricks.com> Closes mesos#345 from mengxr/label-parser and squashes the following commits: ac44409 [Xiangrui Meng] use singleton objects for label parsers 3b1a7c6 [Xiangrui Meng] add tests for label parsers c2e571c [Xiangrui Meng] rename LabelParser.apply to LabelParser.parse use extends for singleton 11c94e0 [Xiangrui Meng] add return types 7f8eb36 [Xiangrui Meng] change labelParser from annoymous function to trait
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This addresses an issue where Spark could delete files in the current working directory that were added to the job using
addFile(). I encountered this issue when working on PySpark's code deployment mechanism, which is based onaddFile().From user-code's perspective (e.g. UDFs), files added through
addFile()are assumed to be in the current working directory. For jobs that are run locally usingDAGScheduler.runLocally(), tasks run with the driver's current working directory. As a result, files added throughaddFile()must be copied to the driver's current working directory. There's no mechanism to change the CWD in Java.clearFiles()andclearJars()clean up these files when the driver exits. This can be a problem if the original files that were added were in the driver's current working directory, because this will cause them to be deleted.A long-term fix would be to hide the location of fetched files from user code by requiring it to access files through an API like
SparkFiles.get("my-file-name.txt"). This will require changes to user code and may require changes to Shark.As a short-term fix, this pull request removes the code that deletes files in the current working directory and adds checks to
Utils.fetchFiles()to avoid overwriting existing local files with new data. The one downside of this change is that it may add junk to the current working directory. This is preferable to accidentally deleting files.I've also added
addFile()/addJar()to the Java API.I also added synchronization to
LocalScheduler.updateDependenciesto avoid performing multiple parallel fetches for the same file.