[SPARK-17850][Core]Add a flag to ignore corrupt files (branch 1.6) #15454
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zsxwing
changed the title
zsxwing
commented
Oct 12, 2016
| try { | ||
| finished = !reader.nextKeyValue | ||
| } catch { | ||
| case _: EOFException if ignoreCorruptFiles => finished = true |
There was a problem hiding this comment.
This is a behavior change to NewHadoopRDD, which may surprise the existing 1.6 users.
There was a problem hiding this comment.
Yeah, I am slightly worried about this change of behavior too.
Though I think it should be fine.
zsxwing
commented
Oct 12, 2016
| @@ -245,8 +248,7 @@ class HadoopRDD[K, V]( | |||
| try { | |||
| finished = !reader.next(key, value) | |||
| } catch { | |||
| case eof: EOFException => | |||
| finished = true | |||
| case _: EOFException if ignoreCorruptFiles => finished = true | |||
There was a problem hiding this comment.
I didn't use IOException to keep the default behavior is same as before.
zsxwing
commented
Oct 12, 2016
|
Test build #66853 has finished for PR 15454 at commit
|
mridulm
approved these changes
Oct 13, 2016
| @@ -245,8 +248,7 @@ class HadoopRDD[K, V]( | |||
| try { | |||
| finished = !reader.next(key, value) | |||
| } catch { | |||
| case eof: EOFException => | |||
| finished = true | |||
| case _: EOFException if ignoreCorruptFiles => finished = true | |||
| try { | ||
| finished = !reader.nextKeyValue | ||
| } catch { | ||
| case _: EOFException if ignoreCorruptFiles => finished = true |
There was a problem hiding this comment.
Yeah, I am slightly worried about this change of behavior too.
Though I think it should be fine.
asfgit
pushed a commit
that referenced
this pull request
Oct 13, 2016
## What changes were proposed in this pull request? This is the patch for 1.6. It only adds Spark conf `spark.files.ignoreCorruptFiles` because SQL just uses HadoopRDD directly in 1.6. `spark.files.ignoreCorruptFiles` is `true` by default. ## How was this patch tested? The added test. Author: Shixiong Zhu <shixiong@databricks.com> Closes #15454 from zsxwing/SPARK-17850-1.6.
zzcclp
pushed a commit
to zzcclp/spark
that referenced
this pull request
Oct 14, 2016
## What changes were proposed in this pull request? This is the patch for 1.6. It only adds Spark conf `spark.files.ignoreCorruptFiles` because SQL just uses HadoopRDD directly in 1.6. `spark.files.ignoreCorruptFiles` is `true` by default. ## How was this patch tested? The added test. Author: Shixiong Zhu <shixiong@databricks.com> Closes apache#15454 from zsxwing/SPARK-17850-1.6. (cherry picked from commit 585c565)
|
Since I have not yet heard complaints about this for 1.6, and this one may break some user's job, I'm going to close it now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is the patch for 1.6. It only adds Spark conf
spark.files.ignoreCorruptFilesbecause SQL just uses HadoopRDD directly in 1.6.spark.files.ignoreCorruptFilesistrueby default.How was this patch tested?
The added test.