Skip to content

[SPARK-25404][SQL] Staging path may not on the expected place when table path contains the stagingDir string #22412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-25404][SQL] Staging path may not on the expected place when table path contains the stagingDir string #22412

wants to merge 1 commit into from

Conversation

fjh100456
Copy link
Contributor

What changes were proposed in this pull request?

As described in #SPARK-25404, staging path may not on the right place we expect. I'm not quiet sure in which case the inputPathName contains the stagingDir, but it seems new Path(inputPathName, stagingDir).toString is enough.

    var stagingPathName: String =
      if (inputPathName.indexOf(stagingDir) == -1) {
        new Path(inputPathName, stagingDir).toString
      } else {
        inputPathName.substring(0, inputPathName.indexOf(stagingDir) + stagingDir.length)
      }

How was this patch tested?

Manually test with debug mode, and check the staging files on right path.

@fjh100456
[SPARK-25404][SQL] Staging path may not on the expected place when ta…
a6d4855 
…ble path contains the stagingDir string

## What changes were proposed in this pull request?
Fix Staging path.

## How was this patch tested?
Manual test.
xuanyuanking
xuanyuanking reviewed Sep 13, 2018
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala
@@ -217,12 +217,7 @@ private[hive] trait SaveAsHiveFile extends DataWritingCommand {
val inputPathUri: URI = inputPath.toUri
val inputPathName: String = inputPathUri.getPath
val fs: FileSystem = inputPath.getFileSystem(hadoopConf)
var stagingPathName: String =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code is introduced form #12770, do you check the relative UT? Maybe UT relating with SessionState and DDLSuite.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I had checked the pr #12770 too. But I'm not very clear why we should do like that.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@github-actions
Copy link

github-actions bot commented Jan 7, 2020

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

@github-actions github-actions bot added the Stale label Jan 7, 2020
@github-actions github-actions bot closed this Jan 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuanyuanking xuanyuanking xuanyuanking left review comments

Assignees
No one assigned
Labels
SQL Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@fjh100456 @AmplabJenkins @xuanyuanking @dongjoon-hyun