Skip to content

[SPARK-5723][SQL]Change the default file format to Parquet for CTAS statements. #4639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

[SPARK-5723][SQL]Change the default file format to Parquet for CTAS statements. #4639

wants to merge 4 commits into from

Conversation

yhuai
Copy link
Contributor

@yhuai yhuai commented Feb 17, 2015

@SparkQA
Copy link

SparkQA commented Feb 17, 2015

Test build #27601 has started for PR 4639 at commit 5a67903.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 17, 2015

Test build #27601 has finished for PR 4639 at commit 5a67903.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class ShowTablesCommand(databaseName: Option[String]) extends RunnableCommand

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27601/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 17, 2015

Test build #27646 has started for PR 4639 at commit 8af5b2a.

  • This patch merges cleanly.

marmbrus
marmbrus reviewed Feb 17, 2015
View reviewed changes
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
}.getMessage
assert(
message.contains("Cannot specify database name in a CTAS statement"),
"When spark.sql.sources.convertHiveCTASWithoutStorageSpec is true, we should not allow " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this message is out of date. Also, "should.." -> "do not currently allow the database name to be specified"

@SparkQA
Copy link

SparkQA commented Feb 17, 2015

Test build #27646 has finished for PR 4639 at commit 8af5b2a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27646/
Test PASSed.

marmbrus
marmbrus reviewed Feb 17, 2015
View reviewed changes
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
* and no SerDe is specified (no ROW FORMAT SERDE clause).
*/
protected[sql] def convertCTAS: Boolean =
getConf("spark.sql.hive.convertCTAS", "false") == "true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: .toBoolean

@SparkQA
Copy link

SparkQA commented Feb 18, 2015

Test build #27666 has started for PR 4639 at commit a568137.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 18, 2015

Test build #27666 has finished for PR 4639 at commit a568137.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
    • class SparkStageInfo(namedtuple("SparkStageInfo",
    • class StatusTracker(object):

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27666/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 18, 2015
@yhuai @marmbrus
[SPARK-5723][SQL]Change the default file format to Parquet for CTAS s…
6e82c46 
…tatements.

JIRA: https://issues.apache.org/jira/browse/SPARK-5723

Author: Yin Huai <yhuai@databricks.com>

This patch had conflicts when merged, resolved by
Committer: Michael Armbrust <michael@databricks.com>

Closes #4639 from yhuai/defaultCTASFileFormat and squashes the following commits:

a568137 [Yin Huai] Merge remote-tracking branch 'upstream/master' into defaultCTASFileFormat
ad2b07d [Yin Huai] Update tests and error messages.
8af5b2a [Yin Huai] Update conf key and unit test.
5a67903 [Yin Huai] Use data source write path for Hive's CTAS statements when no storage format/handler is specified.

(cherry picked from commit e50934f)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in e50934f Feb 18, 2015
@viirya viirya mentioned this pull request Mar 13, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@yhuai @SparkQA @AmplabJenkins @marmbrus