Add supports to Insert Into #9
Conversation
|
cc: @dongjoon-hyun May I add unit tests in another PR? |
|
I'll add the unit tests in this PR soon |
weiqingy
force-pushed
the
InsertInto
branch
2 times, most recently
from
83c6cbd to
e96d7ff
Compare
|
Hi @dongjoon-hyun , I have added the unit tests. cc: @jerryshao |
| super.beforeAll() | ||
| sparkSession = SparkSession.builder() | ||
| .master("local") | ||
| .config("spark.sql.catalogImplementation", "hive") |
There was a problem hiding this comment.
.config("spark.sql.catalogImplementation", "hive") ==> enableHiveSupport()?
| override def beforeAll(): Unit = { | ||
| super.beforeAll() | ||
| sparkSession = SparkSession.builder() | ||
| .master("local") |
There was a problem hiding this comment.
Both local and local[1] will run Spark locally with one worker thread
| sparkSession.sql(s"CREATE TABLE $sourceHiveTblName (name string)") | ||
| sparkSession.sql(s"INSERT INTO TABLE $sourceHiveTblName VALUES ('a'), ('b'), ('c')") | ||
|
|
||
| sparkSession.sql(s"CREATE TABLE $sourceSparkTblName (name string) USING PARQUET") |
There was a problem hiding this comment.
Can we use USING ORC instead of USING ORC? :)
|
Thank you for merging and adding UTs. |
|
Next time when you update some code in this PR, Travis CI will be triggered. |
|
Thanks, @dongjoon-hyun . I'll update this PR soon. |
| case c: CreateDataSourceTableAsSelectCommand => | ||
| logDebug(s"CREATE TABLE USING xx AS SELECT query: ${qd.qe}") | ||
| CommandsHarvester.CreateDataSourceTableAsSelectHarvester.harvest(c, qd) | ||
|
|
||
| case c: CreateViewCommand => |
There was a problem hiding this comment.
Do we have a test case for this? Otherwise, can we add this in another PR?
There was a problem hiding this comment.
Ohh, not yet. I'll add unit tests for CreateViewCommand
|
@weiqingy . Please update PR description because you added the test case and Travis CI verified that. Overall, LGTM. I think we can proceed to another PRs. cc @jerryshao |
|
Thanks @dongjoon-hyun . The unit test for CreateViewHarvester was added. |
Changes:
Merge code supporting "Insert Into" and "Create View xx As Select * From xx" into master branch.
Testing:
New unit tests were added.
I manually tested this PR in HDP cluster:
Case1:
sql("create table table_parquet_200 (a int) using parquet")
sql("insert into table table_parquet_200 values (200)")
sql("create table table_parquet_300 (a int) using parquet")
sql("INSERT INTO table_parquet_300 SELECT * FROM table_parquet_200")
Case2:
sql("CREATE TABLE t00000(a int)")
sql("INSERT INTO t00000 VALUES (1)")
sql("CREATE TABLE t10000(a int)")
sql("INSERT INTO t10000 SELECT * FROM t00000")
Case 3:
sql("CREATE TABLE sourceTable(a int)")
sql("INSERT INTO sourceTable VALUES (1)")
sql("CREATE view s_view1 as select * from sourceTable")