[SPARK-4131] [SQL] Support INSERT OVERWRITE [LOCAL] DIRECTORY '/path/to/dir' [ROW FORMAT row_format] [STORED AS file_format] query. #13067
Conversation
| r(0) = r(0) + a.name + "," | ||
| r(1) = r(1) + a.dataType.typeName + ":" | ||
| r | ||
| }) |
There was a problem hiding this comment.
Maybe it would be nicer if the additional closure is removed. This was discussed in (http://apache-spark-developers-list.1001551.n3.nabble.com/Question-about-Scala-style-explicit-typing-within-transformation-functions-and-anonymous-val-td17173.html)
val (cols, types) = child.output.foldLeft(("", "")) { case (r, a) =>
r(0) = r(0) + a.name + ","
r(1) = r(1) + a.dataType.typeName + ":"
r
}
|
( it seems the last part of the title os truncated.) |
|
@HyukjinKwon I don't understand your last comment
|
|
@Parth-Brahmbhatt I think the title of this PR, |
Parth-Brahmbhatt
changed the title
|
@HyukjinKwon fixed. |
|
Any update on this? Seems like a useful feature. |
|
Please let me cc @hvanhovell because it seems the most of codes were recently updated by him. |
|
@hvanhovell @rxin can one of you review this PR. I was hoping to get it in the 2.0 branch if its not too late. |
| @@ -713,6 +714,7 @@ WITH: 'WITH'; | |||
| VALUES: 'VALUES'; | |||
| CREATE: 'CREATE'; | |||
| TABLE: 'TABLE'; | |||
| DIRECTORY: 'DIRECTORY'; | |||
There was a problem hiding this comment.
Add this keyword to the nonReserved rule.
…to/dir' [ROW FORMAT row_format] [STORED AS file_format] query.
Parth-Brahmbhatt
force-pushed
the
SPARK-4131
branch
from
326c27f to
056eac6
Compare
|
Is this feature available yet? |
|
This sounds a syntactic sugar. Users can achieve it by creating an external table and then inserting the data. Another way is to use our DataFrameWriter API. Please let me know whether we should still continue this PR? |
|
Can one of the admins verify this patch? |
|
Since this PR #18975 will be merged soon. Could you close this? Thanks! |
What changes were proposed in this pull request?
Support INSERT OVERWRITE [LOCAL] DIRECTORY '/path/to/dir' [ROW FORMAT row_format] [STORED AS file_format] query
How was this patch tested?
I have tested the patch locally and I have some unit tests that I am finishing up.