Introduce GenericAppenderFactory and GenericAppenderHelper #1340
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JingsongLi
force-pushed
the
appender
branch
3 times, most recently
from
f80e419 to
38e0605
Compare
openinx
reviewed
Aug 18, 2020
| return Avro.write(outputFile) | ||
| .schema(schema) | ||
| .createWriterFunc(DataWriter::create) | ||
| .named(fileFormat.name()) |
There was a problem hiding this comment.
The name is usually a table name for the given schema. not fileFormat. we may need to remove this line, or set it with a reasonable table name.
| return Parquet.write(outputFile) | ||
| .schema(schema) | ||
| .createWriterFunc(GenericParquetWriter::buildWriter) | ||
| .named(fileFormat.name()) |
| return ORC.write(outputFile) | ||
| .schema(schema) | ||
| .createWriterFunc(GenericOrcWriter::buildWriter) | ||
| .setAll(config) |
There was a problem hiding this comment.
To keep the consistent semantics, I suggest to set overwrite=true. Both flink and spark's FileAppenderFactory have enabled the overwrite.
openinx
reviewed
Aug 18, 2020
| .withPath(file.toURI().toString()) | ||
| .withMetrics(appender.metrics()) | ||
| .withFormat(format); | ||
| if (partition != null) { |
There was a problem hiding this comment.
We could call the withPartition(partition) directly without this nullable check, because builder will handle this inside it.
|
Thanks @JingsongLi, looks good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #1293, we found lot of places use
switch case avro orc parquetto create appender, these logicals can be extract toGenericAppenderFactory.