Skip to content

Commit bf43bb2

Browse files
authored
Add partial serde support for ParquetWriterOptions (#8627)
* Add serde support for ParquetWriterOptions * save progress * test passes * Improve test * Refactor and add link to follow on issue * remove duplicate code * clippy * Regen * remove comments from proto file * change proto types from i32 to u32 pre feedback on PR * change to u64
1 parent 8524d58 commit bf43bb2

File tree

6 files changed

+524
-34
lines changed

6 files changed

+524
-34
lines changed

datafusion/proto/proto/datafusion.proto

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1206,13 +1206,28 @@ message PartitionColumn {
12061206
message FileTypeWriterOptions {
12071207
oneof FileType {
12081208
JsonWriterOptions json_options = 1;
1209+
ParquetWriterOptions parquet_options = 2;
12091210
}
12101211
}
12111212

12121213
message JsonWriterOptions {
12131214
CompressionTypeVariant compression = 1;
12141215
}
12151216

1217+
message ParquetWriterOptions {
1218+
WriterProperties writer_properties = 1;
1219+
}
1220+
1221+
message WriterProperties {
1222+
uint64 data_page_size_limit = 1;
1223+
uint64 dictionary_page_size_limit = 2;
1224+
uint64 data_page_row_count_limit = 3;
1225+
uint64 write_batch_size = 4;
1226+
uint64 max_row_group_size = 5;
1227+
string writer_version = 6;
1228+
string created_by = 7;
1229+
}
1230+
12161231
message FileSinkConfig {
12171232
reserved 6; // writer_mode
12181233

datafusion/proto/src/generated/pbjson.rs

Lines changed: 321 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)