append row groups to already exist parquet file #557
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The SerializedFileWriter can only write row groups to new file , I want to append data to already existing parquet file
Describe the solution you'd like
Since we dont want to break SerializedFileWriter api， we can create a new struct SerializedFileAppender, when new this struct, just fetch and truncate the existing parquet file meta and footer, the rest is just the same as SerializedFileWriter.
Describe alternatives you've considered
Maybe change the SerializedFileWriter to supprot append mode.
Additional context
the code maybe like this
impl<F: ParquetWriter + ChunkReader> SerializedFileAppender<F> {
pub fn new(
mut read_writer: F,
schema: TypePtr,
properties: WriterPropertiesPtr,
) -> Result<Self> {
let mut total_num_rows = 0;
let mut row_groups = vec![];
if read_writer.len() != 0 {
let (metadata, meta_footer_size) = footer::parse_metadata(&read_writer)?;
total_num_rows = metadata.file_metadata().num_rows();
read_writer.seek(SeekFrom::End(-(meta_footer_size as i64)))?;
row_groups = metadata
.row_groups()
.iter()
.cloned()
.map(|r| Arc::new(r))
.collect::<Vec<RowGroupMetaDataPtr>>();
}
Ok(Self {
read_writer,
schema: schema.clone(),
descr: Arc::new(SchemaDescriptor::new(schema)),
props: properties,
total_num_rows,
row_groups,
previous_writer_closed: true,
is_closed: false,
})
}
}