TieringService support convert arrow to parquet directly

[leonardBang]

Search before asking

• I searched in the issues and found nothing similar.

Description

TieringService support convert arrow to parquet directly for higher performance

Willingness to contribute

• I'm willing to submit a PR!

[SteNicholas]

@leonardBang, does this ticket mean that we could introduce utilities to convert arrow to parquet format? IMO, we could reuse the conversion in Iceberg like ArrowReader, ParquetConversions.

[wuchong]

@SteNicholas, yes, we can certainly leverage some external libraries or implementations. However, the links you provided focus on converting Parquet to Arrow, whereas our requirement is the reverse: converting Arrow to Parquet.

[SteNicholas]

@wuchong, thanks for explanation for the ticket purpose. I would like to do a try to use jni wrapper way to invoke WriteRecordBatch of arrow.

[luoyuxia]

I created a issue in arrow-java to track this. Support parquet write from Arrow record batch apache/arrow-java#735

[xx789633]

Hi @luoyuxia , to implement the scenario you described in issue apache/arrow-java#735 (reading Arrow RecordBatches continuously from a remote server and writing to Parquet) using Arrow Java's DatasetFileWriter, we can subclass ArrowReader and implement a facade over an iterator. For example, arrow jdbc driver demonstrates such approach in its RootArrowReader. Specifically, we need to override loadNextBatch in ArrowReader to manually populate VectorSchemaRoot.

Regarding the performance comparison between Arrow Java's Parquet writing and the C++ implementation, do we have performance numbers?

[luoyuxia]

Hi @luoyuxia , to implement the scenario you described in issue apache/arrow-java#735 (reading Arrow RecordBatches continuously from a remote server and writing to Parquet) using Arrow Java's DatasetFileWriter, we can subclass ArrowReader and implement a facade over an iterator. For example, arrow jdbc driver demonstrates such approach in its RootArrowReader. Specifically, we need to override loadNextBatch in ArrowReader to manually populate VectorSchemaRoot.

Regarding the performance comparison between Arrow Java's Parquet writing and the C++ implementation, do we have performance numbers?

Just as discuss offline, that's hacky and will require us to implement a new arrow filesystem for the filesystems that not supported by arrow.