Open
Description
Description
In this issue, the segment size refers to the row count in the segment. In dynamic partitioning, the task compares maxRowsPerSegment
and the current Sink
size to determine whether the segment hits the max segment size limit and needs to be pushed to deep storage. Here, the Sink
size is the sum of in-memory segment size and the size of all persisted segments before they are merged. As a result, when the rollup is on, the Sink
size can be larger than the actual segment size. This can create smaller segments than maxRowsPerSegment
unnecessarily.