Skip to content

Checking maxRowsPerSegment after segment merge in dynamic partitioning #11252

Open
@jihoonson

Description

@jihoonson

Description

In this issue, the segment size refers to the row count in the segment. In dynamic partitioning, the task compares maxRowsPerSegment and the current Sink size to determine whether the segment hits the max segment size limit and needs to be pushed to deep storage. Here, the Sink size is the sum of in-memory segment size and the size of all persisted segments before they are merged. As a result, when the rollup is on, the Sink size can be larger than the actual segment size. This can create smaller segments than maxRowsPerSegment unnecessarily.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions