Allowing empty segments with no offset advancing

[jadami10]

This is a followup from #8929, but in the case of 0 data being consumed. We've since found a poor interaction between Pinot and our s3 lifecycling.

• we have a few partitions that have 0 data all the time (not filtered out, literally 0 events)
• Pinot keeps that consuming segment open indefinitely (in this case we noticed it had been 1 year)
• Pinot also keeps the last completed segment for each partition (in this case, it's from 2023)
• We wanted to use s3 lifecycling to delete all data < 30 days old (our tables had 10 day retention) rather than just rely on Pinot's mechanisms, but this sent segments into error state since they had been around for 1 year

Is there any reason we can't seal a segment where the offset hasn't advanced? In this case, we would have had N segments for this partition all with 0 records and the same start/end offset.

cc @Jackie-Jiang @priyen-stripe

[Jackie-Jiang]

I think we don't seal them right now because we don't support empty segment before. Since we can support empty segment now, we should be able to seal them. We want to revisit the timestamp used for empty segment (using current time should work) so that retention manager can remove them properly.

[tibrewalpratik17]

Hey @Jackie-Jiang any updates on this?

Is this as straight-forward as removing these lines?

pinot/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java

Lines 334 to 338 in 0f92742

	if (!_hasMessagesFetched) {
	_segmentLogger.info("No events came in, extending time by {} hours", TIME_EXTENSION_ON_EMPTY_SEGMENT_HOURS);
	_consumeEndTime += TimeUnit.HOURS.toMillis(TIME_EXTENSION_ON_EMPTY_SEGMENT_HOURS);
	return false;
	}

Saw this in one of our tables today. The table existed for more than a year and we enabled snapshot on it recently. Now since the last consuming segment (which was like months old) never received any message, the snapshot flow didn't run and that entire partition was knocked out of upsert-compaction flow. [Update] I have tried to fix this issue for partial-upsert tables here - #13285 (comment)

[Jackie-Jiang]

Is this as straight-forward as removing these lines?

I think so.

The drawback is that we might end up with a lot of segments that are all empty, but I guess we can make retention manager keep cleaning up empty segments (it might already do) as long as they are not the last completed segment.

[jadami10]

One other issue we've seen (QA only so far) from our multi-kafka cluster setup

• table is created and creates as segment for partition A on cluster 1
• partition A has no data
• cluster 1 is later removed for this topic, so Pinot should no longer get this partition
• Pinot keeps trying to consume because the segment is open indefinitely and spews endless errors