Skip to content

Improve segment build auto retry logics#16663

Merged
chenboat merged 1 commit intoapache:masterfrom
lnbest0707:upstream_fork_retry
Aug 25, 2025
Merged

Improve segment build auto retry logics#16663
chenboat merged 1 commit intoapache:masterfrom
lnbest0707:upstream_fork_retry

Conversation

@lnbest0707
Copy link
Contributor

@lnbest0707 lnbest0707 commented Aug 21, 2025

ingestion
Followup on #15234
There could be some other similar "overflow" issues with different types of exceptions, add them to auto retry
(Sample error traceback)

java.nio.BufferOverflowException: null at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:1210) at
org.apache.pinot.segment.local.io.util.VarLengthValueWriter.add(VarLengthValueWriter.java:119) at
org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator.seal(OffHeapJsonIndexCreator.java:149) at
org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:470) at
org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:372) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.buildByColumn(SegmentIndexCreationDriverImpl.java:344) at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:115) at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentInternal(RealtimeSegmentDataManager.java:1086) at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.buildSegmentForCommit(RealtimeSegmentDataManager.java:947) at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:850) at java.base/java.lang.Thread.run(Thread.java:1583)

@lnbest0707 lnbest0707 force-pushed the upstream_fork_retry branch from 45f78f6 to ed5f794 Compare August 21, 2025 22:34
@codecov-commenter
Copy link

codecov-commenter commented Aug 21, 2025

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 63.34%. Comparing base (76f2b7f) to head (ed5f794).

Files with missing lines Patch % Lines
...a/manager/realtime/RealtimeSegmentDataManager.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #16663      +/-   ##
============================================
+ Coverage     63.32%   63.34%   +0.02%     
+ Complexity     1380     1379       -1     
============================================
  Files          3035     3035              
  Lines        176714   176714              
  Branches      27121    27121              
============================================
+ Hits         111896   111938      +42     
+ Misses        56232    56190      -42     
  Partials       8586     8586              
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 63.27% <0.00%> (+<0.01%) ⬆️
java-21 63.30% <0.00%> (+<0.01%) ⬆️
temurin 63.34% <0.00%> (+0.02%) ⬆️
unittests 63.34% <0.00%> (+0.02%) ⬆️
unittests1 56.43% <0.00%> (-0.03%) ⬇️
unittests2 33.16% <0.00%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chenboat chenboat merged commit 85fbe16 into apache:master Aug 25, 2025
16 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants