-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segment 'Online' but not properly sealed (no segment file created) #13149
Comments
Thanks, @cbalci for drafting this message. We have faced this issue 2 times since last month Setup
Series of Events in Increasing Order of Time
This is how the segment ended in such undesirable state in S2 . we ended up losing the segment entirely as S1 container was replaced before it even uploaded to Deepstore and S2 container was restarted (_segmentDataManagerMap in memory map getting cleared ) . We are aware that Deepstore upload retry could have solved data loss . Some improvements found during this debugging
Looking forward to see if any-others have faced a similar kind of issue and open to hear about improvements and additional guardrails we can setup to prevent such occurrence |
We observed this interesting behavior from a Realtime table. For a single segment, something apparently went wrong at commit time that the server wasn't able to commit the segment to disk and remove files under 'consumers'. However the ingestion continued without an issue and the server has been consuming and sealing segments since then just fine.
At this point this segment:
consumers/
, which are still memory mappedRealtimeSegmentValidationManager
(controller) which attempted to upload segment files to deep store but failed. (Unfortunately segment build time logs from the server are lost due to retention)My two questions are:
cc @chenboat @eaugene
The text was updated successfully, but these errors were encountered: