Pauseless Consumption #3: Disaster Recovery with Reingestion#14920

Merged

Jackie-Jiang merged 70 commits intoapache:masterfrom

KKcorps:pauseless-reingestion

Feb 12, 2025

Commits on Jan 3, 2025

WIP: Changes in the expected state while performing recovery during pauseless ingestion in RealtimeSegmentValidationManager
9aman
committed
Adding changes for ensurePartitionsConsuming function that is utilized by RealtimeSegmentValitdationManager to fix commit protocol failures
9aman
committed
Add Server side Reingestion API

Kartik Khare
authored and
9aman
committed
run segment level validation

Kartik Khare
authored and
9aman
committed
Add method to trigger reingestion

Kartik Khare
authored and
9aman
committed
Linting fixes

Kartik Khare
authored and
9aman
committed
Adding integration tests for 3 failure scenarios that can occur during commit protocol
9aman
committed

Commits on Jan 6, 2025

WIP: Reingestion test

Kartik Khare
authored and
Kartik Khare
committed
Fix bug with indexes in reingestion

Kartik Khare
authored and
Kartik Khare
committed

Commits on Jan 8, 2025

Add tests for reingestion

Kartik Khare
authored and
Kartik Khare
committed

Commits on Jan 14, 2025

Fix controller url in Reingestion
KKCorps
committed
Support https and auth in reingestion
KKCorps
committed

Commits on Jan 16, 2025

Commits on Jan 17, 2025

Fix reingestion test
KKCorps
committed
refactoring file upload download client
KKCorps
committed

Commits on Jan 21, 2025

Commits on Jan 23, 2025

Removing check in replaceSegmentIfCrcMismatch
9aman
committed

Commits on Jan 24, 2025

Commits on Jan 27, 2025

Revert "Remove reingestion code"
KKCorps
committed
Revert "Removing files related to reingestion tests"
KKCorps
committed
Fix reingestion issue where consuming segemnts are not replaced
KKCorps
committed
Copy full segment to deep store before triggerring metadata upload
KKCorps
committed
Refactoring: added support for tracking running reingestion jobs
KKCorps
committed
Refactor: fix doc comments
KKCorps
committed

Commits on Jan 28, 2025

Make SegmentZKMetadata JSON serializable

Jackie-Jiang
authored and
KKCorps
committed
Minor API name change

Jackie-Jiang
authored and
KKCorps
committed
Refactor PinotLLC class to add ability to inject failures
KKCorps
committed

Commits on Jan 29, 2025

Minor improvements

9aman
authored and
KKCorps
committed
Moving ZkMetadaptaUtils to commons and reusing the code in the upload segment API's

9aman
authored and
KKCorps
committed
Fix lint failures
KKCorps
committed
Misc fix and cleanup

Jackie-Jiang
authored and
KKCorps
committed
The tests were running slow due to the condition that the IdealState can only be updated after a fixed time has elapsed

9aman
authored and
KKCorps
committed
Refactor Reingestion integration test
KKCorps
committed
Merge remote-tracking branch 'upstream/master' into pauseless-reingestion
KKCorps
committed
Fix error in tests post rebase
KKCorps
committed

Commits on Jan 30, 2025

refactoring
KKCorps
committed

Commits on Feb 3, 2025

Remove redundant code
KKCorps
committed
Add support for queue in reingestion
KKCorps
committed

Commits on Feb 4, 2025

Refactoring
KKCorps
committed
refaactoring
KKCorps
committed
Honour segment build semaphore during reingestion
KKCorps
committed

Commits on Feb 5, 2025

Fix test
KKCorps
committed
Add a seperate API to upload reingested segments
KKCorps
committed
Cleanup code
KKCorps
committed

Commits on Feb 6, 2025

Replace reIngest with reingest
KKCorps
committed
Replace reIngest with reingest
KKCorps
committed
Ensure correctness in reingestion
KKCorps
committed
Decouple reingest completion and upload ZK path
KKCorps
committed
Refactor Stateless segment writer to have least number of arguments
KKCorps
committed
Fix segment reset for pauseless tables
KKCorps
committed
Cleaning up Stateless segment writer
KKCorps
committed
Rename metric
KKCorps
committed

Commits on Feb 12, 2025