[ILM]: Fix TSDS unfollow timing with WaitUntilTimeSeriesEndTimePassesStep #128361
Conversation
|
💚 CLA has been signed |
elasticsearchmachine
added
needs:triage
|
Hi, @gmarouli |
|
Hi @happysubin , thank you for your contribution, @samxbr and I will review it as soon as possible. |
gmarouli
added
:Data Management/ILM+SLM
elasticsearchmachine
added
Team:Data Management
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
|
Pinging @elastic/es-data-management (Team:Data Management) |
There was a problem hiding this comment.
@happysubin Thank you for contributing to Elasticsearch! We value external contributions and would love to work with you to get this PR merged. I have left some comments on the PR, please feel free to take your time to address them.
| WaitForFollowShardTasksStep step2 = new WaitForFollowShardTasksStep( | ||
| waitForFollowShardTasks, | ||
| waitUntilTimeSeriesEndTimePassesStep, | ||
| client | ||
| ); | ||
| WaitUntilTimeSeriesEndTimePassesStep step3 = new WaitUntilTimeSeriesEndTimePassesStep( | ||
| waitUntilTimeSeriesEndTimePassesStep, | ||
| pauseFollowerIndex, | ||
| Instant::now | ||
| ); |
There was a problem hiding this comment.
The WaitUntilTimeSeriesEndTimePassesStep should be prior to WaitForFollowShardTasksStep, because the follower index can sync with the leader index one last time after the end_time has passed, to make sure there's no new docs coming in to the leader index.
There was a problem hiding this comment.
Please add new integration test in the CCRIndexLifecycleIT test suite to cover this scenario. You can refer to this test as an example for verifying the WaitUntilTimeSeriesEndTimePassesStep. Essentially we would want to verify that after leader index rollovers, the follower index goes into the WaitUntilTimeSeriesEndTimePassesStep, and new documents to the leader index are synced to the follower index until end_time has passed.
There was a problem hiding this comment.
Thank you for the code review.
I'll review the existing test cases and work on adding new ones.I'm going to add test code!
|
buildkite test this |
fix: #128129
I added the WaitUntilTimeSeriesEndTimePassesStep between the WaitForFollowShardTasksStep and the PauseFollowerIndexStep in the step list of UnFollowAction, and updated the tests accordingly.
Please review my code!