Skip to content

HTTP handler for ingester shutdown #1746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 11, 2019

Conversation

codesome
Copy link
Contributor

@codesome codesome commented Oct 21, 2019

This is based on this design for scaling down using the WAL. This would not block #1103 but this is required for scale downs.

I have yet to verify how it will interact with Kubernetes health checks and pod restarts if the endpoint is hit.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
@codesome codesome marked this pull request as ready for review October 22, 2019 09:33
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
@codesome
Copy link
Contributor Author

Tested it and it works as intended. Ingester goes to LEAVING states, flushes after waiting for other ingesters, and then leaves the ring.

I also added another commit to avoid panic on multiple shutdown calls (during graceful shutdown after having called /shutdown already)

@gouthamve gouthamve requested a review from jtlisi November 4, 2019 11:42
@bboreham
Copy link
Contributor

bboreham commented Nov 4, 2019

Could you add a description of this API to docs/apis.md ?

Also it seems worth mentioning in CHANGELOG.md.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Copy link
Contributor

@jtlisi jtlisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codesome Will this shutdown cause a clean exit from cortex or only shutdown the ingester module?

@bboreham
Copy link
Contributor

bboreham commented Nov 8, 2019

Needs rebase for changelog conflict now.

Copy link
Contributor

@gouthamve gouthamve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after fixing the API nit. And after rebase.

Ganesh Vernekar added 2 commits November 11, 2019 13:09
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
@gouthamve gouthamve merged commit 2ff3fa2 into cortexproject:master Nov 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants