Skip to content

20190115 ruler better flags #1987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 21, 2020
Merged

20190115 ruler better flags #1987

merged 6 commits into from
Jan 21, 2020

Conversation

jtlisi
Copy link
Contributor

@jtlisi jtlisi commented Jan 15, 2020

What this PR does:

The current ruler config exposes a number of flags that should be set to default values. For instance the ruler should never be run with a replication factor >1. However, as things stand currently the flag defaults to 3 which makes configuring the sharded ruler quite difficult. This PR adds a Ruler RingConfig struct based on the Distributor RingConfig that simplifies the ring flags for the ruler.

  • Adds a guide on how to set up a sharded ruler
  • Add custom ruler ring config to remove unused/dangerous lifecylcer flags

Which issue(s) this PR fixes:
Fixes #1986

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated

@gouthamve
Copy link
Contributor

Could you open this against release-0.5 branch?

@jtlisi jtlisi changed the base branch from master to release-0.5 January 15, 2020 23:08
@jtlisi jtlisi changed the base branch from release-0.5 to master January 15, 2020 23:08
@jtlisi jtlisi changed the title 20190115 ruler sharding guide 20190115 ruler better flags Jan 15, 2020
@bboreham
Copy link
Contributor

I don't think it's reasonable to inject a breaking change to release 0.5.0 after we've had two release candidates out.
@gouthamve perhaps you meant just to change the docs?

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change makes much sense to clarify the ruler ring config to the user. @jtlisi could you rebase master, fix the conflicts and run make doc, please?

I don't think it's reasonable to inject a breaking change to release 0.5.0 after we've had two release candidates out.

@bboreham Given the decision to abandon the 0.5.0, I guess we're now OK with this breaking change. Am I wrong?


One option to scale the ruler is by scaling it horizontally. However, with multiple ruler instances running they will need to coordinate to determine which instance will evaluate which rule. Similar to the ingesters, the rulers establish a hash ring to divide up the responsibilities of evaluating rules.

## Config
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After rebasing and make doc you will also get the config file documentation updated. Could be worth adding a link to that doc (each root config block has an anchor, so you could link it to #ruler-config) mentioning it like To see the complete set of config option, please check out... 

HeartbeatTimeout time.Duration `yaml:"heartbeat_timeout,omitempty"`

// Instance details
InstanceID string `yaml:"instance_id"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like we did in the DistributorConfig, I would suggest to add a doc:"hidden" to all Instance* fields.

@jtlisi jtlisi force-pushed the 20190115_ruler_sharding_guide branch from 8c94584 to 1d02d71 Compare January 17, 2020 17:20
@jtlisi
Copy link
Contributor Author

jtlisi commented Jan 17, 2020

@pracucci Thanks for the feedback. I addressed your comments and this should be good for a second look.

Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with a nit.

@@ -61,14 +61,14 @@ type Config struct {

EnableSharding bool // Enable sharding rule groups
SearchPendingFor time.Duration
LifecyclerConfig ring.LifecyclerConfig
RulerRing RingConfig
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resulting yaml field name is rulerring, which looks a bit repetitive considering we're already inside the ruler: {} config. What's your take adding a yaml:"ring"? In case you decide to keep the current name, then ruler_ring may be better (easier to read for the user).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>
@jtlisi jtlisi force-pushed the 20190115_ruler_sharding_guide branch from a7a7a8f to 93fbe0e Compare January 21, 2020 15:58
Copy link
Contributor

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked after the latest commit and it still LGTM 👍

@jtlisi jtlisi merged commit 6774c62 into master Jan 21, 2020
@jtlisi jtlisi deleted the 20190115_ruler_sharding_guide branch January 21, 2020 17:04
@pracucci pracucci mentioned this pull request Jan 22, 2020
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unfriendly Ruler ring flags
4 participants