Enabling availability zone awareness in metric R/W with ingesters. #2317
Conversation
…ded distinct zone check in ring Get ingesters function Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
Signed-off-by: Ken Haines <khaines@microsoft.com>
khaines
force-pushed
the
khaines/zone-aware-replication
branch
from
7c7ae7f to
a978fd4
Compare
Signed-off-by: Ken Haines <khaines@microsoft.com>
|
The lint failure is because of the doc check. The default for the zone setting at the moment is the hostname, which makes the document verification expectations dynamic. I'm working on a solution to this. |
…generated in a consistent/verifiable way but still have a decent default value Signed-off-by: Ken Haines <khaines@microsoft.com>
|
A great suggestion @tomwilkie. I'll work on that right away and add it here. |
Signed-off-by: Ken Haines <khaines@microsoft.com>
|
Looks like this broke my staging system: |
| title: "Ingester Hand-over" | ||
| linkTitle: "Ingester Hand-over" | ||
| weight: 5 | ||
| slug: ingester-handover |
There was a problem hiding this comment.
@khaines This comes from a bad copy-paste. May you submit a PR to fix it, please?
There was a problem hiding this comment.
yes it did... looked at it a few times and this didn't catch my eye.
|
|
||
| ## Configuration | ||
|
|
||
| Cortex can be configured to consider an availability zone value in its replication system. Doing so mitigates risks associated with losing multiple nodes with in the same availability zone. The availability zone for an ingester can be defined on the command line of the ingester using the `ingester.availability-zone` flag or using the yaml configuration: |
There was a problem hiding this comment.
@khaines with in > within
| @@ -103,6 +104,7 @@ func (cfg *LifecyclerConfig) RegisterFlagsWithPrefix(prefix string, f *flag.Flag | |||
| f.StringVar(&cfg.Addr, prefix+"lifecycler.addr", "", "IP address to advertise in consul.") | |||
| f.IntVar(&cfg.Port, prefix+"lifecycler.port", 0, "port to advertise in consul (defaults to server.grpc-listen-port).") | |||
| f.StringVar(&cfg.ID, prefix+"lifecycler.ID", hostname, "ID to register into consul.") | |||
| f.StringVar(&cfg.Zone, prefix+"availability-zone", "", "The availability zone of the host, this instance is running on. Default is the lifecycler ID.") | |||
There was a problem hiding this comment.
@khaines Default is the lifecycler ID. This looks more an internal implementation detail than something we should let the final user know. For the final user the default is an empty string, which means the zone-awareness is disabled.
There was a problem hiding this comment.
Made an edit to this with #2384
What this PR does: This PR adds support for defining an availability zone for ingesters so that metric writes & reads will be assigned across zone boundaries. This is added as a CLI flag or config file field, as this feature is agnostic to how that particular is set/discovered; as it varies greatly based on the given environment configuration.
Which issue(s) this PR fixes:
Fixes #612
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]