mimir-jsonnet: allow disabling ingester anti-affinity (grafana#1581)

* mimir-jsonnet: allow disabling ingester anti-affinity While setting up Mimir on my personal toy-cluster of one node, I got stuck scheduling the ingesters as they have the anti-affinity rules by default and it can't be easily removed without redefining the statefulset. There are similar configs for distributor, querier, etc., so I just added another one, and documented it a little bit. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * make doc Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Also allow disabling anti-affinity on multi-zone ingesters Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Fix multi-zone Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
andrew-waters · Mar 31, 2022 · 3f09c89 · 3f09c89
1 parent 7f5bf4b
commit 3f09c89
Show file tree

Hide file tree

Showing 5 changed files with 33 additions and 2 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -31,6 +31,7 @@
 ### Jsonnet
 
 * [BUGFIX] Pass primary and secondary multikv stores via CLI flags. Introduced new `multikv_switch_primary_secondary` config option to flip primary and secondary in runtime config.
+* [ENHANCEMENT] Ingester anti-affinity can now be disabled by using `ingester_allow_multiple_replicas_on_same_node` configuration key. #1581
 
 ### Mimirtool
 

diff --git a/operations/mimir/README.md b/operations/mimir/README.md
@@ -2,6 +2,8 @@
 
 This folder has the jsonnet for deploying Mimir in Kubernetes. To generate the YAMLs for deploying Mimir follow these instructions.
 
+## Quick start
+
 ### 1. Make sure you have `tanka` and `jb` installed
 
 Follow the steps at https://tanka.dev/install. If you have `go` installed locally you can also use:
@@ -51,3 +53,28 @@ To output YAML manifests to `./manifests`, run:
 $ cd jsonnet-example
 $ tk export manifests environments/default
 ```
+
+## Configuration
+
+### Anti-affinity
+
+Given the distributed nature of Mimir, both performance and reliability are improved when pods are spread across different nodes: for example, multiple queriers can be processing the same query at the same time, so it's better to distribute them across different nodes, and since losing multiple ingesters can cause data loss, it's also important to have them spread.
+This is why, by default, anti-affinity rules are applied to some deployments and statefulsets.
+This can become an issue when playing with Mimir in a single-node kubernetes cluster, so anti-affinity can be disabled by enabling the configuration values `_config.<component>_allow_multiple_replicas_on_same_node`.
+
+For example:
+
+```jsonnet
+local mimir = 'mimir/mimir.libsonnet';
+
+mimir {
+  _config+:: {
+    // ... default configuration values
+    distributor_allow_multiple_replicas_on_same_node: true,
+    ingester_allow_multiple_replicas_on_same_node: true,
+    ruler_allow_multiple_replicas_on_same_node: true,
+    querier_allow_multiple_replicas_on_same_node: true,
+    query_frontend_allow_multiple_replicas_on_same_node: true,
+  },
+}
+```
diff --git a/operations/mimir/config.libsonnet b/operations/mimir/config.libsonnet
@@ -15,7 +15,10 @@
     unregister_ingesters_on_shutdown: true,
 
     // Controls whether multiple pods for the same service can be scheduled on the same node.
+    // Distributing the pods over different nodes improves performance and also realiability,
+    // especially important in case of ingester where losing multiple ingesters can cause data loss.
     distributor_allow_multiple_replicas_on_same_node: false,
+    ingester_allow_multiple_replicas_on_same_node: false,
     ruler_allow_multiple_replicas_on_same_node: false,
     querier_allow_multiple_replicas_on_same_node: false,
     query_frontend_allow_multiple_replicas_on_same_node: false,

diff --git a/operations/mimir/ingester.libsonnet b/operations/mimir/ingester.libsonnet
@@ -95,7 +95,7 @@
     statefulSet.mixin.spec.withPodManagementPolicy('Parallel') +
     (if with_anti_affinity then $.util.antiAffinity else {}),
 
-  ingester_statefulset: self.newIngesterStatefulSet('ingester', $.ingester_container),
+  ingester_statefulset: self.newIngesterStatefulSet('ingester', $.ingester_container, !$._config.ingester_allow_multiple_replicas_on_same_node),
 
   ingester_service:
     $.util.serviceFor($.ingester_statefulset, $._config.service_ignored_labels),

diff --git a/operations/mimir/multi-zone.libsonnet b/operations/mimir/multi-zone.libsonnet
@@ -94,7 +94,7 @@
     statefulSet.mixin.spec.updateStrategy.withType('OnDelete') +
     statefulSet.mixin.spec.template.spec.withTerminationGracePeriodSeconds(1200) +
     statefulSet.mixin.spec.withReplicas(std.ceil($._config.multi_zone_ingester_replicas / 3)) +
-    {
+    if $._config.ingester_allow_multiple_replicas_on_same_node then {} else {
       spec+:
         // Allow to schedule 2+ ingesters in the same zone on the same node, but do not schedule 2+ ingesters in
         // different zones on the same node. In case of 1 node failure in the Kubernetes cluster, only ingesters