Add k8s_metrics track to benchmark refreshes during indexing (#425)

* Add k8s_metrics track to benchmark refreshes during indexing; aka search after write.
* Support refresh_interval as a track parameter
* Skip testing k8s_metrics against stateful Elasticsearch

it/test_all_tracks_and_challenges.py

@@ -21,8 +21,11 @@
 
 
 class TestTrackRepository:
-    skip_tracks = ["elastic/logs", "elastic/security", "sql"]
-    disable_assertions = {"http_logs": ["append-no-conflicts", "runtime-fields"], "nyc_taxis": ["update-aggs-only"]}
+    skip_tracks = ["elastic/logs", "elastic/security", "k8s_metrics", "sql"]
+    disable_assertions = {
+        "http_logs": ["append-no-conflicts", "runtime-fields"],
+        "nyc_taxis": ["update-aggs-only"],
+    }
     skip_challenges = ["esql"]
 
     def test_autogenerated(self, es_cluster, rally, track, challenge, rally_options):

k8s_metrics/README.md

@@ -0,0 +1,73 @@
+# Kubernetes (K8s) Metrics Track
+K8s Metrics is a track intended for benchmarking refresh on Elasticsearch using a Kubernetes pod metrics dataset. A small corpus of Kibana object definition documents is included for indexing with fast refresh enabled.
+
+## Track Parameters
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `bulk_indexing_clients` | `8` | The number of bulk indexing clients. |
+| `bulk_refresh` | `unset` | Control the refresh behavior for bulk requests. Valid values are `true`, `wait_for`, `false`, or unset. This parameter is always specified as a string. |
+| `bulk_size` | `1000` | The batch size of bulk requests. |
+| `fast_refresh_bulk_size` | `1` | The bulk batch size of the fast refresh index. |
+| `fast_refresh_clients` | `1` | The number of bulk indexing clients for fast refresh indexing. |
+| `fast_refresh_indexing_interval` | `15` | The interval, in seconds, for indexing to the fast refresh index. |
+| `ingest_percentage` | `100` | The percentage of the document corpus to index. |
+| `manual_refresh_clients` | `1` | The number of clients to use for manual refresh operations. |
+| `manual_refresh_interval` | `15` | The interval, in seconds, for issuing manual refresh requests. |
+| `number_of_replicas` | `1` | The number of replicas to allocate to the corpus data stream. |
+| `number_of_shards` | `1` | The number of shards to allocate to the corpus data stream. |
+| `refresh_interval` | `unset` | The target data stream refresh interval; e.g., `5s`. If unset, the Elasticsearch default refresh interval is used. |
+
+## Challenges
+
+### `append-no-conflicts-metrics-index-with-refresh` (default)
+
+Index a metrics document corpus and sets the bulk API `refresh` query parameter. By default, Elasticsearch will perform refreshes asynchronously. This challenge is intended to capture refresh latency and the indexing throughput impact of refreshes during bulk indexing. This challenge can be executed against both stateful and serverless Elasticsearch.
+
+#### Parameters
+
+* `bulk_indexing_clients` (default: `8`)
+* `bulk_refresh` (default: `"true"`)
+* `ingest_percentage` (default: `100`)
+* `number_of_replicas` (default: `1`)
+* `number_of_shards` (default: `1`)
+
+### `append-no-conflicts-metrics-index-with-intermittent-refresh`
+
+Index a metrics document corpus while performing intermittent manual refreshes of the target data stream at the specified interval. This challenge is intended to capture refresh latency and the indexing throughput impact of refreshes. This challenge can be executed against both stateful and serverless Elasticsearch.
+
+#### Parameters
+
+* `bulk_indexing_clients` (default: `8`)
+* `ingest_percentage` (default: `100`)
+* `manual_refresh_clients` (default: `1`)
+* `manual_refresh_interval` (default: `15`)
+* `number_of_replicas` (default: `1`)
+* `number_of_shards` (default: `1`)
+* `refresh_interval` (defaulit: `unset`)
+
+### `append-no-conflicts-metrics-index-only`
+
+Index a metrics document corpus. This challenge can be used as a baseline when comparing benchmarking results with other refresh-oriented track challenges. This challenge can be executed against both stateful and serverless Elasticsearch.
+
+#### Parameters
+
+* `bulk_indexing_clients` (default: `8`)
+* `ingest_percentage` (default: `100`)
+* `number_of_replicas` (default: `1`)
+* `number_of_shards` (default: `1`)
+* `refresh_interval` (defaulit: `unset`)
+
+### `append-no-conflicts-metrics-with-fast-refresh`
+
+Index a metrics document corpus while indexing a small Kibana corpus to a separate fast refresh index. This challange simulates indexing to a smaller index with fast refresh enabled while concurrently bulk indexing to a larger data stream. The race ends once all fast refresh documents have been indexed. Test mode is not supported and will result in an error since the fast refresh corpus contains less than 1000 documents. This challenge can be executed against serverless Elasticsearch.
+
+#### Parameters
+
+* `bulk_indexing_clients` (default: `8`)
+* `ingest_percentage` (default: `100`)
+* `fast_refresh_clients` (default: `1`)
+* `fast_refresh_indexing_interval` (default: `30`)
+* `number_of_replicas` (default: `1`)
+* `number_of_shards` (default: `1`)
+* `refresh_interval` (defaulit: `unset`)

k8s_metrics/challenges/default.json

@@ -0,0 +1,252 @@
+{
+  "name": "append-no-conflicts-metrics-index-with-refresh",
+  "description": "Index a metrics document corpus with control over refresh behavior.",
+  "default": true,
+  "schedule": [
+    {
+      "name": "delete-data-streams",
+      "operation": {
+        "operation-type": "delete-data-stream"
+      }
+    },
+    {
+      "name": "delete-track-composable-templates",
+      "operation": {
+        "operation-type": "delete-composable-template",
+        "template": "metrics-kubernetes.pod-template"
+      }
+    },
+    {
+      "name": "delete-track-component-templates",
+      "operation": {
+        "operation-type": "delete-component-template"
+      }
+    },
+    {
+      "name": "create-track-component-templates",
+      "operation": {
+        "operation-type": "create-component-template"
+      }
+    },
+    {
+      "name": "create-track-composable-templates",
+      "operation": {
+        "operation-type": "create-composable-template"
+      }
+    },
+    {
+      "operation": "create-metrics-datastream"
+    },
+    {
+      "name": "check-cluster-health",
+      "operation": {
+        "operation-type": "cluster-health",
+        "request-params": {
+          "wait_for_status": "{{cluster_health | default('green')}}",
+          "wait_for_no_relocating_shards": "true"
+        },
+        "retry-until-success": true
+      }
+    },
+    {
+      "operation": "bulk-index-metrics-with-refresh",
+      "clients": {{bulk_indexing_clients | default(8)}}
+    }
+  ]
+},
+{
+  "name": "append-no-conflicts-metrics-index-with-intermittent-refresh",
+  "description": "Index a metrics document corpus while performing intermittent refreshes of the target data stream.",
+  "schedule": [
+    {
+      "name": "delete-data-streams",
+      "operation": {
+        "operation-type": "delete-data-stream"
+      }
+    },
+    {
+      "name": "delete-track-composable-templates",
+      "operation": {
+        "operation-type": "delete-composable-template",
+        "template": "metrics-kubernetes.pod-template"
+      }
+    },
+    {
+      "name": "delete-track-component-templates",
+      "operation": {
+        "operation-type": "delete-component-template"
+      }
+    },
+    {
+      "name": "create-track-component-templates",
+      "operation": {
+        "operation-type": "create-component-template"
+      }
+    },
+    {
+      "name": "create-track-composable-templates",
+      "operation": {
+        "operation-type": "create-composable-template"
+      }
+    },
+    {
+      "operation": "create-metrics-datastream"
+    },
+    {
+      "name": "check-cluster-health",
+      "operation": {
+        "operation-type": "cluster-health",
+        "request-params": {
+          "wait_for_status": "{{cluster_health | default('green')}}",
+          "wait_for_no_relocating_shards": "true"
+        },
+        "retry-until-success": true
+      }
+    },
+    {
+      "parallel": {
+        "completed-by": "bulk-index-metrics",
+        "tasks": [
+          {
+            "operation": "bulk-index-metrics",
+            "clients": {{bulk_indexing_clients | default(8)}}
+          },
+          {
+            "operation": "intermittent-manual-refresh",
+            "clients": {{manual_refresh_clients | default(1)}},
+            "target-interval": {{manual_refresh_interval | default(15)}}
+          }
+        ]
+      }
+    }
+  ]
+},
+{
+  "name": "append-no-conflicts-metrics-index-only",
+  "description": "Index a metrics document corpus.",
+  "schedule": [
+    {
+      "name": "delete-data-streams",
+      "operation": {
+        "operation-type": "delete-data-stream"
+      }
+    },
+    {
+      "name": "delete-track-composable-templates",
+      "operation": {
+        "operation-type": "delete-composable-template",
+        "template": "metrics-kubernetes.pod-template"
+      }
+    },
+    {
+      "name": "delete-track-component-templates",
+      "operation": {
+        "operation-type": "delete-component-template"
+      }
+    },
+    {
+      "name": "create-track-component-templates",
+      "operation": {
+        "operation-type": "create-component-template"
+      }
+    },
+    {
+      "name": "create-track-composable-templates",
+      "operation": {
+        "operation-type": "create-composable-template"
+      }
+    },
+    {
+      "operation": "create-metrics-datastream"
+    },
+    {
+      "name": "check-cluster-health",
+      "operation": {
+        "operation-type": "cluster-health",
+        "request-params": {
+          "wait_for_status": "{{cluster_health | default('green')}}",
+          "wait_for_no_relocating_shards": "true"
+        },
+        "retry-until-success": true
+      }
+    },
+    {
+      "operation": "bulk-index-metrics",
+      "clients": {{bulk_indexing_clients | default(8)}}
+    }
+  ]
+},
+{
+  "name": "append-no-conflicts-metrics-with-fast-refresh",
+  "description": "Index a metrics document corpus while indexing a small Kibana corpus to a separate fast refresh index.",
+  "schedule": [
+    {
+      "name": "delete-data-streams",
+      "operation": {
+        "operation-type": "delete-data-stream"
+      }
+    },
+    {
+      "name": "delete-track-composable-templates",
+      "operation": {
+        "operation-type": "delete-composable-template",
+        "template": "metrics-kubernetes.pod-template" 
+      }
+    },
+    {
+      "name": "delete-fast-refresh-templates",
+      "operation": {
+        "operation-type": "delete-composable-template",
+        "template": "k8s_metrics-template" 
+      }
+    },
+    {
+      "name": "delete-track-component-templates",
+      "operation": {
+        "operation-type": "delete-component-template"
+      }
+    },
+    {
+      "name": "create-track-component-templates",
+      "operation": {
+        "operation-type": "create-component-template"
+      }
+    },
+    {
+      "name": "create-track-composable-templates",
+      "operation": {
+        "operation-type": "create-composable-template"
+      }
+    },
+    {
+      "operation": "create-metrics-datastream"
+    },
+    {
+      "name": "check-cluster-health",
+      "operation": {
+        "operation-type": "cluster-health",
+        "request-params": {
+          "wait_for_status": "{{cluster_health | default('green')}}",
+          "wait_for_no_relocating_shards": "true"
+        },
+        "retry-until-success": true
+      }
+    },
+    {
+      "parallel": {
+        "completed-by": "fast-refresh-index",
+        "tasks": [
+          {
+            "operation": "bulk-index-metrics",
+            "clients": {{bulk_indexing_clients | default(8)}}
+          },
+          {
+            "operation": "fast-refresh-index",
+            "clients": {{fast_refresh_clients | default(1)}},
+            "target-interval": {{fast_refresh_indexing_interval | default(30)}}
+          }
+        ]
+      }
+    }
+  ]
+}

k8s_metrics/operations/default.json

@@ -0,0 +1,33 @@
+{
+  "name": "create-metrics-datastream",
+  "operation-type": "create-data-stream",
+  "data-stream": "metrics-kubernetes.pod-rally"
+},
+{
+  "name": "bulk-index-metrics",
+    "operation-type": "bulk",
+    "bulk-size": {{bulk_size | default(1000)}},
+    "corpora": "kubernetes-pod-metrics",
+    "ingest-percentage": {{ingest_percentage | default(100)}}
+},
+{
+  "name": "bulk-index-metrics-with-refresh",
+    "operation-type": "bulk",
+    "bulk-size": {{bulk_size | default(1000)}},
+    "corpora": "kubernetes-pod-metrics",
+    "ingest-percentage": {{ingest_percentage | default(100)}},
+    "refresh": "{{bulk_refresh | default('true')}}"
+},
+{
+  "name": "fast-refresh-index",
+    "operation-type": "bulk",
+    "bulk-size": {{fast_refresh_bulk_size | default(1)}},
+    "corpora": "k8s-integration",
+    "include-in-reporting": true
+},
+{
+  "name": "intermittent-manual-refresh",
+  "operation-type": "refresh",
+  "index": "metrics-kubernetes.pod-rally",
+  "include-in-reporting": true
+}