[Performance] Optimize String terms agg

[vinaykpud]

Describe the bug

Problem
String terms aggregations has similar inefficiencies as numeric terms(#18703), with redundant sorting and suboptimal top-N selection.

Solution
Applied analogous optimizations to string terms aggregations, avoiding double sorting and optimizing bucket selection based on N relative to bucket counts.
rishabhmaurya@27d71e8

Impact
Achieved performance improvements, enhancing aggregation efficiency for string-term aggregation queries.

Related component

Search:Performance

Additional Details

Plugins
NA

Screenshots
NA

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[vinaykpud]

Benchmark Testing

Test Setup

Cluster Configuration:

• Test approach: Comparative testing with and without the optimization
• Dataset: big5 dataset with 202,000 documents ingested

Data Ingestion:

opensearch-benchmark execute-test --workload=big5 --target-hosts=localhost:9200 \
  --kill-running-processes --workload-params "corpus_size:60,ingest_percentage:0.29" \
  --exclude-tasks="check-cluster-health"

Index Enhancement:

• Additional field added to the index: Custom generated keyword data with 100,000 cardinality
• Total buckets: 202,000 (maintaining high cardinality-to-document ratio)

Benchmark Execution:

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=big5 \
  --target-hosts=localhost:9200 --kill-running-processes \
  --include-tasks "keyword-terms" --telemetry=node-stats

• Target task: keyword-terms

Modified it like bellow and updated the size based on the run.

 {
      "name": "keyword-terms",
      "operation-type": "search",
      "index": "{{index_name | default('big5')}}",
      "body": {
        "size": 0,
        "aggs": {
          "station": {
            "terms": {
              "field": "word",
              "size": 6700
            }
          }
        }
      }
}

• Iteration details for this task: keyword-terms

Test Scenarios

Scenario 1: Request Size Exceeds Bucket Count

Configuration:

• Request size: 100,000
• Shard size calculation: size × 1.5 + 10 = 150,010

Without the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        0.55 |  ops/s |
|                                                Mean Throughput | keyword-terms |        0.55 |  ops/s |
|                                              Median Throughput | keyword-terms |        0.55 |  ops/s |
|                                                 Max Throughput | keyword-terms |        0.55 |  ops/s |
|                                        50th percentile latency | keyword-terms |      327116 |     ms |
|                                        90th percentile latency | keyword-terms |      378942 |     ms |
|                                        99th percentile latency | keyword-terms |      390588 |     ms |
|                                       100th percentile latency | keyword-terms |      391233 |     ms |
|                                   50th percentile service time | keyword-terms |     1775.57 |     ms |
|                                   90th percentile service time | keyword-terms |     1793.25 |     ms |
|                                   99th percentile service time | keyword-terms |     1810.02 |     ms |
|                                  100th percentile service time | keyword-terms |     1813.05 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

With the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.51 |  ops/s |
|                                                Mean Throughput | keyword-terms |        1.52 |  ops/s |
|                                              Median Throughput | keyword-terms |        1.52 |  ops/s |
|                                                 Max Throughput | keyword-terms |        1.52 |  ops/s |
|                                        50th percentile latency | keyword-terms |     39361.4 |     ms |
|                                        90th percentile latency | keyword-terms |     45191.5 |     ms |
|                                        99th percentile latency | keyword-terms |     46488.1 |     ms |
|                                       100th percentile latency | keyword-terms |     46570.4 |     ms |
|                                   50th percentile service time | keyword-terms |     626.015 |     ms |
|                                   90th percentile service time | keyword-terms |     643.689 |     ms |
|                                   99th percentile service time | keyword-terms |     664.456 |     ms |
|                                  100th percentile service time | keyword-terms |     664.565 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

Scenario 2: Request size close to the number of buckets

size : 55000

Without the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        0.61 |  ops/s |
|                                                Mean Throughput | keyword-terms |        0.61 |  ops/s |
|                                              Median Throughput | keyword-terms |        0.61 |  ops/s |
|                                                 Max Throughput | keyword-terms |        0.61 |  ops/s |
|                                        50th percentile latency | keyword-terms |      286354 |     ms |
|                                        90th percentile latency | keyword-terms |      331596 |     ms |
|                                        99th percentile latency | keyword-terms |      341759 |     ms |
|                                       100th percentile latency | keyword-terms |      342336 |     ms |
|                                   50th percentile service time | keyword-terms |     1622.32 |     ms |
|                                   90th percentile service time | keyword-terms |      1641.3 |     ms |
|                                   99th percentile service time | keyword-terms |     1656.39 |     ms |
|                                  100th percentile service time | keyword-terms |      1660.9 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

With the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.15 |  ops/s |
|                                                Mean Throughput | keyword-terms |        1.16 |  ops/s |
|                                              Median Throughput | keyword-terms |        1.16 |  ops/s |
|                                                 Max Throughput | keyword-terms |        1.16 |  ops/s |
|                                        50th percentile latency | keyword-terms |     90826.3 |     ms |
|                                        90th percentile latency | keyword-terms |      104996 |     ms |
|                                        99th percentile latency | keyword-terms |      108146 |     ms |
|                                       100th percentile latency | keyword-terms |      108322 |     ms |
|                                   50th percentile service time | keyword-terms |      839.26 |     ms |
|                                   90th percentile service time | keyword-terms |     860.183 |     ms |
|                                   99th percentile service time | keyword-terms |     876.299 |     ms |
|                                  100th percentile service time | keyword-terms |     877.297 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

Scenario 3: Request size is just above the 20% of bucketSize

size: 13400

Withou the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.89 |  ops/s |
|                                                Mean Throughput | keyword-terms |         1.9 |  ops/s |
|                                              Median Throughput | keyword-terms |         1.9 |  ops/s |
|                                                 Max Throughput | keyword-terms |         1.9 |  ops/s |
|                                        50th percentile latency | keyword-terms |     6400.26 |     ms |
|                                        90th percentile latency | keyword-terms |     7075.83 |     ms |
|                                        99th percentile latency | keyword-terms |     7227.96 |     ms |
|                                       100th percentile latency | keyword-terms |     7233.64 |     ms |
|                                   50th percentile service time | keyword-terms |     512.689 |     ms |
|                                   90th percentile service time | keyword-terms |     526.873 |     ms |
|                                   99th percentile service time | keyword-terms |      545.52 |     ms |
|                                  100th percentile service time | keyword-terms |     555.157 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

With the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.98 |  ops/s |
|                                                Mean Throughput | keyword-terms |        1.98 |  ops/s |
|                                              Median Throughput | keyword-terms |        1.98 |  ops/s |
|                                                 Max Throughput | keyword-terms |        1.98 |  ops/s |
|                                        50th percentile latency | keyword-terms |      384.64 |     ms |
|                                        90th percentile latency | keyword-terms |     400.091 |     ms |
|                                        99th percentile latency | keyword-terms |     445.032 |     ms |
|                                       100th percentile latency | keyword-terms |     447.561 |     ms |
|                                   50th percentile service time | keyword-terms |     383.181 |     ms |
|                                   90th percentile service time | keyword-terms |     398.664 |     ms |
|                                   99th percentile service time | keyword-terms |     443.744 |     ms |
|                                  100th percentile service time | keyword-terms |     446.919 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

Scenario 4: Request size bellow 20% of bucket size

size: 6700

Without the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.99 |  ops/s |
|                                                Mean Throughput | keyword-terms |        1.99 |  ops/s |
|                                              Median Throughput | keyword-terms |        1.99 |  ops/s |
|                                                 Max Throughput | keyword-terms |        1.99 |  ops/s |
|                                        50th percentile latency | keyword-terms |     230.428 |     ms |
|                                        90th percentile latency | keyword-terms |     247.177 |     ms |
|                                        99th percentile latency | keyword-terms |     264.711 |     ms |
|                                       100th percentile latency | keyword-terms |     275.375 |     ms |
|                                   50th percentile service time | keyword-terms |     229.189 |     ms |
|                                   90th percentile service time | keyword-terms |      245.87 |     ms |
|                                   99th percentile service time | keyword-terms |     263.389 |     ms |
|                                  100th percentile service time | keyword-terms |     274.248 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

With the change:

|                                                  Segment count |               |           9 |        |
|                                                 Min Throughput | keyword-terms |        1.98 |  ops/s |
|                                                Mean Throughput | keyword-terms |        1.99 |  ops/s |
|                                              Median Throughput | keyword-terms |        1.99 |  ops/s |
|                                                 Max Throughput | keyword-terms |        1.99 |  ops/s |
|                                        50th percentile latency | keyword-terms |     213.564 |     ms |
|                                        90th percentile latency | keyword-terms |      229.39 |     ms |
|                                        99th percentile latency | keyword-terms |     238.325 |     ms |
|                                       100th percentile latency | keyword-terms |     238.657 |     ms |
|                                   50th percentile service time | keyword-terms |     211.716 |     ms |
|                                   90th percentile service time | keyword-terms |     227.405 |     ms |
|                                   99th percentile service time | keyword-terms |       236.1 |     ms |
|                                  100th percentile service time | keyword-terms |     236.644 |     ms |
|                                                     error rate | keyword-terms |           0 |      % |

So in the 1, 2, 3 cases above we observed huge improvement in the latency with this change. shows 70% to 90% p90 improvements in the latency. In the 4th case we have 8% p90 improvements in the latency. The optimization delivers improvements for keyword field aggregations, particularly for large request sizes.

Bellow graph shows all the latency comparisons:

Bellow graph shows the heap used, we don't see much difference when we compared with the change.

Bellow graph shows the CPU used, we don't see much difference when we compared with the change.