Add completion count in SearchBackpressure nodeStats API

[kaushalmahi12]

Description

With this change the nodeStats API will also return the completionCount of search tasks at shard and coordinator level.

new sample response for an API call like curl "localhost:9200/_nodes/stats/search_backpressure?pretty" will look like the following

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "runTask",
  "nodes" : {
    "dPpgbce2STSxlvH7LDaGiA" : {
      "timestamp" : 1693433903192,
      "name" : "runTask-0",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1:9300",
      "roles" : [
        "cluster_manager",
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes" : {
        "testattr" : "test",
        "shard_indexing_pressure_enabled" : "true"
      },
      "search_backpressure" : {
        "search_task" : {
          "resource_tracker_stats" : {
            "cpu_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "elapsed_time_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "heap_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_bytes" : 0,
              "current_avg_bytes" : 0,
              "rolling_avg_bytes" : 0
            }
          },
          "completion_count" : 0,
          "cancellation_stats" : {
            "cancellation_count" : 0,
            "cancellation_limit_reached_count" : 0
          }
        },
        "search_shard_task" : {
          "resource_tracker_stats" : {
            "cpu_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "elapsed_time_tracker" : {
              "cancellation_count" : 0,
              "current_max_millis" : 0,
              "current_avg_millis" : 0
            },
            "heap_usage_tracker" : {
              "cancellation_count" : 0,
              "current_max_bytes" : 0,
              "current_avg_bytes" : 0,
              "rolling_avg_bytes" : 0
            }
          },
          "completion_count" : 0,
          "cancellation_stats" : {
            "cancellation_count" : 0,
            "cancellation_limit_reached_count" : 0
          }
        },
        "mode" : "monitor_only"
      },
      "is_node_under_duress" : false
    }
  }
}

Related Issues

Resolves #8698

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20211/
• CommitID: f38a21a
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[PritLadani]

Shall we also include a metric suggesting whether the node is under duress or not? It will be useful for investigation.
IsNodeUnderDuress can be added in the SearchBackpressureStats pojo.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20466/
• CommitID: e0cf424
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21018/
• CommitID: e94c4ab
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21024/
• CommitID: c0344b4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25412/
• CommitID: 9d497bf
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25413/
• CommitID: ec40d92
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/25414/
• CommitID: 1f56190
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[stephen-crawford]

Please rebase--this says 331 commits are involved with this change. Thank you

[kaushalmahi12]

Closing this PR in favor of #10028 due to DCO validation failure in a past commit.