Backport of improve client RPC metrics consistency into release/1.17.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #19721 to be assessed for backporting due to the inclusion of the label backport/1.17.

The below text is copied from the body of the original PR.

---

The client.rpc metric now excludes internal retries for consistency with client.rpc.exceeded and client.rpc.failed. All of these metrics now increment at most once per RPC method call, allowing for accurate calculation of failure / rate limit application occurrence.

Additionally, if an RPC fails because no servers are present, client.rpc.failed is now incremented.

Note: The client.rpc.failed metric used to increment on internal retries as well, but that was changed in this commit (CC @kisunji).

Testing & Reproduction steps

Manual testing conducted by reviewer (@rboyer).

1. Set up Consul cluster.
2. Run curl -sL -XPUT 172.17.0.3:8500/v1/catalog/register -d'{"Node":"foo", "Address":"4.4.4.4", "ID":"blah"}'. Note: this should succeed on the first try.
3. Run curl -sL '172.17.0.3:8500/v1/catalog/nodes?filter=^^^''. Note: this should fail and have several internal retries.
4. Read relevant metrics: $ curl -sL 172.17.0.3:8500/v1/agent/metrics | jq '.Counters[] | select (.Name | test("^consul\\.client\\.rpc"))'

Expect:

• consul.client.rpc: 2
• consul.client.rpc.failed: 1

Actual: Matches expectation

{
  "Name": "consul.client.rpc",
  "Count": 2,
  "Rate": 0.2,
  "Sum": 2,
  "Min": 1,
  "Max": 1,
  "Mean": 1,
  "Stddev": 0,
  "Labels": {}
}
{
  "Name": "consul.client.rpc.failed",
  "Count": 1,
  "Rate": 0.1,
  "Sum": 1,
  "Min": 1,
  "Max": 1,
  "Mean": 1,
  "Stddev": 0,
  "Labels": {
    "server": "651d982bed6b"
  }
}

PR Checklist

• updated test coverage: no - manual test performed, no existing applicable metrics test to modify
• external facing docs updated: Will be updated in a separate PR if applicable
• appropriate backport labels added
• not a security concern

Additional Notes

Added the pr/no-metrics-test label since the affected metric predates that CI check, currently has no tests, and correct behavior was manually verified as a part of this review.

---

Overview of commits

• 06507fe - 14e1605 - 088ec70

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR