[feat] Linode Client metrics

[rsienko]

General:

• Have you removed all sensitive information, including but not limited to access keys and passwords?
• Have you checked to ensure there aren't other open or closed Pull Requests for the same bug/feature/question?

Pull Request Guidelines:

1. Does your submission pass tests?
2. Have you added tests?
3. Are you addressing a single feature in this PR?
4. Are your commits atomic, addressing one change per commit?
5. Are you following the conventions of the language?
6. Have you saved your large formatting changes for a different PR, so we can focus on your work?
7. Have you explained your rationale for why this feature is needed?
8. Have you linked your PR to an issue

[rsienko]

Example new metrics, visible right after start (requires --authorization-always-allow-paths="/metrics").

# HELP ccm_linode_client_requests_total client counters for each operation and its result
# TYPE ccm_linode_client_requests_total counter
ccm_linode_client_requests_total{method="ListInstances",result="ok"} 1
ccm_linode_client_requests_total{method="ListNodeBalancerConfigs",result="ok"} 1
ccm_linode_client_requests_total{method="ListNodeBalancerFirewalls",result="ok"} 1
ccm_linode_client_requests_total{method="ListNodeBalancerNodes",result="ok"} 1
ccm_linode_client_requests_total{method="ListNodeBalancers",result="ok"} 1
ccm_linode_client_requests_total{method="ListVPCIPAddresses",result="ok"} 1
ccm_linode_client_requests_total{method="ListVPCs",result="ok"} 1
ccm_linode_client_requests_total{method="RebuildNodeBalancerConfig",result="ok"} 1

If the token is incorrect, this can be inferred from these operations both failing.

# HELP ccm_linode_client_requests_total client counters for each operation and its result
# TYPE ccm_linode_client_requests_total counter
ccm_linode_client_requests_total{method="ListInstances",result="error"} 8
ccm_linode_client_requests_total{method="ListNodeBalancers",result="error"} 6

[codecov]

Codecov Report

Attention: Patch coverage is 8.57143% with 224 lines in your changes missing coverage. Please review.

Project coverage is 64.76%. Comparing base (441aec9) to head (23837ba).
Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
cloud/linode/client/client_with_metrics.go	1.75%	224 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #293      +/-   ##
==========================================
- Coverage   71.07%   64.76%   -6.31%     
==========================================
  Files          12       14       +2     
  Lines        2216     2455     +239     
==========================================
+ Hits         1575     1590      +15     
- Misses        490      714     +224     
  Partials      151      151              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rsienko]

I didn't find a good way to fix coverage for the generated file.

It doesn't look trivial to generate calls to all the methods. Gowrap provides method declaration as a string. I'd need to parse the Interface from AST. Doable but is it worth it?

A generated test case would look like so and it's pretty much all boilerplate.

func TestAddInstanceIPAddress(t *testing.T) {
	ctrl := gomock.NewController(t)
	defer ctrl.Finish()

	mock := mocks.NewMockClient(ctrl)
	expIP := linodego.InstanceIP{}
	var m1 = &dto.Metric{}
	var m2 = &dto.Metric{}

	client := NewClientWithPrometheus(mock)
	mock.EXPECT().AddInstanceIPAddress(gomock.Any(), 1, true).DoAndReturn(
		func(ctx context.Context, id int, isActive bool) (*linodego.InstanceIP, error) {
			return &expIP, nil
		})
	ec, err := ClientMethodCounterVec.GetMetricWithLabelValues("AddInstanceIPAddress", resultOk)
	if err != nil {
		t.Error(err)
	}

	ec.Write(m1)
	retIP, err := client.AddInstanceIPAddress(context.TODO(), 1, true)
	ec.Write(m2)

	assert.Equal(t, m1.Counter.GetValue()+1, m2.Counter.GetValue(), "'ok' counter should be incremented")
	assert.Equal(t, expIP, *retIP, "IP should be returned unmodified")
	assert.Nil(t, err, "nil error should be returned unmodified")
}

Note that Client interface methods expose error returned in case call fails with no response. Thus, an approach using Resty's callbacks would be incomplete.

client.OnAfterResponse(clientMetrics)
...
func clientMetrics(response *resty.Response) (err error) {
	code := response.StatusCode()
	ClientMethodCounterVec.WithLabelValues("CreateFirewallDevice", strconv.Itoa(code)).Inc()
}

Note, it also brings other aspects - retries and path patterns into label cardinality consideration (we cannot expose resource IDs). Example form linodego:

func (c *Client) ListNodeBalancerFirewalls(ctx context.Context, nodebalancerID int, opts *ListOptions) ([]Firewall, error) {
  response, err := getPaginatedResults[Firewall](ctx, c, formatAPIPath("nodebalancers/%d/firewalls", nodebalancerID), opts)
  ...