feat: add metric 'check_cluster_health_total' and 'sync_operation_total'

[Sindweller]

• Why submit this pull request?

• Bugfix

• New feature provided

• Improve performance

• Backport patches

• Related issues

---

New feature or improvement

• Describe the details and related test reports.
  add two Promethues metrics for apisix-ingress-controller:

---

HELP apisix_ingress_controller_check_cluster_health_success Number of cluster health check operations
TYPE apisix_ingress_controller_check_cluster_health_success counter
Label:

• controller_class
• controller_namespace
• controller_pod
• name (cluster name)

---

HELP apisix_ingress_controller_sync_success Number of success sync operations
TYPE apisix_ingress_controller_sync_success counter
Labels:

• controller_class
• controller_namespace
• controller_pod
• resource ( cache, schema, ssl, endpoint, consumer)
• result (success, failure)

---

Show in Prometheus:

# HELP apisix_ingress_controller_check_cluster_health_total Number of cluster health check operations
# TYPE apisix_ingress_controller_check_cluster_health_total counter
apisix_ingress_controller_check_cluster_health{controller_namespace="default",controller_pod="",name="default"} 1
# HELP apisix_ingress_controller_sync_operation_total Number of sync operations
# TYPE apisix_ingress_controller_sync_operation_total counter
apisix_ingress_controller_sync_success_total{controller_namespace="default",controller_pod="",resource="schema",result="success"} 1

---

[Sindweller]

for the metric: apisix_ingress_controller_sync_success, how to describe the following:
cache, schema, ssl, endpoint, consumer
with a label? <Label: resource> is not suitable.

[Sindweller]

since we put podName and podNamespace inside NewPrometheusCollector(), transfer param "test" to collector is disabled.
so the test result of controller_pod should be "".

[tao12345666333]

--- FAIL: TestPrometheusCollector (0.00s)
    --- FAIL: TestPrometheusCollector/sync_operation_total (0.00s)
        prometheus_test.go:159: 
            	Error Trace:	prometheus_test.go:159
            	Error:      	Not equal: 
            	            	expected: "failure"
            	            	actual  : "error"
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1 +1 @@
            	            	-failure
            	            	+error
            	Test:       	TestPrometheusCollector/sync_operation_total
FAIL

[Sindweller]

--- FAIL: TestPrometheusCollector (0.00s)
    --- FAIL: TestPrometheusCollector/sync_operation_total (0.00s)
        prometheus_test.go:159: 
            	Error Trace:	prometheus_test.go:159
            	Error:      	Not equal: 
            	            	expected: "failure"
            	            	actual  : "error"
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1 +1 @@
            	            	-failure
            	            	+error
            	Test:       	TestPrometheusCollector/sync_operation_total
FAIL

done.

[codecov-commenter]

Codecov Report

Merging #627 (7ba9956) into master (f78248a) will decrease coverage by 0.04%.
The diff coverage is 67.27%.

@@            Coverage Diff             @@
##           master     #627      +/-   ##
==========================================
- Coverage   34.79%   34.74%   -0.05%     
==========================================
  Files          57       57              
  Lines        5722     5773      +51     
==========================================
+ Hits         1991     2006      +15     
- Misses       3492     3521      +29     
- Partials      239      246       +7     

Impacted Files	Coverage Δ
pkg/ingress/apisix_consumer.go	0.00% <0.00%> (ø)
pkg/ingress/apisix_tls.go	0.00% <0.00%> (ø)
pkg/ingress/controller.go	1.06% <0.00%> (-0.01%)	⬇️
pkg/ingress/endpoint.go	0.00% <0.00%> (ø)
pkg/ingress/endpointslice.go	0.00% <0.00%> (ø)
pkg/ingress/secret.go	0.00% <0.00%> (ø)
pkg/apisix/cluster.go	27.25% <60.00%> (-3.23%)	⬇️
pkg/metrics/prometheus.go	84.07% <89.47%> (+2.49%)	⬆️
pkg/apisix/plugin.go	80.00% <0.00%> (-20.00%)	⬇️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f78248a...7ba9956. Read the comment docs.