Improve Monitoring/Alerting/Metrics

[PadmaB]

Story

As a provider I want timely alerts raised based on the metrics to take informed decisions

Motivation

• MCM exposes a number of metrics like number of API calls to Cloud Provider, Freeze status of MCM Enhance metrics endpoint #189
• Define some alerts based on the metrics will help the Ops to react in a timely manner, in case of any action required
• Challenges with Azure Cloud Provider during deletion of machines Improve logic of VM-Deletion and Safey-controller for Azure #200

Acceptance Criteria

• Define alerts for the above situations to take required action

Definition of Done

• Knowledge is distributed: Have you spread your knowledge in pair programming/code review?
• Unit tests are provided: Have you written automated unit tests?
• Integration tests are provided: Have you written automated integration tests?
• Minimum API exposure: If you have added/changed public API, was it really necessary/is it minimal?
• Operations guide: Have you updated the operations guide about ops-relevant changes?
• User documentation: Have you updated the READMEs/docs/how-tos about user-relevant changes?

Possible metrices to add (Rough work)

• we could provide metrices on number of machines with different statuses , so filtering on that can be done (if already not exposed)
• metrics about time taken for machine to join can be added, this will help to know overall average joining time on any provider
• when MCM did scale-up , scale-down and when CA did.
• metices that could solve typical DoD issues, like node not joining.
• how much each resource took to get created like VM, disk especially in Azure.

[prashanth26]

I have tried to at least expose a few crucial metrics into the Gardener Prometheus for now. Refer - gardener/gardener#948.

However, we will need to further enhance metrics to always return values and not return blank values (like mcm_cloud_api_requests_failed_total, mcm_cloud_api_requests_total, mcm_machine_deployment_failed_machines ) for all the metrics before trying to create a dashboard and raise alerts. Refer - gardener/gardener#948 (comment)

[elankath]

We need to introduce metrics for following cases:

• machine drain - especially metrics related to pod eviction/deletion, number of times drain was evicted, number of times health-check failed,
• We also need to fix the failed machines metric which repeatedly delivers machines with lastOperation as failed . MCM does not reset .status.failedMachines of MachineDeployment #456 . Also this can be confused with machine with phase as Failed so the name needs to be changed from failed_machines to something like failed_last_operation_machines .We need an alternate metric for users.
• We need metrics for provider/driver API calls made by MCM. Switch to exponential backoff while creating/deletion machines #483
• Machine Phase Metrics to accompany proposed Node Condition metrics
• Add Metrics for when VM status fine but Node is un-healthy.
• Add Metrics to determine how many times did the AWS Auto Recovery feature kicked-in for our VMs. And whether such auto-recovers occurred within our health check timeout. Related to decision for: Turn automatic-recovery off by default machine-controller-manager-provider-aws#94
• Update stale_machines_total metric name to stale_machines_removed_total , Add metric to track Unhealthy Stale machine removal count #808 (comment)
• Add documentation like that in etcd-druid and etcd-br
• Update requests_failed_total , requests_total in different mcm-provider, its currently exposed without update
• Update the new metrices introduced by PR Introduce new metrics to track API call duration and Update status.Status to capture underlying cause #842 . Make sure to also update g-extension , as it deploys the prometheus scrape config

[gardener-robot]

@elankath You have mentioned internal references in the public. Please check.