MM-60020: Cloudwatch exporter #798

agarciamontoro · 2024-09-12T17:35:03Z

Summary

This PR adds a new service to the metrics instance: YACE, an exporter for retrieving Cloudwatch metrics into Prometheus. I needed to make a couple of adjustments to make this work:

I added a new ClusterName tag to all resources, since YACE only discovers tagged resources. This will also help us analyze the cost of individual resources and whole clusters after we run large load-tests
I also added a missing Name tag to RDS clusters and instances
I added a couple of new panels in the default and the Elasticsearch dashboards (see screenshots below)
I converted the old dashboard_data.json file into a template file, and renamed it to default_dashboard_tmpl.json

A couple of important issues to note with Cloudwatch metrics:

These metrics are inherently delayed due to how AWS export them.
Scraping the Cloudwatch API is not free: $0.01 per 1,000 metrics requested (see the Metrics tab in this doc, the entry for GetMetricData).

Considering these two points, YACE is right now configured to get a single data point every 5 minutes, with a 5 minute delay. That is: YACE scrapes Cloudwatch at times T, T+5m, T+10m... and, respectively, get data from T-5m, T, T+5m... This is all configured with the Period, Length and Delay options, which are all set to 300 seconds. This issue comment is very helpful to understand how that works. We can tune this in the future, probably having more resolution, but I wanted to be conservative at the beginning.

Snapshots

New section and panels in default dashboard

New section and panels in Elasticsearch dashboard

New panel at the very beginning for instance health checks

Ticket Link

https://mattermost.atlassian.net/browse/MM-60020

This allows the metrics instance to access accounts on behalf of the AWS account that created it, with the permissions specified in the metrics_policy_document data block

A good resource on how to configure the delay, period and length: prometheus-community/yet-another-cloudwatch-exporter#865 (comment)

agnivade · 2024-09-16T04:58:14Z

deployment/terraform/assets/cluster.tf

@@ -198,17 +262,21 @@ EOF
 resource "aws_elasticache_cluster" "redis_server" {
  cluster_id           = "${var.cluster_name}-redis"
  engine               = "redis"
-  node_type            = "${var.redis_node_type}"
+  node_type            = var.redis_node_type


Thanks for fixing my mistakes 🙏

It was actually my editor 😂

Thank you emacs 🙏

agnivade

Great work! Left a non-blocking comment for your consideration.

agnivade · 2024-09-16T05:01:26Z

deployment/terraform/strings.go

+    - type: AWS/EC2
+      regions:
+        - us-east-1
+      period: {{.Period}}
+      length: {{.Length}}
+      delay: {{.Delay}}
+      addCloudwatchTimestamp: true
+      searchTags:
+        - key: ClusterName
+          value: {{.ClusterName}}
+      metrics:
+        - name: CPUUtilization
+          statistics: [Average]
+        - name: NetworkIn
+          statistics: [Average, Sum]
+        - name: NetworkOut
+          statistics: [Average, Sum]
+        - name: NetworkPacketsIn
+          statistics: [Sum]
+        - name: NetworkPacketsOut
+          statistics: [Sum]
+        - name: DiskReadBytes
+          statistics: [Sum]
+        - name: DiskWriteBytes
+          statistics: [Sum]
+        - name: DiskReadOps
+          statistics: [Sum]
+        - name: DiskWriteOps
+          statistics: [Sum]
+        - name: StatusCheckFailed
+          statistics: [Sum]
+        - name: StatusCheckFailed_Instance
+          statistics: [Sum]
+        - name: StatusCheckFailed_System
+          statistics: [Sum]


I don't think we need the EC2 metrics. 0/5

Yeah, I added those when I was testing the whole thing and did not remove them even though I'm not using them in any of the panels. The only interesting ones may be the StatusCheckFailed ones to monitor if an instance is down. Let me see if I can add a panel with that info and I can remove the other ones.

Set nilToZero to true as well to get metrics for all instances, since 0 is the success value.

agarciamontoro · 2024-09-16T16:01:51Z

Added a new panel plotting the health check of all instances:

….exporter

cpoile

I'm not sure what I can add, looks good. :)

streamer45 · 2024-09-25T22:02:18Z

deployment/terraform/assets/cluster.tf

+resource "aws_iam_instance_profile" "metrics_profile" {
+  name = "metrics_profile"
+  role = aws_iam_role.metrics_role.name
+}


@agarciamontoro Heads up this seems to be causing issues on my side, apparently conflicting with other deployments. Getting the following error when deploying:

info [2024-09-25 15:49:43.890 -06:00] │ Error: creating IAM Instance Profile (metrics_profile): operation error IAM: CreateInstanceProfile, https response error StatusCode: 409, RequestID: 5a5d50ea-fa2d-411b-b185-2009a4975393, EntityAlreadyExists: Instance Profile metrics_profile already exists. caller="terraform/engine.go:97"

Of course! 🤦 Thank you for catching this!

agarciamontoro requested review from cpoile and agnivade September 12, 2024 17:35

agarciamontoro added 11 commits September 12, 2024 19:35

Add ClusterName tag to all resources

85a9b46

Add missing Name tag to RDS cluster and instances

73ad547

Install YACE in the metrics instance

8dbe737

Add needed policies and roles for YACE to work

63b771b

This allows the metrics instance to access accounts on behalf of the AWS account that created it, with the permissions specified in the metrics_policy_document data block

Remove unneeded formatted strings

bbb89de

Configure and enable YACE

05d2d46

A good resource on how to configure the delay, period and length: prometheus-community/yet-another-cloudwatch-exporter#865 (comment)

Add new graphs to default and ES dashboards

6973095

Update default dashboard ID

f4650f3

Escape double curly braces in template file

32c3933

Rename dashboard_data to default_dashboard_tmpl

de31782

make assets

ffdd6f5

agarciamontoro force-pushed the MM-60020.cloudwatch.exporter branch from 0b9d31e to ffdd6f5 Compare September 12, 2024 17:36

agnivade reviewed Sep 16, 2024

View reviewed changes

agnivade approved these changes Sep 16, 2024

View reviewed changes

agarciamontoro added 4 commits September 16, 2024 17:08

Keep only StatusCheckFailed metrics for EC2

574b58e

Set nilToZero to true as well to get metrics for all instances, since 0 is the success value.

Add new panel reporting status of all instances

70b2be2

make assets

481d1ee

make assets (now with the actual changes)

fa8e9e1

agarciamontoro added 5 commits September 16, 2024 18:05

Escape curly brances in tag_Name

a54771d

And make assets again

8c154c6

Fix value mapping, metric can report values > 1

67b2fe5

make assets

927c2de

Merge remote-tracking branch 'origin/master' into MM-60020.cloudwatch…

4030cd1

….exporter

cpoile approved these changes Sep 23, 2024

View reviewed changes

agarciamontoro merged commit 3afe53d into master Sep 23, 2024
1 check passed

agarciamontoro deleted the MM-60020.cloudwatch.exporter branch September 23, 2024 13:44

streamer45 reviewed Sep 25, 2024

View reviewed changes

agarciamontoro mentioned this pull request Sep 26, 2024

Prefix metrics_profile name with ClusterName #812

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MM-60020: Cloudwatch exporter #798

MM-60020: Cloudwatch exporter #798

agarciamontoro commented Sep 12, 2024 •

edited

Loading

agnivade Sep 16, 2024

agarciamontoro Sep 16, 2024

agnivade Sep 16, 2024

agnivade left a comment

agnivade Sep 16, 2024

agarciamontoro Sep 16, 2024

agarciamontoro commented Sep 16, 2024 •

edited

Loading

cpoile left a comment

streamer45 Sep 25, 2024

agarciamontoro Sep 26, 2024

MM-60020: Cloudwatch exporter #798

MM-60020: Cloudwatch exporter #798

Conversation

agarciamontoro commented Sep 12, 2024 • edited Loading

Summary

Snapshots

New section and panels in default dashboard

New section and panels in Elasticsearch dashboard

New panel at the very beginning for instance health checks

Ticket Link

agnivade Sep 16, 2024

Choose a reason for hiding this comment

agarciamontoro Sep 16, 2024

Choose a reason for hiding this comment

agnivade Sep 16, 2024

Choose a reason for hiding this comment

agnivade left a comment

Choose a reason for hiding this comment

agnivade Sep 16, 2024

Choose a reason for hiding this comment

agarciamontoro Sep 16, 2024

Choose a reason for hiding this comment

agarciamontoro commented Sep 16, 2024 • edited Loading

cpoile left a comment

Choose a reason for hiding this comment

streamer45 Sep 25, 2024

Choose a reason for hiding this comment

agarciamontoro Sep 26, 2024

Choose a reason for hiding this comment

agarciamontoro commented Sep 12, 2024 •

edited

Loading

agarciamontoro commented Sep 16, 2024 •

edited

Loading