Cherry-pick #15260 to 7.x: [Metricbeat] Add lambda metricset in aws m…

…odule (#15642) * [Metricbeat] Add lambda metricset in aws module (#15260) * Add lambda metricset (cherry picked from commit 9c1e3a6)
elastic · Jan 20, 2020 · 4b50d36 · 4b50d36
1 parent 62aed3f
commit 4b50d36
Show file tree

Hide file tree

Showing 18 changed files with 889 additions and 13 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -160,7 +160,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 
 *Metricbeat*
 
-
+- Add lambda metricset in aws module. {pull}15260[15260]
 - Expand data for the `system/memory` metricset {pull}15492[15492]
 - Add azure `storage` metricset in order to retrieve metric values for storage accounts. {issue}14548[14548] {pull}15342[15342]
 - Add cost warnings for the azure module. {pull}15356[15356]

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -1738,6 +1738,12 @@ Metrics that returned from Cloudwatch API query. - name: billing
     `elb` contains the metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS ELB.
   release: ga
   fields:
+- name: lambda
+  type: group
+  description: >
+    `lambda` contains the metrics that were scraped from AWS CloudWatch which contains monitoring metrics sent by AWS Lambda.
+  release: beta
+  fields:
 - name: rds
   type: group
   description: >

diff --git a/metricbeat/docs/images/metricbeat-aws-lambda-overview.png b/metricbeat/docs/images/metricbeat-aws-lambda-overview.png
diff --git a/metricbeat/docs/modules/aws.asciidoc b/metricbeat/docs/modules/aws.asciidoc
@@ -12,7 +12,7 @@ This module periodically fetches monitoring metrics from AWS CloudWatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services.
 Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2`, `sqs`, `s3_request`, `s3_daily_storage`, `cloudwatch` and `rds`.
+All metrics are enabled by default.
 
 [float]
 == Module-specific configuration notes
@@ -34,8 +34,12 @@ image::./images/metricbeat-aws-overview.png[]
 [float]
 == Metricsets
 
-Currently, we have `ec2`, `sqs`, `s3_request`, `s3_daily_storage`, `cloudwatch`, `billing`,`ebs`, `elb`, `rds`, `sns`, `sqs` and `usage` metricset in `aws` module.
-Collecting `tags` for `ec2`, `cloudwatch`, `ebs` and `elb` metricset is supported.
+Currently, we have `billing`, `cloudwatch`, `dynamodb`, `ebs`, `ec2`, `elb`,
+`lambda`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs` and `usage`
+metricset in `aws` module.
+
+Collecting `tags` for `ec2`, `cloudwatch`, and metricset created based on
+`cloudwatch` using light module is supported.
 
 * *tags.*: Tag key value pairs from aws resources. A tag is a label that user assigns to an AWS resource.
 
@@ -113,7 +117,7 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
 | CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per namespace per collection period
 | CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
 |===
-`billing`, `ebs`, `elb`, `sns` and `usage` are the same as `cloudwatch` metricset.
+`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
 
 [float]
 === `ec2`
@@ -232,6 +236,8 @@ The following metricsets are available:
 
 * <<metricbeat-metricset-aws-elb,elb>>
 
+* <<metricbeat-metricset-aws-lambda,lambda>>
+
 * <<metricbeat-metricset-aws-rds,rds>>
 
 * <<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>
@@ -256,6 +262,8 @@ include::aws/ec2.asciidoc[]
 
 include::aws/elb.asciidoc[]
 
+include::aws/lambda.asciidoc[]
+
 include::aws/rds.asciidoc[]
 
 include::aws/s3_daily_storage.asciidoc[]

diff --git a/metricbeat/docs/modules/aws/lambda.asciidoc b/metricbeat/docs/modules/aws/lambda.asciidoc
@@ -0,0 +1,24 @@
+////
+This file is generated! See scripts/mage/docs_collector.go
+////
+
+[[metricbeat-metricset-aws-lambda]]
+=== aws lambda metricset
+
+beta[]
+
+include::../../../../x-pack/metricbeat/module/aws/lambda/_meta/docs.asciidoc[]
+
+This is a default metricset. If the host module is unconfigured, this metricset is enabled by default.
+
+==== Fields
+
+For a description of each field in the metricset, see the
+<<exported-fields-aws,exported fields>> section.
+
+Here is an example document generated by this metricset:
+
+[source,json]
+----
+include::../../../../x-pack/metricbeat/module/aws/lambda/_meta/data.json[]
+----
diff --git a/metricbeat/docs/modules_list.asciidoc b/metricbeat/docs/modules_list.asciidoc
@@ -16,12 +16,13 @@ This file is generated! See scripts/mage/docs_collector.go
 |<<metricbeat-module-appsearch,App Search>>  beta[]   |image:./images/icon-no.png[No prebuilt dashboards]    |  
 .1+| .1+|  |<<metricbeat-metricset-appsearch-stats,stats>> beta[]  
 |<<metricbeat-module-aws,aws>>     |image:./images/icon-yes.png[Prebuilt dashboards are available]    |  
-.12+| .12+|  |<<metricbeat-metricset-aws-billing,billing>> beta[]  
+.13+| .13+|  |<<metricbeat-metricset-aws-billing,billing>> beta[]  
 |<<metricbeat-metricset-aws-cloudwatch,cloudwatch>>   
 |<<metricbeat-metricset-aws-dynamodb,dynamodb>> beta[]  
 |<<metricbeat-metricset-aws-ebs,ebs>>   
 |<<metricbeat-metricset-aws-ec2,ec2>>   
 |<<metricbeat-metricset-aws-elb,elb>>   
+|<<metricbeat-metricset-aws-lambda,lambda>> beta[]  
 |<<metricbeat-metricset-aws-rds,rds>>   
 |<<metricbeat-metricset-aws-s3_daily_storage,s3_daily_storage>>   
 |<<metricbeat-metricset-aws-s3_request,s3_request>>   

diff --git a/x-pack/metricbeat/module/aws/_meta/config.yml b/x-pack/metricbeat/module/aws/_meta/config.yml
@@ -17,12 +17,14 @@
 - module: aws
   period: 5m
   metricsets:
+    - dynamodb
     - ebs
     - ec2
     - elb
+    - lambda
+    - rds
     - sns
     - sqs
-    - rds
 - module: aws
   period: 12h
   metricsets:

diff --git a/x-pack/metricbeat/module/aws/_meta/docs.asciidoc b/x-pack/metricbeat/module/aws/_meta/docs.asciidoc
@@ -4,7 +4,7 @@ This module periodically fetches monitoring metrics from AWS CloudWatch using
 https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services.
 Note: extra AWS charges on GetMetricData API requests will be generated by this module.
 
-The default metricsets are `ec2`, `sqs`, `s3_request`, `s3_daily_storage`, `cloudwatch` and `rds`.
+All metrics are enabled by default.
 
 [float]
 == Module-specific configuration notes
@@ -26,8 +26,12 @@ image::./images/metricbeat-aws-overview.png[]
 [float]
 == Metricsets
 
-Currently, we have `ec2`, `sqs`, `s3_request`, `s3_daily_storage`, `cloudwatch`, `billing`,`ebs`, `elb`, `rds`, `sns`, `sqs` and `usage` metricset in `aws` module.
-Collecting `tags` for `ec2`, `cloudwatch`, `ebs` and `elb` metricset is supported.
+Currently, we have `billing`, `cloudwatch`, `dynamodb`, `ebs`, `ec2`, `elb`,
+`lambda`, `rds`, `s3_daily_storage`, `s3_request`, `sns`, `sqs` and `usage`
+metricset in `aws` module.
+
+Collecting `tags` for `ec2`, `cloudwatch`, and metricset created based on
+`cloudwatch` using light module is supported.
 
 * *tags.*: Tag key value pairs from aws resources. A tag is a label that user assigns to an AWS resource.
 
@@ -105,7 +109,7 @@ GetMetricData max page size: 100, based on https://docs.aws.amazon.com/AmazonClo
 | CloudWatch ListMetrics | Total number of results / ListMetrics max page size | Per region per namespace per collection period
 | CloudWatch GetMetricData | Total number of results / GetMetricData max page size | Per region per namespace per collection period
 |===
-`billing`, `ebs`, `elb`, `sns` and `usage` are the same as `cloudwatch` metricset.
+`billing`, `ebs`, `elb`, `sns`, `usage` and `lambda` are the same as `cloudwatch` metricset.
 
 [float]
 === `ec2`