Skip to content

Conversation

kwinstonix
Copy link
Contributor

@kwinstonix kwinstonix commented Jun 26, 2020

What does this PR do?

This PR is to add a parameter aws_partition for metricbeat aws cloudwatch module, so metricbeat can support aws china and aws us-gov account.

Why is it important?

Because there are three partitions aws, aws-cn, aws-us-gov of aws regions, now metricbeat can only support aws global region, here is a hardcode.

// Get IAM account name
awsConfig.Region = "us-east-1"
svcIam := iam.New(awscommon.EnrichAWSConfigWithEndpoint(
config.AWSConfig.Endpoint, "iam", "", awsConfig))
req := svcIam.ListAccountAliasesRequest(&iam.ListAccountAliasesInput{})

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. add aws_partition param in aws module config
metricbeat.modules:
  - module: aws
    aws_partition: aws-cn
    period: 300s
    access_key_id: 
    secret_access_key:
  1. run metricbeat , open debug log, then you can see it can retrive metrics from correct regions

Related issues

Use cases

  • no side affect
  • user can use metricbeat to get metrics from aws china and aws us-gov account except aws global account

Screenshots

aws global account, set aws_partition: aws in aws module config or remove the param

image

aws china account, set aws_partition: aws-cn in aws module config

image

@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2020
@kwinstonix kwinstonix changed the title add params aws_partition to support other aws region: aws-cn, aws-u… [MetricBeat] add params aws_partition to support other aws region: aws-cn, aws-u… Jun 26, 2020
@kwinstonix kwinstonix changed the title [MetricBeat] add params aws_partition to support other aws region: aws-cn, aws-u… [MetricBeat] add params aws_partition to support: aws-cn, aws-us-gov region Jun 26, 2020
@kwinstonix kwinstonix changed the title [MetricBeat] add params aws_partition to support: aws-cn, aws-us-gov region [MetricBeat] add param aws_partition to support: aws-cn, aws-us-gov region Jun 26, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 26, 2020

❕ Build Aborted

The PR is not allowed to run in the CI yet

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19423 updated]

  • Reason: The PR is not allowed to run in the CI yet

  • Start Time: 2020-06-26T11:17:48.971+0000

  • Duration: 3 min 50 sec

  • Commit: 5b6dc38

Steps errors

Expand to view the steps failures

  • Name: Error signal
    • Description: githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permission

    • Duration: 0 min 0 sec

    • Start Time: 2020-06-26T11:20:35.163+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-26T11:20:36.739Z] Stage "Elastic Agent x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.740Z] Stage "Elastic Agent x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.741Z] Stage "Elastic Agent Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.742Z] Stage "Filebeat oss" skipped due to earlier failure(s)
[2020-06-26T11:20:36.742Z] Stage "Filebeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.743Z] Stage "Filebeat Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.744Z] Stage "Filebeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.744Z] Stage "Filebeat Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.745Z] Stage "Filebeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.746Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.746Z] Stage "Auditbeat oss Linux" skipped due to earlier failure(s)
[2020-06-26T11:20:36.747Z] Stage "Auditbeat crosscompile" skipped due to earlier failure(s)
[2020-06-26T11:20:36.748Z] Stage "Auditbeat oss Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.748Z] Stage "Auditbeat oss Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.749Z] Stage "Auditbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.750Z] Stage "Auditbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.751Z] Stage "Auditbeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.751Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.752Z] Stage "Libbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.753Z] Stage "Metricbeat OSS Unit tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.753Z] Stage "Metricbeat OSS Integration tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.754Z] Stage "Metricbeat Python integration tests" skipped due to earlier failure(s)
[2020-06-26T11:20:36.755Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.755Z] Stage "Metricbeat crosscompile" skipped due to earlier failure(s)
[2020-06-26T11:20:36.756Z] Stage "Metricbeat Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.757Z] Stage "Metricbeat x-pack Mac OS X" skipped due to earlier failure(s)
[2020-06-26T11:20:36.758Z] Stage "Metricbeat Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.758Z] Stage "Metricbeat x-pack Windows" skipped due to earlier failure(s)
[2020-06-26T11:20:36.759Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.759Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.760Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.760Z] Stage "Winlogbeat Windows x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.761Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.762Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.762Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:36.763Z] Stage "Kubernetes" skipped due to earlier failure(s)
[2020-06-26T11:20:36.835Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.836Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.837Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:36.838Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.839Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.839Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.840Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.841Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:36.842Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.322Z] Failed in branch Elastic Agent x-pack
[2020-06-26T11:20:37.322Z] Failed in branch Elastic Agent x-pack Windows
[2020-06-26T11:20:37.323Z] Failed in branch Elastic Agent Mac OS X
[2020-06-26T11:20:37.324Z] Failed in branch Filebeat oss
[2020-06-26T11:20:37.324Z] Failed in branch Filebeat x-pack
[2020-06-26T11:20:37.325Z] Failed in branch Filebeat Mac OS X
[2020-06-26T11:20:37.325Z] Failed in branch Filebeat x-pack Mac OS X
[2020-06-26T11:20:37.326Z] Failed in branch Filebeat Windows
[2020-06-26T11:20:37.327Z] Failed in branch Filebeat x-pack Windows
[2020-06-26T11:20:37.327Z] Failed in branch Auditbeat oss Linux
[2020-06-26T11:20:37.328Z] Failed in branch Auditbeat crosscompile
[2020-06-26T11:20:37.328Z] Failed in branch Auditbeat oss Mac OS X
[2020-06-26T11:20:37.329Z] Failed in branch Auditbeat oss Windows
[2020-06-26T11:20:37.329Z] Failed in branch Auditbeat x-pack
[2020-06-26T11:20:37.330Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-06-26T11:20:37.330Z] Failed in branch Auditbeat x-pack Windows
[2020-06-26T11:20:37.331Z] Failed in branch Libbeat x-pack
[2020-06-26T11:20:37.331Z] Failed in branch Metricbeat OSS Unit tests
[2020-06-26T11:20:37.332Z] Failed in branch Metricbeat OSS Integration tests
[2020-06-26T11:20:37.332Z] Failed in branch Metricbeat Python integration tests
[2020-06-26T11:20:37.333Z] Failed in branch Metricbeat crosscompile
[2020-06-26T11:20:37.334Z] Failed in branch Metricbeat Mac OS X
[2020-06-26T11:20:37.334Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-06-26T11:20:37.335Z] Failed in branch Metricbeat Windows
[2020-06-26T11:20:37.335Z] Failed in branch Metricbeat x-pack Windows
[2020-06-26T11:20:37.336Z] Failed in branch Winlogbeat Windows x-pack
[2020-06-26T11:20:37.336Z] Failed in branch Kubernetes
[2020-06-26T11:20:37.572Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.573Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.574Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-26T11:20:37.575Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.576Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.577Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.630Z] Failed in branch Packetbeat
[2020-06-26T11:20:37.631Z] Failed in branch dockerlogbeat
[2020-06-26T11:20:37.632Z] Failed in branch Journalbeat
[2020-06-26T11:20:37.816Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.817Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.818Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-26T11:20:37.819Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:37.855Z] Failed in branch Metricbeat x-pack
[2020-06-26T11:20:37.856Z] Failed in branch Winlogbeat
[2020-06-26T11:20:38.017Z] Failed in branch Heartbeat
[2020-06-26T11:20:38.018Z] Failed in branch Libbeat
[2020-06-26T11:20:38.018Z] Failed in branch Functionbeat
[2020-06-26T11:20:38.019Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-26T11:20:38.091Z] Failed in branch Generators
[2020-06-26T11:20:38.386Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19423
[2020-06-26T11:20:38.484Z] [INFO] getVaultSecret: Getting secrets
[2020-06-26T11:20:38.542Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-26T11:20:39.275Z] + chmod 755 generate-build-data.sh
[2020-06-26T11:20:39.275Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4 ABORTED 170045
[2020-06-26T11:20:39.275Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/steps/?limit=10000 -o steps-info.json
[2020-06-26T11:20:39.826Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/tests/?status=FAILED -o tests-errors.json
[2020-06-26T11:20:39.826Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19423/runs/4/log/ -o pipeline-log.txt

@kwinstonix kwinstonix changed the title [MetricBeat] add param aws_partition to support: aws-cn, aws-us-gov region [MetricBeat] add param aws_partition to support aws-cn, aws-us-gov regions Jun 26, 2020
@kwinstonix kwinstonix marked this pull request as ready for review June 26, 2020 07:55
@kwinstonix kwinstonix force-pushed the add_aws_partition_param branch from 55b3d1b to 5b6dc38 Compare June 26, 2020 11:17
@kaiyan-sheng kaiyan-sheng self-assigned this Jun 26, 2020
@kaiyan-sheng kaiyan-sheng added the Team:Platforms Label for the Integrations - Platforms team label Jun 26, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2020
@kaiyan-sheng
Copy link
Contributor

jenkins, test this please

@kaiyan-sheng
Copy link
Contributor

Thanks for working on both changes!! Sorry for the delay, I will review them today!!

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and it works. Thank you for the contribution! I will merge this PR and get it backported.

@kaiyan-sheng
Copy link
Contributor

run tests please

@kaiyan-sheng kaiyan-sheng merged commit 1ea8c84 into elastic:master Jun 30, 2020
kaiyan-sheng added a commit that referenced this pull request Jun 30, 2020
kaiyan-sheng added a commit that referenced this pull request Jul 1, 2020
kaiyan-sheng added a commit that referenced this pull request Jul 1, 2020
…support aws-cn, aws-us-gov regions (#19519)

* [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov

(cherry picked from commit 1ea8c84)
Co-authored-by: martin <kwinstonix@users.noreply.github.com>
v1v added a commit to v1v/beats that referenced this pull request Jul 2, 2020
…ne-beats

* upstream/master: (105 commits)
  ci: enable packaging job (elastic#19536)
  ci: disable upstream trigger on PRs for the packaging job (elastic#19490)
  Implement memlog on-disk handling (elastic#19408)
  fix go.mod for PR elastic#19423 (elastic#19521)
  [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (elastic#19423)
  Input v2 stateless manager (elastic#19406)
  Input v2 compatibility layer (elastic#19401)
  [Elastic Agent] Fix artifact downloading to allow endpoint-security to be downloaded (elastic#19503)
  fix: ignore target changes on scans (elastic#19510)
  Add more helpers to pipeline/testing package (elastic#19405)
  Report dependencies in CSV format (elastic#19506)
  [Filebeat] Fix reference leak in TCP and Unix socket inputs (elastic#19459)
  Cursor input skeleton (elastic#19378)
  Add changelog. (elastic#19495)
  [DOC] Typo in Kerberos (elastic#19265)
  Remove accidentally commited unused NOTICE template (elastic#19485)
  [Elastic Agent] Support the install, control, and uninstall of Endpoint (elastic#19248)
  [Filebeat][httpjson] Add split_events_by config setting (elastic#19246)
  ci: disabling packaging job until we fix it (elastic#19481)
  Fix golang.org/x/tools to release1.13 (elastic#19478)
  ...
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
…regions (elastic#19423)

* add params `aws_partition` to support other aws region: aws-cn, aws-us-gov
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Platforms Label for the Integrations - Platforms team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Metricbeat] Unable to get account id in China regions
4 participants