Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[8.x](backport #40006) OOTB support AWS Eventbridge #40814

Open
wants to merge 2 commits into
base: 8.x
Choose a base branch
from
Open

[8.x](backport #40006) OOTB support AWS Eventbridge #40814

wants to merge 2 commits into from
+320 −74

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Sep 13, 2024

Proposed commit message

Adding OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

The logic is triggered as a fall-through and is best effort.

Author's Checklist

  • Check for idiomatic golang constructs not being used, with golang not being my primary lang there's a good chance for non-idiomatic constructs to sneak in.

How to test this PR locally

Since it relies on an AWS setup, testing locally beyond units tests is not possible.
See /beats/x-pack/filebeat/input/awss3/_meta/terraform/README.md Don't have localstack so not truly 'local' but can run against aws.

Related issues

As it was discovered as part of support ticket there should be an ER linked.

Use cases

AWS has added a new way to notify on S3 changes, next to the existing plain SQS and SNS->SQS, AWS now also support using AWS EventBridge -> SQS

This PR achieves two goals:

  • no longer need a custom script (which is the current workaround
  • Improve user experience by allowing it to 'work out of the box'
This is an automatic backport of pull request #40006 done by [Mergify](https://mergify.com).

@mjmbischoff @mergify
OOTB support AWS Eventbridge (#40006)
13a7758 
Adding code to OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html

(cherry picked from commit c37159e)
@mergify mergify bot requested a review from a team as a code owner September 13, 2024 14:30
@mergify mergify bot added the backport label Sep 13, 2024
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 13, 2024
@botelastic
Copy link

botelastic bot commented Sep 13, 2024

This pull request doesn't have a Team:<team> label.

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 16, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

@mjmbischoff
Copy link
Contributor

@elastic/obs-cloud-monitoring Can you have a look at this?

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 23, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

kaiyan-sheng
kaiyan-sheng reviewed Sep 23, 2024
View reviewed changes
CHANGELOG.next.asciidoc
@@ -280,7 +280,8 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Allow elision of set and append failure logging. {issue}34544[34544] {pull}39929[39929]
- Add ability to remove request trace logs from CEL input. {pull}39969[39969]
- Add ability to remove request trace logs from HTTPJSON input. {pull}40003[40003]
- Update CEL mito extensions to v1.13.0. {pull}40035[40035]
- Added out of the box support for Amazon EventBridge notifications over SQS to S3 input {pull}40006[40006]
- Update CEL mito extensions to v1.13.0 {pull}40035[40035]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this entry needs to be removed?

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Sep 30, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

2 similar comments
@mergify Mergify
Copy link
Contributor Author

mergify bot commented Oct 7, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Oct 14, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Oct 21, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.x/pr-40006 upstream/mergify/bp/8.x/pr-40006
git merge upstream/8.x
git push upstream mergify/bp/8.x/pr-40006

@mergify Mergify
Copy link
Contributor Author

mergify bot commented Oct 21, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

1 similar comment
@mergify Mergify
Copy link
Contributor Author

mergify bot commented Oct 28, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

@mjmbischoff mjmbischoff requested review from a team as code owners October 31, 2024 15:09
@mjmbischoff mjmbischoff requested review from belimawr, VihasMakwana, MichaelKatsoulis and tetianakravchenko and removed request for a team October 31, 2024 15:09
@mjmbischoff
Merge branch 'refs/heads/main' into ootb-aws-eventbridge
949135b 
# Conflicts:
#	go.mod
#	x-pack/filebeat/input/awss3/sqs_test.go
@mergify Mergify
Copy link
Contributor Author

mergify bot commented Nov 4, 2024

This pull request has not been merged yet. Could you please review and merge it @mjmbischoff? 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaiyan-sheng kaiyan-sheng kaiyan-sheng left review comments

@belimawr belimawr Awaiting requested review from belimawr belimawr was automatically assigned from elastic/elastic-agent-data-plane

@VihasMakwana VihasMakwana Awaiting requested review from VihasMakwana VihasMakwana was automatically assigned from elastic/elastic-agent-data-plane

@MichaelKatsoulis MichaelKatsoulis Awaiting requested review from MichaelKatsoulis MichaelKatsoulis was automatically assigned from elastic/obs-cloudnative-monitoring

@tetianakravchenko tetianakravchenko Awaiting requested review from tetianakravchenko tetianakravchenko was automatically assigned from elastic/obs-cloudnative-monitoring

At least 1 approving review is required to merge this pull request.

Assignees

@mjmbischoff mjmbischoff

Labels
backport needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mjmbischoff @kaiyan-sheng