Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support in aws-s3 input for s3 notification from SNS to SQS #28800

Merged
merged 6 commits into from
Nov 9, 2021
Merged

Add support in aws-s3 input for s3 notification from SNS to SQS #28800

merged 6 commits into from
Nov 9, 2021

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Nov 3, 2021 •
edited
Loading

What does this PR do?

This PR is to add support for S3 notification from SNS to SQS for aws-s3 input. When parsing the SQS message body, if there is a topicArn field that exists, then it means the notification message is from SNS to SQS.

Quoting from @christianherweg0807 from #25492:
Screen Shot 2021-11-03 at 4 06 12 PM

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

arn:aws:sqs:us-east-1:428152502467:test-fb-ks is setup with SNS to SQS for testing.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 3, 2021
@mergify
Copy link
Contributor

mergify bot commented Nov 3, 2021

This pull request does not have a backport label. Could you fix it @kaiyan-sheng? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Nov 3, 2021
@kaiyan-sheng kaiyan-sheng self-assigned this Nov 3, 2021
@kaiyan-sheng kaiyan-sheng added the Team:Integrations Label for the Integrations team label Nov 3, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 3, 2021
@kaiyan-sheng kaiyan-sheng added backport-v8.0.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Nov 3, 2021
@kaiyan-sheng kaiyan-sheng mentioned this pull request Nov 3, 2021
Closed
@kaiyan-sheng kaiyan-sheng requested a review from aspacca November 3, 2021 22:13
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 3, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-08T23:45:05.871+0000

  • Duration: 100 min 46 sec

  • Commit: 91e894e

Test stats 🧪

Test Results
Failed 0
Passed 7919
Skipped 1201
Total 9120

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@aspacca
Copy link

aspacca commented Nov 4, 2021

👍

could you add an integration test as well?

@kaiyan-sheng
Copy link
Contributor Author

@aspacca integration test added 🙂

aspacca
aspacca reviewed Nov 8, 2021
View reviewed changes
x-pack/filebeat/input/awss3/input_integration_test.go
@@ -440,8 +442,74 @@ func TestPaginatorListPrefix(t *testing.T) {
}

func TestGetProviderFromDomain(t *testing.T) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realised this should not be in input_integration_test.go :)

@kaiyan-sheng kaiyan-sheng merged commit 2fe1842 into elastic:master Nov 9, 2021
mergify bot pushed a commit that referenced this pull request Nov 9, 2021
@kaiyan-sheng
Add support in aws-s3 input for s3 notification from SNS to SQS (#28800)
765b345 
* Add support in aws-s3 input for s3 notification from SNS to SQS

* add integration test for sns

* fix integration test

* add changelog

* add documentation

(cherry picked from commit 2fe1842)
@mergify mergify bot mentioned this pull request Nov 9, 2021
Merged
@kaiyan-sheng kaiyan-sheng deleted the s3_sns branch November 9, 2021 14:07
@andrewkroh andrewkroh added the backport-v7.16.0 Automated backport with mergify label Nov 10, 2021
@mergify mergify bot mentioned this pull request Nov 10, 2021
Merged
andrewkroh pushed a commit that referenced this pull request Nov 10, 2021
@kaiyan-sheng @andrewkroh
Add support in aws-s3 input for s3 notification from SNS to SQS (#28800)
8f50643 
* Add support in aws-s3 input for s3 notification from SNS to SQS

* add integration test for sns

* fix integration test

* add changelog

* add documentation
kaiyan-sheng added a commit that referenced this pull request Nov 10, 2021
@mergify @kaiyan-sheng
Add support in aws-s3 input for s3 notification from SNS to SQS (#28800…
a4b233f 
…) (#28874)

(cherry picked from commit 2fe1842)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
kaiyan-sheng added a commit that referenced this pull request Nov 10, 2021
@mergify @kaiyan-sheng
Add support in aws-s3 input for s3 notification from SNS to SQS (#28800…
1472c0a 
…) (#28910)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aspacca aspacca aspacca approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify Team:Integrations Label for the Integrations team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Filebeat] Support SQS notifications through SNS for S3 input
4 participants
@kaiyan-sheng @elasticmachine @aspacca @andrewkroh