Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix bug related to wrong state.ID in awss3 direct s3 input #32164

Merged
merged 12 commits into from
Jul 11, 2022
Merged

fix bug related to wrong state.ID in awss3 direct s3 input #32164

merged 12 commits into from
Jul 11, 2022

Conversation

aspacca
Copy link

@aspacca aspacca commented Jun 30, 2022

Bug

What does this PR do?

Fix a bug related to using to different ID values when comparing states of s3 objects in order to skip them in the awss3 s3 direct input

Why is it important?

When an S3 object in a bucket get updated we will skip it otherwise, instead of ingesting the new content

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@aspacca aspacca self-assigned this Jun 30, 2022
@aspacca aspacca requested a review from a team as a code owner June 30, 2022 04:32
@aspacca aspacca requested review from cmacknz and fearful-symmetry and removed request for a team June 30, 2022 04:32
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 30, 2022
@aspacca aspacca added bug Team:Cloud-Monitoring Label for the Cloud Monitoring team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 30, 2022
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 30, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-07-07T03:23:29.239+0000

  • Duration: 131 min 51 sec

Test stats 🧪

Test Results
Failed 0
Passed 4282
Skipped 338
Total 4620

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@aspacca aspacca added backport-v8.3.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify backport-7.17 Automated backport to the 7.17 branch with mergify and removed backport-v8.2.0 Automated backport with mergify labels Jun 30, 2022
Andrea Spacca added 2 commits June 30, 2022 17:51
Merge remote-tracking branch 'upstream/main' into 2216-filebeat-does-…
4b58cda 
…not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly
fix race in benchmark
9c513d8
@cmacknz cmacknz requested a review from andrewkroh June 30, 2022 14:10
@cmacknz
Copy link
Member

cmacknz commented Jun 30, 2022

Adding @andrewkroh as a reviewer since I see he's done a lot of work on this input in the past.

@cmacknz cmacknz removed the request for review from fearful-symmetry June 30, 2022 14:10
@mergify
Copy link
Contributor

mergify bot commented Jul 5, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly upstream/2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly
git merge upstream/main
git push upstream 2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly

Merge branch 'main' into 2216-filebeat-does-not-read-the-file-from-s3…
6f23d09 
…-input-occasionally-leading-to-no-data-getting-generated-weekly
@aspacca
Copy link
Author

aspacca commented Jul 6, 2022

@andrewkroh good to merge?

@mergify
Copy link
Contributor

mergify bot commented Jul 6, 2022

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly upstream/2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly
git merge upstream/main
git push upstream 2216-filebeat-does-not-read-the-file-from-s3-input-occasionally-leading-to-no-data-getting-generated-weekly

Merge branch 'main' into 2216-filebeat-does-not-read-the-file-from-s3…
2be0fae 
…-input-occasionally-leading-to-no-data-getting-generated-weekly
@aspacca aspacca merged commit 7e7755c into elastic:main Jul 11, 2022
mergify bot pushed a commit that referenced this pull request Jul 11, 2022
@mergify
fix bug related to wrong state.ID in awss3 direct s3 input (#32164)
d1667af 
* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function

(cherry picked from commit 7e7755c)

# Conflicts:
#	x-pack/filebeat/input/awss3/input_benchmark_test.go
#	x-pack/filebeat/input/awss3/states_test.go
@mergify mergify bot mentioned this pull request Jul 11, 2022
Merged
mergify bot pushed a commit that referenced this pull request Jul 11, 2022
@mergify
fix bug related to wrong state.ID in awss3 direct s3 input (#32164)
b32b178 
* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function

(cherry picked from commit 7e7755c)
@mergify mergify bot mentioned this pull request Jul 11, 2022
Merged
aspacca pushed a commit that referenced this pull request Jul 11, 2022
@mergify
fix bug related to wrong state.ID in awss3 direct s3 input (#32164) (#…
e0a4cae 
…32289)

* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function

(cherry picked from commit 7e7755c)

Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
rdner pushed a commit that referenced this pull request Jul 13, 2022
@mergify
[7.17](backport #32164) fix bug related to wrong state.ID in awss3 di…
f139d33 
…rect s3 input (#32288)

* fix bug related to wrong state.ID in awss3 direct s3 input (#32164)

* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function

(cherry picked from commit 7e7755c)

# Conflicts:
#	x-pack/filebeat/input/awss3/input_benchmark_test.go
#	x-pack/filebeat/input/awss3/states_test.go

* fix conflict

Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@mergify
fix bug related to wrong state.ID in awss3 direct s3 input (elastic#3…
3b1d2b1 
…2164) (elastic#32289)

* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function

(cherry picked from commit 7e7755c)

Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@chrisberkhout
fix bug related to wrong state.ID in awss3 direct s3 input (#32164)
637fd0c 
* fix bug related to wrong state.ID in awss3 direct s3 input

* changelog

* fix everywhere

* fix everywhere

* fix race in benchmark

* fix race in benchmark

* more linting

* introduce stateID function
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewkroh andrewkroh andrewkroh approved these changes

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

@cmacknz cmacknz Awaiting requested review from cmacknz cmacknz was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@aspacca aspacca

Labels
backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.3.0 Automated backport with mergify bug Team:Cloud-Monitoring Label for the Cloud Monitoring team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@aspacca @elasticmachine @cmacknz @andrewkroh @kaiyan-sheng