Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick to 7.x: Rate limit processor (#22883) #23095

Merged
merged 1 commit into from
Dec 14, 2020
Merged

Cherry-pick to 7.x: Rate limit processor (#22883) #23095

merged 1 commit into from
Dec 14, 2020

Conversation

ycombinator
Copy link
Contributor

Backports the following commits to 7.x:

* Implement basic scaffolding for rate_limit processor

* Fixing import cycle

* Adding skeleton for token bucket algo

* Set default algorithm in default config

* Using an algo constructor

* Implement token bucket rate limiting algorithm

* Resolving some TODOs

* Adding license header

* Removing old TODO comment

* Adding tests

* Reverting to previous logic

* Adding CHANGELOG entry

* Adding TODOs for more tests

* Fixing comment

* Fixing error messages

* Fixing comment

* Fleshing out godoc comments

* Fixing logger location

* Fixing up docs a bit

* Adding test for "fields" config

* WIP: adding test for burst multiplier

* Return pointer to bucket from getBucket

* Keep pointers to buckets in map to avoid map reassignment

* Fix test

* Fix logic as we cannot allow withdrawal of fractional (<1) tokens

* Move burst multiplier default to token_bucket algo

* Implementing GC

* Making the factory take the algo config and return the algo

* Reduce nesting level

* Use mitchellh/hashstructure

* Using sync.Map

* Removing algorithm and algorithm options from documentation

* Mocking clock

* Adding license headers

* Using atomic.Uints for metrics counters

* Fixing logic

* Use github.com/jonboulle/clockwork

* Running make update

* Add mutex to prevent only one GC thread from running at any time

* Adding logging

* Remove NumBuckets GC threshold

* Use non-blocking mutex

* Perform actual GC in own goroutine

* Running mage fmt

* Fixing processor name

* Importing rate limit processor

* Initialize mutex

* Do not register as a JS processor

* Remove unused field

* Mark processor as beta

* Renaming package

* Flatenning package hierarchy

* Remove SetClock from algorithm interface
# Conflicts:
#	go.mod
#	go.sum
@ycombinator ycombinator self-assigned this Dec 12, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 12, 2020
@ycombinator ycombinator requested review from a team and jsoriano December 12, 2020 19:51
@ycombinator ycombinator added the Team:Platforms Label for the Integrations - Platforms team label Dec 12, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 12, 2020
@elasticmachine
Copy link
Collaborator

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #23095 opened

  • Start Time: 2020-12-12T19:50:43.867+0000

  • Duration: 95 min 17 sec

Test stats 🧪

Test Results
Failed 0
Passed 17385
Skipped 1404
Total 18789

Steps errors 2

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 16 sec . View more details on here

Log output

Expand to view the last 100 lines of log output

[2020-12-12T20:42:55.895Z] + GVM_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/gvm
[2020-12-12T20:42:55.895Z] + command -v go
[2020-12-12T20:42:55.895Z] /var/lib/jenkins/workspace/Beats_beats_PR-23095/.gvm/versions/go1.14.12.linux.amd64/bin/go
[2020-12-12T20:42:55.895Z] + echo 'Found Go. Checking version..'
[2020-12-12T20:42:55.895Z] Found Go. Checking version..
[2020-12-12T20:42:55.895Z] ++ go version
[2020-12-12T20:42:55.895Z] ++ awk '{print $3}'
[2020-12-12T20:42:55.895Z] ++ sed s/go//
[2020-12-12T20:42:55.896Z] + FOUND_GO_VERSION=1.14.12
[2020-12-12T20:42:55.896Z] + '[' 1.14.12 == 1.14.12 ']'
[2020-12-12T20:42:55.896Z] + echo 'Versions match. No need to install Go. Exiting.'
[2020-12-12T20:42:55.896Z] Versions match. No need to install Go. Exiting.
[2020-12-12T20:42:55.896Z] + exit 0
[2020-12-12T20:42:55.896Z] + .ci/scripts/install-docker-compose.sh
[2020-12-12T20:42:55.896Z] + MSG='parameter missing.'
[2020-12-12T20:42:55.896Z] + DOCKER_COMPOSE_VERSION=1.21.0
[2020-12-12T20:42:55.896Z] + HOME=/var/lib/jenkins/workspace/Beats_beats_PR-23095
[2020-12-12T20:42:55.896Z] + DC_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/docker-compose
[2020-12-12T20:42:55.896Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin
[2020-12-12T20:42:55.896Z] ++ uname -s
[2020-12-12T20:42:55.896Z] ++ uname -m
[2020-12-12T20:42:55.896Z] + curl -sSLo /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/docker-compose https://github.com/docker/compose/releases/download/1.21.0/docker-compose-Linux-x86_64
[2020-12-12T20:42:56.465Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/docker-compose
[2020-12-12T20:42:56.465Z] + .ci/scripts/install-terraform.sh
[2020-12-12T20:42:56.465Z] + MSG='parameter missing.'
[2020-12-12T20:42:56.465Z] + TERRAFORM_VERSION=0.12.24
[2020-12-12T20:42:56.465Z] + HOME=/var/lib/jenkins/workspace/Beats_beats_PR-23095
[2020-12-12T20:42:56.465Z] + TERRAFORM_CMD=/var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform
[2020-12-12T20:42:56.465Z] ++ uname -s
[2020-12-12T20:42:56.465Z] ++ tr '[:upper:]' '[:lower:]'
[2020-12-12T20:42:56.465Z] + OS=linux
[2020-12-12T20:42:56.465Z] + mkdir -p /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin
[2020-12-12T20:42:56.465Z] + curl -sSLo - https://releases.hashicorp.com/terraform/0.12.24/terraform_0.12.24_linux_amd64.zip
[2020-12-12T20:42:57.407Z] ++ dirname /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform
[2020-12-12T20:42:57.407Z] + unzip -o /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform.zip -d /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin
[2020-12-12T20:42:57.407Z] Archive:  /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform.zip
[2020-12-12T20:42:57.666Z]   inflating: /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform  
[2020-12-12T20:42:57.666Z] + rm /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform.zip
[2020-12-12T20:42:57.666Z] + chmod +x /var/lib/jenkins/workspace/Beats_beats_PR-23095/bin/terraform
[2020-12-12T20:42:57.666Z] + make mage
[2020-12-12T20:42:57.666Z] Installing mage v1.10.0.
[2020-12-12T20:42:58.604Z] go: downloading github.com/magefile/mage v1.10.0
[2020-12-12T20:42:59.542Z] /var/lib/jenkins/workspace/Beats_beats_PR-23095/.magefile cleaned
[2020-12-12T20:42:59.906Z] + git config --get user.email
[2020-12-12T20:42:59.906Z] + [ -z  ]
[2020-12-12T20:42:59.906Z] + git config --global user.email beatsmachine@users.noreply.github.com
[2020-12-12T20:42:59.906Z] + git config --global user.name beatsmachine
[2020-12-12T20:43:00.223Z] + go mod download
[2020-12-12T20:43:15.579Z] + .ci/scripts/terraform-cleanup.sh x-pack/metricbeat
[2020-12-12T20:43:15.579Z] + DIRECTORY=x-pack/metricbeat
[2020-12-12T20:43:15.579Z] + FAILED=0
[2020-12-12T20:43:15.580Z] ++ find x-pack/metricbeat -name terraform.tfstate
[2020-12-12T20:43:15.580Z] + for tfstate in $(find $DIRECTORY -name terraform.tfstate)
[2020-12-12T20:43:15.580Z] ++ dirname x-pack/metricbeat/module/aws/terraform.tfstate
[2020-12-12T20:43:15.580Z] + cd x-pack/metricbeat/module/aws
[2020-12-12T20:43:15.580Z] + terraform destroy -auto-approve
[2020-12-12T20:43:19.776Z] 
[2020-12-12T20:43:19.777Z] Destroy complete! Resources: 0 destroyed.
[2020-12-12T20:43:19.777Z] + cd -
[2020-12-12T20:43:19.777Z] /var/lib/jenkins/workspace/Beats_beats_PR-23095/src/github.com/elastic/beats/src/github.com/elastic/beats
[2020-12-12T20:43:19.777Z] + exit 0
[2020-12-12T20:43:20.189Z] Client: Docker Engine - Community
[2020-12-12T20:43:20.189Z]  Version:           19.03.14
[2020-12-12T20:43:20.189Z]  API version:       1.40
[2020-12-12T20:43:20.189Z]  Go version:        go1.13.15
[2020-12-12T20:43:20.189Z]  Git commit:        5eb3275d40
[2020-12-12T20:43:20.189Z]  Built:             Tue Dec  1 19:20:17 2020
[2020-12-12T20:43:20.189Z]  OS/Arch:           linux/amd64
[2020-12-12T20:43:20.189Z]  Experimental:      false
[2020-12-12T20:43:20.189Z] 
[2020-12-12T20:43:20.189Z] Server: Docker Engine - Community
[2020-12-12T20:43:20.189Z]  Engine:
[2020-12-12T20:43:20.189Z]   Version:          19.03.14
[2020-12-12T20:43:20.189Z]   API version:      1.40 (minimum version 1.12)
[2020-12-12T20:43:20.189Z]   Go version:       go1.13.15
[2020-12-12T20:43:20.189Z]   Git commit:       5eb3275d40
[2020-12-12T20:43:20.189Z]   Built:            Tue Dec  1 19:18:45 2020
[2020-12-12T20:43:20.189Z]   OS/Arch:          linux/amd64
[2020-12-12T20:43:20.189Z]   Experimental:     false
[2020-12-12T20:43:20.189Z]  containerd:
[2020-12-12T20:43:20.189Z]   Version:          1.3.9
[2020-12-12T20:43:20.189Z]   GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
[2020-12-12T20:43:20.189Z]  runc:
[2020-12-12T20:43:20.189Z]   Version:          1.0.0-rc10
[2020-12-12T20:43:20.189Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-12-12T20:43:20.189Z]  docker-init:
[2020-12-12T20:43:20.190Z]   Version:          0.18.0
[2020-12-12T20:43:20.190Z]   GitCommit:        fec3683
[2020-12-12T20:43:51.273Z] Scheduling project: Beats » Beats Packaging » PR-23095
[2020-12-12T20:43:52.997Z] Starting building: Beats » Beats Packaging » PR-23095 #2
[2020-12-12T21:25:57.822Z] [INFO] For detailed information see: https://beats-ci.elastic.co/job/Beats/job/packaging/job/PR-23095/2/display/redirect
[2020-12-12T21:25:59.030Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-23095/src/github.com/elastic/beats
[2020-12-12T21:25:59.877Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-23095
[2020-12-12T21:25:59.994Z] [INFO] getVaultSecret: Getting secrets
[2020-12-12T21:26:00.082Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-12-12T21:26:00.963Z] + chmod 755 generate-build-data.sh
[2020-12-12T21:26:00.963Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23095/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23095/runs/1 FAILURE 5716820
[2020-12-12T21:26:00.963Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23095/runs/1/steps/?limit=10000 -o steps-info.json
[2020-12-12T21:26:05.086Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23095/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-12-12T21:26:05.337Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-23095/runs/1/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 17385
Skipped 1404
Total 18789

@ycombinator
Copy link
Contributor Author

Similar to the original PR for this backport, #22883, this PR's CI build is failing on an unrelated packaging job. Merging...

@ycombinator ycombinator merged commit 73ec647 into elastic:7.x Dec 14, 2020
@ycombinator ycombinator deleted the backport/7.x/pr-22883 branch December 14, 2020 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants