chore: increase max idle connections per host for kinesis #5652
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Waiting for go-kit release. Once that is done, will include latest version of it in go.mod to resolve these errors. |
atzoum
reviewed
Mar 27, 2025
vinayteki95
force-pushed
the
chore.kinesis-tls-overhead
branch
from
4e62408 to
65ec907
Compare
atzoum
reviewed
Mar 28, 2025
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5652 +/- ##
==========================================
- Coverage 76.99% 76.88% -0.11%
==========================================
Files 476 476
Lines 65396 65398 +2
==========================================
- Hits 50350 50283 -67
- Misses 12297 12355 +58
- Partials 2749 2760 +11 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
atzoum
changed the title
atzoum
approved these changes
Apr 1, 2025
mihir20
approved these changes
Apr 1, 2025
This was referenced Apr 1, 2025
Merged
atzoum
pushed a commit
that referenced
this pull request
Apr 1, 2025
🤖 I have created a release *beep* *boop* --- ## [1.46.0-rc.2](v1.45.0...v1.46.0-rc.2) (2025-04-01) ### Features * introduce workers per partition in processor ([#5607](#5607)) ([46d61b0](46d61b0)) * move async batch router destinations to use OAuth v2 flow ([#5574](#5574)) ([3e35b23](3e35b23)) * option for disabling view creation for bigquery ([#5630](#5630)) ([c804547](c804547)) * **processor:** break down transformations step ([#5639](#5639)) ([379fcbd](379fcbd)) * **processor:** count pending events without blocking ([#5605](#5605)) ([a41c63d](a41c63d)) ### Bug Fixes * compilation error in events_test.go ([#5671](#5671)) ([e3ead37](e3ead37)) * increased archival table count alert firing after starting using dslimit ([#5649](#5649)) ([ff799d4](ff799d4)) * remove the noisy combination for version deprecation detection ([#5629](#5629)) ([4516a40](4516a40)) * sonnet panic while unmarshalling float64 types ([#5616](#5616)) ([c1236e4](c1236e4)) * warehouse transformations for data_warehouse json paths ([#5653](#5653)) ([2bbe140](2bbe140)) * warehouse transformations for mandatory fields ([#5658](#5658)) ([5019422](5019422)) * warehouse transformations for tracking plans ([#5662](#5662)) ([3692063](3692063)) ### Miscellaneous * add limiter to pretransform ([#5622](#5622)) ([57ba242](57ba242)) * badger configuration tuning ([#5634](#5634)) ([1936b98](1936b98)) * bump sqlconnect-go to 1.18.1 ([#5635](#5635)) ([f4d78bf](f4d78bf)) * dedup service improvements ([#5602](#5602)) ([2e7497e](2e7497e)) * **deps:** bump docker/login-action from 3.3.0 to 3.4.0 ([#5604](#5604)) ([7e5cea3](7e5cea3)) * **deps:** bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in the go_modules group ([#5643](#5643)) ([4510413](4510413)) * **deps:** bump golangci/golangci-lint-action from 6 to 7 ([#5641](#5641)) ([1110b6a](1110b6a)) * **deps:** bump the go-deps group across 1 directory with 5 updates ([#5633](#5633)) ([a5a8978](a5a8978)) * **deps:** bump the go-deps group across 1 directory with 5 updates ([#5642](#5642)) ([89070bd](89070bd)) * increase max idle connections per host for kinesis ([#5652](#5652)) ([6fd8e7c](6fd8e7c)) * migrate sample event column to text for reporting ([#5503](#5503)) ([7d6cbf9](7d6cbf9)) * optimise schema generation function ([#5597](#5597)) ([f1818d0](f1818d0)) * recover from badgerdb panic ([#5678](#5678)) ([9a47bbd](9a47bbd)) * remove transformations v2 flag ([#5650](#5650)) ([3182f9a](3182f9a)) * sync release v1.45.0 to main branch ([#5617](#5617)) ([3669407](3669407)) * use rss for calculating used memory in adaptive payload limiter ([#5656](#5656)) ([63ff163](63ff163)) * use sonnet as the default json library ([#5657](#5657)) ([4c6e5e0](4c6e5e0)) * version deprecation detection avoid regex ([#5625](#5625)) ([0d0e7dd](0d0e7dd)) * version deprecation detection logic ([#5644](#5644)) ([345162a](345162a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Kinesis is causing high CPU usage due to high number of TLS handshakes happening. Solution is to increase number of connections that can stay idle leaving room for not killing the connections once done which will reduce TLS connections.
To begin with we are increasing the number of connections from 2 to 64.
Linear Ticket
Resolves INT-3397
https://linear.app/rudderstack/issue/INT-3397/evaluate-workarounds-for-high-tls-connections-in-expressvpn-due-to
Security