Add scalability good practices doc #740
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
the
cncf-cla: yes
gmarek
force-pushed
the
throughput
branch
4 times, most recently
from
c08d6aa to
79aaa10
Compare
thockin
reviewed
Jul 1, 2017
| to profile a component is to create an ssh tunnel to the machine running it, and run `go tool pprof localhost:<your_tunnel_port>` locally | ||
|
|
||
| ## Summary | ||
| Summing it up, when writing code you should: |
There was a problem hiding this comment.
Did you mean this to be a bulleted list?
thockin
added
the
lgtm
wojtek-t
reviewed
Jul 4, 2017
| By "breaking scalability" we mean causing performance SLO violation in one of our performance tests. Performance SLOs for Kubernetes are <sup>[2](#2)</sup>: | ||
| - 99th percentile of API call latencies <= 1s | ||
| - 99th percentile of e2e Pod startup, excluding image pulling, latencies <= 5s | ||
| Tests that we run are Density and Load, we invite everyone interested in details to read the code. |
There was a problem hiding this comment.
Break the line here (see how it looks in md file).
| to profile a component is to create an ssh tunnel to the machine running it, and run `go tool pprof localhost:<your_tunnel_port>` locally | ||
|
|
||
| ## Summary | ||
| Summing it up, when writing code you should: |
|
|
||
| So for each `Secret` periodically we were GETting it's value and updating underlying variables/volumes if necessary. We have the same logic for `ConfigMaps`. Everything was great until we turned on `ServiceAccount` admission controller in our performance tests. Then everything went to hell for a very simple reason. `ServiceAccount` admission controller creates a `Secret` that it attached to every `Pod` (different one in every Namespace, but this doesn't change anything). Multiply above behavior by 150k. Given refresh period of 60s it meant that additional 2.5k QPS went to the API server, which of course blew up. | ||
|
|
||
| What we did to mitigate this issue was, in a way, reimplementing Informers using GETs instead of WATCHes. Current solution consists of a `Secret` cache shared between all `Pod`s. When a `Pod` wants to check if `Secret` changed to looks into the cache. If the `Secret` stored in the cache is too old, cache issues a GET request to API server to get current value. As `Pods` within a single `Namespace` share the `Secret` for `ServiceAccount` it means that Kubelet will need to refresh the `Secret` only once a while per `Namespace`, not per `Pod`, as it was before. This of course is a stopgap, not a final solution, which is currently (as of early May 2017) being designed as a "Bulk Watch". |
There was a problem hiding this comment.
You can link the design at the end: #443
gmarek
force-pushed
the
throughput
branch
3 times, most recently
from
3f030ba to
8724777
Compare
|
Done. @wojtek-t PTAL |
|
/lgtm |
danehans
pushed a commit
to danehans/community
that referenced
this pull request
Jul 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.