-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add flag to white list container labels for prometheus metrics #2113
Conversation
This adds a flag `white_listed_container_labels` which allows specifying a comma separated list of container labels that are allowed to be attached to prometheus metrics. This helps limit the label space of each data point and limits the memory used by cadvisor compared to just add all labels from all containers which is the default. This is somewhat similar to the `store_container_labels` flag but a bit more flexible because it allows white listing some common labels e.g. `io.kubernetes.pod.name`, `io.kubernetes.pod.namespace` etc. Without this white listing in place we saw the memory footprint of cadvisor explode in our production environment because users add a lot of data in pod labels, data which is not useful for metrics but may be useful for the individual applications. Signed-off-by: Mikkel Oscar Lyderik Larsen <mikkel.larsen@zalando.de>
Hi @mikkeloscar. Thanks for your PR. I'm waiting for a google or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/ok-to-test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple nits. I think this change is good overall.
Signed-off-by: Mikkel Oscar Lyderik Larsen <mikkel.larsen@zalando.de>
@dashpole Thanks for the review, I addressed your comments! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
This adds a flag
white_listed_container_labels
which allows specifyinga comma separated list of container labels that are allowed to be
attached to prometheus metrics. This helps limit the label space of
each data point and limits the memory used by cadvisor compared to just
add all labels from all containers which is the default.
This is somewhat similar to the
store_container_labels
flag but a bitmore flexible because it allows white listing some common labels e.g.
io.kubernetes.pod.name
,io.kubernetes.pod.namespace
etc.Without this white listing in place we saw the memory footprint of
cadvisor explode in our production environment because users add a lot
of data in pod labels, data which is not useful for metrics but may be
useful for the individual applications.