-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[new] promtail pipeline: Promtail Rate Limit stage #5048 #5051
Conversation
@Whyeasy ,rate limit stage |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for proposing this change 🙂
Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for responding so quickly. I wonder how we could add a test 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice. thanks for your patience!
Probably a flaky test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Every other stage provides one or more examples. Can you provide one for this limit stage?
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
@KMiller-Grafana We need your approval to merge this one. Thank you ! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm suggesting changes that give a bit of general understanding of what these rate limits are about. Specifically, they don't implement an ingestion rate limit within Promtail; they instead limit what Promtail pushes to Loki.
@liguozhong -- We made a few suggested improvements to the documentation. Can you take a look and merge them in if they look good to you? Then we can get this merged to main :) |
Head branch was pushed to by a user without write access
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
done,thanks for your help. |
done,thanks for your help. |
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
done,hi @KMiller-Grafana ,what else do i need to do? thanks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now. Thanks for the quick response.
…ana#5051) * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update clients/pkg/logentry/stages/limit.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update docs/sources/clients/promtail/stages/limit.md * Update docs/sources/clients/promtail/stages/limit.md * fix import package * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
* [new] promtail pipeline: Promtail Rate Limit stage #5048 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update clients/pkg/logentry/stages/limit.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/stages/limit.md * Update docs/sources/clients/promtail/stages/limit.md * fix import package * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> (cherry picked from commit 098b5a2)
* [new] promtail pipeline: Promtail Rate Limit stage #5048 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update clients/pkg/logentry/stages/limit.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * [new] promtail pipeline: Promtail Rate Limit stage #5048 #5051 * Update docs/sources/clients/promtail/stages/limit.md * Update docs/sources/clients/promtail/stages/limit.md * fix import package * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> (cherry picked from commit 098b5a2) Co-authored-by: 李国忠 <249032432@qq.com>
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-5051-to-release-2.5.x origin/release-2.5.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 098b5a2d37d51d2e9c01adac6a4abd46a80fbc95
# Push it to GitHub
git push --set-upstream origin backport-5051-to-release-2.5.x
git switch main
# Remove the local backport branch
git branch -D backport-5051-to-release-2.5.x Then, create a pull request where the |
1 similar comment
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-5051-to-release-2.5.x origin/release-2.5.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 098b5a2d37d51d2e9c01adac6a4abd46a80fbc95
# Push it to GitHub
git push --set-upstream origin backport-5051-to-release-2.5.x
git switch main
# Remove the local backport branch
git branch -D backport-5051-to-release-2.5.x Then, create a pull request where the |
The backport to
To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-5051-to-release-2.5.x origin/release-2.5.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 098b5a2d37d51d2e9c01adac6a4abd46a80fbc95
# Push it to GitHub
git push --set-upstream origin backport-5051-to-release-2.5.x
git switch main
# Remove the local backport branch
git branch -D backport-5051-to-release-2.5.x Then, create a pull request where the |
…ana#5051) (grafana#5793) * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update clients/pkg/logentry/stages/limit.go Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * [new] promtail pipeline: Promtail Rate Limit stage grafana#5048 grafana#5051 * Update docs/sources/clients/promtail/stages/limit.md * Update docs/sources/clients/promtail/stages/limit.md * fix import package * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/stages/limit.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> * Update docs/sources/clients/promtail/pipelines.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Karsten Jeschkies <k@jeschkies.xyz> Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> Co-authored-by: Cyril Tovena <cyril.tovena@gmail.com> (cherry picked from commit 098b5a2) Co-authored-by: 李国忠 <249032432@qq.com>
…ng (#7127) <!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Name your PR as `<Feature Area>: Describe your change`. a. Do not end the title with punctuation. It will be added in the changelog. b. Start with an imperative verb. Example: Fix the latency between System A and System B. c. Use sentence case, not title case. d. Use a complete phrase or sentence. The PR title will appear in a changelog, so help other people understand what your change will be. 3. Rebase your PR if it gets out of sync with main --> **What this PR does / why we need it**: The sampling stage can be directly sampled. The implementation of sampling is to use the algorithm in jaeger go client ``` pipeline_stages: - sampling: rate: 0.1 ``` or it can be used with match for precise sampling. ``` pipeline_stages: - json: expressions: app: - match: pipeline_name: "app2" selector: "{app=\"poki\"}" stages: - sampling: rate: 0.1 ``` **Which issue(s) this PR fixes**: Fixes #6654 **Special notes for your reviewer**: The promtail 'rate' stage is also used with the 'match' stage for log filtering.This design makes the code very clean. Other log agents vector have log filtering built into the sampling operator, which I think is too complicated https://vector.dev/docs/reference/configuration/transforms/sample/ ``` transforms: my_transform_id: type: sample inputs: - my-source-or-transform-id exclude: null rate: 10 ``` 'rate' stage review suggestions . #5051 ![image](https://user-images.githubusercontent.com/9583245/189461481-6ee4d835-2573-4b8e-8dec-2814620d758a.png) <!-- Note about CHANGELOG entries, if a change adds: * an important feature * fixes an issue present in a previous release, * causes a change in operation that would be useful for an operator of Loki to know then please add a CHANGELOG entry. For documentation changes, build changes, simple fixes etc please skip this step. We are attempting to curate a changelog of the most relevant and important changes to be easier to ingest by end users of Loki. Note about the upgrade guide, if this changes: * default configuration values * metric names or label names * changes existing log lines such as the metrics.go query output line * configuration parameters * anything to do with any API * any other change that would require special attention or extra steps to upgrade Please document clearly what changed AND what needs to be done in the upgrade guide. --> **Checklist** - [x] Documentation added - [x] Tests updated - [ ] Is this an important fix or new feature? Add an entry in the `CHANGELOG.md`. - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Co-authored-by: J Stickler <julie.stickler@grafana.com>
Promtail Rate Limit stage #5048