Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dannykopping/remove cache stats #14

Draft
wants to merge 225 commits into
base: master
Choose a base branch
from

Conversation

dannykopping
Copy link
Owner

MasslessParticle and others added 30 commits April 20, 2022 14:14
- milli -> nano conversion in the querier
- add deletes to store requests from the ingester
* Add some comments

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Use deletes in tail

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Change error message

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
* fix(gateway): make DNS resolver configurable

Signed-off-by: Ivan Rizzante <i.rizzante@gmail.com>

* fix: lint
This fixes a bug in the index gateway when querying values for a label.
Since the gRPC handler for LabelValuesForMetricName in the index gateway
allows empty matchers in the LabelValuesForMetricNameRequest, we need to
check if the matchers string is an empty matcher (`{}`) before we parse
the string.

This bug was introduced with the index gateway api refactoring in #5892

Fixes: #5965

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
when index gateway does not implement the new API.

This bug did not yield an error but only returned incorrect results
because both functions `c.IndexStore.LabelNamesForMetricName` and
`c.IndexStore.LabelValuesForMetricName` return `([]string, error)`.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: irizzant <i.rizzante@gmail.com>
* Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value

* Add tests for isWithinIngesterMaxLookbackPeriod

* Adding tests for calculateIngesterMaxLookbackPeriod

* Only send query for range overlapping with ingesters

* Restrict gRPC calls to ingesters when requesting series if there is no overlap within the query_ingesters_within value

* Add tests for isWithinIngesterMaxLookbackPeriod

* Adding tests for calculateIngesterMaxLookbackPeriod

* Only send query for range overlapping with ingesters

* Add test for when to query ingester and storage

* Fix deadlock in Series when just queriying the ingesters

* Refactored slightly for better readability

* Added CHANGELOG entry

* Fix typo

* Rephrase test cases names

* Move block to outer loop

Co-authored-by: Salva Corts <salva.corts@grafana.com>
* Generate cache invalidation numbers in the delete store

* Get cache generation numbers from the store on request

* changlog

* rename tombstones to something more meaningful

* User invisible module

* query frontend relies on a compactor to get the cache generation number

* fix serialization

* source -> name

* lint errors

* lint errors

* log non-200 responses

* add jsonnet changes

* lint

* review feedback

* review feedback

* client rename
This is part 1 of 2 PRs and updates the build image to version
0.10.4 and adding the buf utility which will later be used for
formatting *.proto files
* Disable calling new index-gateway client's API.

* Remove infinite recursive call

* Correct mock setup for test
* Remove empty match within concat of regex

* add back tests
* Revendor grafana/tail.

- This revendoring brings the fix to relative symlinks, as described in
  #3374.

* Prune package from `go mod tidy`.

* noop
…ranteed query sort order when receiving entries for the same stream that have duplicate timestamps. (#6042)

* Add a per tenant ability to fudge duplicate timestamps in the distributor

Signed-off-by: Edward Welch <edward.welch@grafana.com>

* update docs

Signed-off-by: Edward Welch <edward.welch@grafana.com>

* lint

Signed-off-by: Edward Welch <edward.welch@grafana.com>

* update changelog

Signed-off-by: Edward Welch <edward.welch@grafana.com>
* Add two metrics to the IndexGateway.

- Add a new `query_readiness_duration_seconds` metric, that reports
  query readiness duration of a tablemanager/index gateway instance. We
  should use it later to report performance against the ring mode
- Add a new `usersToBeQueryReadyForTotal` metric, that reports number of
  users involved in the query readiness operation. We should use it
  later to correlate number of users with the query readiness duration.

* Remove `usersToBeQueryReadyForTotal`.

- It will report all users always for now, so it isn't too helpful the
  way it is.

* Rename metric help text to not mislead people.

* Log queryReadiness duration.

* Fix where log message and duration and triggered.

* Join users list in a single string.

- This is necessary since go-kit doesn't support array type.

* Tweak queryReadiness log messages.

- As suggested by Ed on
  #5972 (comment) and
  #5972 (comment)

* Ensure queryReadinessDuration metric.

- It is redundant with a recently added log line.

* noop
* Revise the getting started section

* Docs: revise getting started guide

* Intermediate commit of getting started material.

* Intermediate revision of the getting started guide.

    Add instructions to download, configure, and deploy the test environment.

* Add sample queries and instructions for breaking down the test env

* Revise prose using reviewer comments.

* update to use promtail and flog in the docker compose file

Signed-off-by: Edward Welch <edward.welch@grafana.com>

* Revise guide to put all parts into Docker containers.

* Add README files for examples directory. Revise getting started instructions.

* Address DylanGuedes review comments

* Getting started guide: incorporate suggestions from ssncferreira

Co-authored-by: Edward Welch <edward.welch@grafana.com>
This component has been around for a while but has not been added to the
loki-mixin yet.

Signed-off-by: Christian Haudum <christian.haudum@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
…tor (#5879)

* Add filter parameter to rebound

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix linting issues

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add filter function to delete request

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix linting issues

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Enable api for filter and delete mode

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add settings for retention

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Use labels to check and add test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Simplify filter function

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Also enable filter mode

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove test settings in config file for docker

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add extra (unused) param for ProcessString in filter

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Empty commit to trigger CI again

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Update changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix flapping test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove commented out unit tests and add some more

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add extra test case for delete request without line filter

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Use chunk bounds

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* check if the log selector has a filter if the whole chunk is selected

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* fix lint issue: use correct go-kit import

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
KMiller-Grafana and others added 29 commits June 8, 2022 11:29
)

* Docs: Add changelog entry for configuration parameter name change

* Update changelog message, per review request
…by the AST (#6346)

* Panic with an actual error if clone (which calls range aggregation
expr's validate) errors, instead of swallowing the error and then
attempting to execute a nil function, which panics anyways.

Signed-off-by: Callum Styan <callumstyan@gmail.com>

* Call the validation function on a modified expression before attempting
to execute it. If the validation fails we should return the original
expression.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Christopher Norris <christopher.norris@grafana.com>
So that you can do quantile calculation on metadata queries.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
This updates to a version that contains the fix to CVE-2022-28948.
#6310)

* Add support for client-side TLS certs to loki-canary

* fix lint errors, make the case of using TLS but not client certs more clear, check that -tls is set if client certs are used

* move changelog to canary section instead of main

* wrap httpClient errors and simplify dialer creation for websocket
* index sampling endpoint

* fix bug in withTenantLabelMatcher

* show zero values in stats

* handles new errcase

* fixes comments
Signed-off-by: Trevor Wood <Trevor.G.Wood@gmail.com>
* cmd/loki: minor consistency patch

* cmd/loki: fixed a typo
* begins building queryrange compatible types for index sampling

* adds MatcherGroups() method to SampleExpr

* MatcherGroups test + ShardResolver interface

* index stats codec wiring

* millisecond precision on index stats codec

* lint

* cleanMatchers in tsdb.IndexClient for consistency + adds catchall matcher if nil
* Add metrics for journal target

There are no metrics for the journal target yet, this adds two basic
ones:
- Total number of lines processed
- Total number of errors processing lines (with a reason)

Because of the way the journal works, tailing one file with many
possible process writing to it, I chose note to expose counts per unit.

* Add changelog entry

* Fix compilation of (unsupported) journalmanager target on windows

* Extract possible errors to constants & add tests
* Backport memberlist dependency version pin to k102

Signed-off-by: JordanRushing <rushing.jordan@gmail.com>

* Revert "Loki Prometheus mixin: templatize cluster label" (#6377)

* Revert "Loki Prometheus mixin: templatize cluster label (#6353)"

This reverts commit 459036f.

* Replace `loki-mixin` dependency to working commit.

Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com>
(cherry picked from commit 666bac2)

Co-authored-by: Kaviraj Kanagaraj <kavirajkanagaraj@gmail.com>
(cherry picked from commit 372961d)

Co-authored-by: JordanRushing <rushing.jordan@gmail.com>
) (#6493)

(cherry picked from commit efda5b2)

Co-authored-by: Travis Patterson <travis.patterson@grafana.com>
(cherry picked from commit f79ff95)
Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit 671afe6)

Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
(cherry picked from commit a7f8be2)
* Add more logging for delete requests

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix typo

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Rephrase logging message and add chunk ID

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Rename userID field to user

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* rename more userID

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix existing log messages

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit 20db4df)

Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
(cherry picked from commit c707faf)
…) (#6496)

* Compactor: add per tenant compaction delete enabled flag (#6410)

* Add per tenant compaction delete enabled flag

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove changes in wrong place

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add compactor deletion enabled field

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Use limit in compactor

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Use http middleware and add test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix lint issue

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Revert to default setting if no override

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add default value command line option

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Update the docs

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Enable access to deletion API for integration test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Rename flag to allow_deletes

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Update per review comments

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit b4e6c59)

* Fix changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit df48f93)

Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
)

* Remove whole stream deletion mode (#6435)

* Remove whole-stream-deletion mode

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove whole-stream-deletion from docs

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Update the changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Sort changelog entries

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove link to wrong configuration

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix integration test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Set default deletion mode to disabled

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Remove extra white line in documentation

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix default value in docs

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Add DeletionEnabled method on mode

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Rename test

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit f80e487)

* Fix changelog

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>

* Fix changelog link

Signed-off-by: Michel Hollands <michel.hollands@grafana.com>
(cherry picked from commit da3fbd9)

Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
* disable tests which are failing from hardcoded certs which are now expired.

Signed-off-by: Edward Welch <edward.welch@grafana.com>

* lint

Signed-off-by: Edward Welch <edward.welch@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
@dannykopping dannykopping force-pushed the dannykopping/remove-cache-stats branch from 2a5bb86 to 7c90e53 Compare June 28, 2022 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.