Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document multi-tenant queries. #5994

Merged

Conversation

jeschkies
Copy link
Contributor

What this PR does / why we need it:
This change adds documentation for multi-tenant queries. It describes the API usage, limitations and possible filters in queries.

Which issue(s) this PR fixes:
Fixes #5993

Special notes for your reviewer:

Checklist

  • Documentation added
  • Tests updated
  • Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

@@ -17,3 +17,19 @@ To run in multi-tenant mode, Loki should be started with `auth_enabled: true`.
Loki can be run in "single-tenant" mode where the `X-Scope-OrgID` header is not
required. In single-tenant mode, the tenant ID defaults to `fake`.

## Multi-tenant Queries
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether we should document how to setup a datasource in Grafana for multi-tenant queries. It is not trivial IMHO.

If run in multi-tenant mode, queries across different tenants can be enabled via
`multi_tenant_queries_enabled: true` option in the querier. Once enabled multiple
tenant IDs can be defined in the HTTP header `X-Scope-OrgID` by concatenating them
with `|`. For instance a query for tenant A and B can set `X-Scope-OrgID: A|B`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didnt work.
When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that:
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inserting. The piping worked for me. Let me investigate. Thanks for trying it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've tried it with the Docker compose setup in tools/dev/loki-boltdb-storage-s3. How did you try it out?

Copy link
Contributor

@xom4ek xom4ek May 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

analytics:
  reporting_enabled: false
auth_enabled: true
common:
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    s3:
      access_key_id: access
      bucketnames: bucketname
      endpoint: endpoint
      http_config:
        idle_conn_timeout: 90s
        insecure_skip_verify: false
        response_header_timeout: 0s
      region: region
      secret_access_key: secret
compactor:
  shared_store: aws
  working_directory: /var/loki/compactor
frontend:
  log_queries_longer_than: 20s
memberlist:
  join_members:
  - loki-memberlist
querier:
  multi_tenant_queries_enabled: true
schema_config:
  configs:
  - from: "2021-09-07"
    index:
      period: 24h
      prefix: loki_index_
    object_store: aws
    schema: v11
    store: boltdb-shipper
storage_config:
  aws:
    access_key_id: access
    bucketnames: bucketname
    endpoint: endpoint
    http_config:
      idle_conn_timeout: 90s
      insecure_skip_verify: false
      response_header_timeout: 0s
    region: region
    secret_access_key: secret
  boltdb_shipper:
    active_index_directory: /var/loki/index
    cache_location: /var/loki/index_cache
    shared_store: s3
tracing:
  enabled: true

So i found that when use simple-scale helm chart.

If use X-Scope-OrgID='default|system'
image

If use X-Scope-OrgID='default
image

Copy link
Contributor

@xom4ek xom4ek May 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like on my instance multi-tenant queries didnt work.
I need send __tenant_id__ from log collector? Or loki create that labels selves?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the screenshots showing? What are tags here? X-Scope-OrgID='default|system' image should be correct on the datasource. However, that also means Promtail is sending logs either as default or system tenant.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me explain.

When i use Loki datasource and set X-Scope-OrgID=default or X-Scope-OrgID=system - we get that graph:
image

But when i set X-Scope-OrgID=default|system i get "no data".

So my first screenshots just trace from loki where read service get 0 Series when tenants_ids=[default|system] and 3k when tenants_ids=[default]

About promtail - already have tenants in loki and X-Scope-OrgID without multitenant_query work fine - just in code and your example i see __tenant_id label and didnt see that in my loki installation

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That didnt work. When use X-Scope-OrgID=onetenant|twotenant loki response 200 but empty data.

You must use X-Scope-OrgID x2 like that: image

That didnt work correct - loki just take first of that Headers

Copy link
Contributor Author

@jeschkies jeschkies May 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience. I'm not really sure where you see tenants_ids=[default|system].

Anyhow, which version of Loki are you using? It must be a recent main build after at least commit c7cbeab.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - i found problem.

We didnt update master branch after release.
So right now everything fine.
Thanks for help!

jeschkies and others added 3 commits May 2, 2022 10:08
Co-authored-by: Aleksey <a.alexey.lazarev@gmail.com>
…lti-tenant-queries' into karsten/document-multi-tenant-queries
Co-authored-by: JordanRushing <rushing.jordan@gmail.com>
docs/sources/operations/multi-tenancy.md Outdated Show resolved Hide resolved
docs/sources/operations/multi-tenancy.md Outdated Show resolved Hide resolved
jeschkies and others added 2 commits May 6, 2022 20:21
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
Copy link
Contributor

@KMiller-Grafana KMiller-Grafana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs LGTM.

@owen-d owen-d merged commit ff5ce4e into grafana:main Jun 2, 2022
@jeschkies jeschkies deleted the karsten/document-multi-tenant-queries branch June 16, 2022 12:15
@osg-grafana osg-grafana added type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories and removed area/docs labels Oct 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/S type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task] Document multi-tenant queries.
7 participants