All prometheusrules are reported back always for all tenants

[slawekww]

Thanos, Prometheus and Golang version used:
image: bitnami/thanos:0.36.1-debian-12-r3
thanos helm chart: thanos-15.8.0

Object Storage Provider:
Azure Storage Account

What happened:
Prometheus operator deploys prometheus instances.
Each prometheus instance has own set of prometheusrules for alerting and use thanos sidecar.
Prometheus rules are correctly inserted to thanos via thanos sidecar.
Thanos query responds with all prometheus rules for all prometheus instances.

1, Thanos query is defined for multi-tenancy:

        query:
          extraFlags:
            - --query.replica-label=replica
            - --query.enforce-tenancy
            - --query.tenant-header=THANOS-TENANT
            - --query.tenant-label-name=tenant

2. Each prometheus instance has thanos sidecar:

            thanos:
              objectStorageConfig:
                existingSecret:
                  name: thanos-object-shared-storage-account
                  key: objstore.yml
...
            externalLabels:
              env: dev
              tenant: tenant2

Prometheus rules are correctly seen on thanos UI:

alert:AlertmanagerClusterCrashlooping
  expr:(count by (namespace, service, cluster) (changes(process_start_time_seconds{job="prometheus-tenant2- prom-te-alertmanager",namespace="tenant2"}[10m]) > 4) / count by (namespace, service, cluster) (up{job="prometheus-tenant2-prom-te-alertmanager",namespace="tenant2"})) >= 0.5
  for: 5m
  labels:
    cluster: abc
    env: dev
    severity: critical
    tenant: tenant2 # each instance has different tenant value

4. Grafana datasource is defined to use header:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: thanos
  namespace: tenant2
spec:
  instanceSelector:
    matchLabels:
      dashboards: "tenant2"
  allowCrossNamespaceImport: false
  datasource:
    name: "Global - Prometheus"
    type: prometheus
    access: proxy
    uid: "global-prometheus"
    url: http://thanos-query-frontend.thanos.svc.cluster.local:9090
    isDefault: true
    jsonData:
      httpHeaderName1: 'THANOS-TENANT'
    secureJsonData:
      httpHeaderValue1: 'tenant2'

Grafana query alert rules like is like curl command:

curl -H 'THANOS-TENANT: tenant2' http://thanos-query-frontend.thanos.svc.cluster.local:9090/api/v1/rules

5. Output is that all prometheus rules are responded regardless if it is tenant2 or different value.

What you expected to happen:

Thanos query should respond with prometheusrules with specific label

How to reproduce it (as minimally and precisely as possible):

1. configure thanos with mulitenancy
2. configure two different prometheus instances using different external labels and thanos sidecar
3. insert different prometheusrules to each prometheus instance
4. see thanos rules on UI
5. see thanos query response via curl or via grafana datasource using tenant header

Full logs to relevant components:

Can deliver if required

Anything else we need to know:

Run on K8S 1.32

[dosubot]

Hey there, @slawekww! I'm here to help you with any bugs, questions, or anything else you need assistance with. Let's tackle this issue together!

I found a similar closed issue that might be relevant to your situation. The issue discussed the requirement for the THANOS_TENANT header on queries against the querier, enforcing the query towards that tenant and rejecting queries that do not specify it. The solution involved using a special configurable tenant label to support multi-tenancy more natively, which would make the process more efficient and secure by not having to parse the PromQL query but doing it at selection time. The feature has been completed as per the final comments in the discussion ^[1].

Additionally, ensure that the --query.enforce-tenancy flag is correctly set in your Thanos Query configuration, as this flag is crucial for enforcing tenant-specific queries. The tenant-label-name should match the label used in your Prometheus instances, which seems to be correctly set to tenant in your configuration.

If the issue persists, double-check that the THANOS-TENANT header is being correctly passed and recognized by the Thanos Query component. You might also want to verify that the tenant label is correctly applied and enforced during query execution, as Thanos uses several mechanisms to ensure this, including tenant header configuration and tenant label enforcement ^[2][3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[slawekww]

@dosu
Yes, for metrics query all works as expected - only tenants metrics are replied and feature works as expected.
Problem reported is for prometheusrule which are not metrics but alert/recording rules calling via /api/v1/rules endpoint to thanos query and this quert does not take into account THANOS_TENANT: tenant header and always responded with all prometheusrules defined for all tenants - this is real issue reported.

[dosubot]

The issue you're experiencing with the /api/v1/rules endpoint not respecting the THANOS_TENANT header for tenant-specific rule queries is indeed a known challenge. The Thanos codebase does have a mechanism for processing tenant headers, specifically through the GetTenantFromHTTP function in the pkg/tenancy/tenancy.go file, which is responsible for extracting tenant information from HTTP requests ^[1]. However, it seems that this mechanism might not be applied to non-metric endpoints like /api/v1/rules.

Unfortunately, I couldn't find specific documentation or code comments in the Thanos repository that describe how tenant headers are processed for non-metric endpoints like /api/v1/rules. This might indicate a gap in the current implementation regarding tenant enforcement for alerting and recording rules.

To address this, you might need to explore the possibility of implementing or requesting an enhancement in Thanos to ensure tenant headers are respected for the /api/v1/rules endpoint. This would involve ensuring that tenant-specific logic is applied consistently across all endpoints, not just metric queries.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[slawekww]

@dosu
Could you accept it as bug?

[dosubot]

This is a tough nut to crack, could really use your help on this, @kakkoyun.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}