[Helm chart 6.29] Upgrading chart to 6.29.0 bricks basic auth functionality and allows reading X-Scope-OrgID not owned by user

[Phsacar]

Describe the bug
When I upgrade my local scalable loki from 6.28.0 to 6.29.0 , basic auth does not work as expected. You can even get data of other X-Scope-OrgIDs

To Reproduce
Steps to reproduce the behavior:

1. Setup loki helm in scalable mode and define some users with basic auth
2. setup Datasource in grafana with username and password without specifying X-Scope-OrgID
3. get no org errors on every query
4. Try setting X-Scope-OrgID in the datasource manually ( you can insert any X-Scope-OrgID - the username wont get enforced )
5. Datasource works - however - you can now access data of different X-Scope-OrgID

Expected behavior

1. Setup loki helm in scalable mode and define some users with basic auth
2. setup Datasource in grafana with username and password without specifying X-Scope-OrgID
3. working queries and only access to the X-Scope-OrgID that corresponds to my username

Environment:

• Infrastructure: k8s running on talos linux
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
na - can post configmap of the autodeployed nginx ( standard when using scalable loki ) if needed

Thanks!

[a5r0n]

Same here, the changes in #16671 seems to cause this.
as workaround, comment out this line:

        location ^~ /loki/api/v1/ {
          # pass custom headers set by Grafana as X-Query-Tags which are logged as key/value pairs in metrics.go log messages
==>  # proxy_set_header X-Query-Tags "${query_tags},user=${http_x_grafana_user},dashboard_id=${http_x_dashboard_uid},dashboard_title=${http_x_dashboard_title},panel_id=${http_x_panel_id},panel_title=${http_x_panel_title},source_rule_uid=${http_x_rule_uid},rule_name=${http_x_rule_name},rule_folder=${http_x_rule_folder},rule_version=${http_x_rule_version},rule_source=${http_x_rule_source},rule_type=${http_x_rule_type}";
          proxy_pass       http://loki-read.logs.svc.cluster.local:3100$request_uri;
        }

in logi-gateway config and restart the gateway pods.

[Phsacar]

Can confirm that this works - thanks!

[a5r0n]

https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_set_header

Allows redefining or appending fields to the request header passed to the proxied server. The value can contain text, variables, and their combinations. These directives are inherited from the previous configuration level if and only if there are no proxy_set_header directives defined on the current level.

I'll open PR to fix that.