chore: remove loki annotations from gateway nginx deployment

[fcjack]

What this PR does / why we need it:

The gateway nginx deployment was applying the loki annotation values and the gateway values.
This can cause a configuration error since the gateway container is not a loki container with the same ports.

Inside Enterprise gateway we already use only the gateway annotations.

Which issue(s) this PR fixes:
Fixes #16918

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[github-actions]

Helm Diff Output - Summary

Default Values Scenario-diff-output

Ingress Values Scenario-diff-output

default, loki-release-gateway, Deployment (apps) has changed:
  # Source: loki/templates/gateway/deployment-gateway-nginx.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: loki-release-gateway
    namespace: default
    labels:
      helm.sh/chart: loki-6.29.0
      app.kubernetes.io/name: loki
      app.kubernetes.io/instance: loki-release
      app.kubernetes.io/version: "3.4.2"
      app.kubernetes.io/component: gateway
  spec:
    replicas: 1
    strategy:
      type: RollingUpdate
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app.kubernetes.io/name: loki
        app.kubernetes.io/instance: loki-release
        app.kubernetes.io/component: gateway
    template:
      metadata:
        annotations:
          checksum/config: 440a9cd2e87de46e0aad42617818d58f1e2daacb1ae594bad1663931faa44ebc
-         prometheus.io/port: "3100"
-         prometheus.io/scrape: "true"
        labels:
          app.kubernetes.io/name: loki
          app.kubernetes.io/instance: loki-release
          app.kubernetes.io/component: gateway
      spec:
        serviceAccountName: loki
        enableServiceLinks: true
        
        securityContext:
          fsGroup: 101
          runAsGroup: 101
          runAsNonRoot: true
          runAsUser: 101
        terminationGracePeriodSeconds: 30
        containers:
          - name: nginx
            image: docker.io/nginxinc/nginx-unprivileged:1.27-alpine
            imagePullPolicy: IfNotPresent
            ports:
              - name: http-metrics
                containerPort: 8080
                protocol: TCP
            readinessProbe:
              httpGet:
                path: /
                port: http-metrics
              initialDelaySeconds: 15
              timeoutSeconds: 1
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
            volumeMounts:
              - name: config
                mountPath: /etc/nginx
              - name: tmp
                mountPath: /tmp
              - name: docker-entrypoint-d-override
                mountPath: /docker-entrypoint.d
            resources:
              {}
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: gateway
              topologyKey: kubernetes.io/hostname
        volumes:
          - name: config
            configMap:
              name: loki-release-gateway
          - name: tmp
            emptyDir: {}
          - name: docker-entrypoint-d-override
            emptyDir: {}

Legacy Monitoring Values Scenario-diff-output

default, loki-release-gateway, Deployment (apps) has changed:
  # Source: loki/templates/gateway/deployment-gateway-nginx.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: loki-release-gateway
    namespace: default
    labels:
      helm.sh/chart: loki-6.29.0
      app.kubernetes.io/name: loki
      app.kubernetes.io/instance: loki-release
      app.kubernetes.io/version: "3.4.2"
      app.kubernetes.io/component: gateway
  spec:
    replicas: 1
    strategy:
      type: RollingUpdate
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app.kubernetes.io/name: loki
        app.kubernetes.io/instance: loki-release
        app.kubernetes.io/component: gateway
    template:
      metadata:
        annotations:
          checksum/config: 440a9cd2e87de46e0aad42617818d58f1e2daacb1ae594bad1663931faa44ebc
-         prometheus.io/port: "3100"
-         prometheus.io/scrape: "true"
        labels:
          app.kubernetes.io/name: loki
          app.kubernetes.io/instance: loki-release
          app.kubernetes.io/component: gateway
      spec:
        serviceAccountName: loki
        enableServiceLinks: true
        
        securityContext:
          fsGroup: 101
          runAsGroup: 101
          runAsNonRoot: true
          runAsUser: 101
        terminationGracePeriodSeconds: 30
        containers:
          - name: nginx
            image: docker.io/nginxinc/nginx-unprivileged:1.27-alpine
            imagePullPolicy: IfNotPresent
            ports:
              - name: http-metrics
                containerPort: 8080
                protocol: TCP
            readinessProbe:
              httpGet:
                path: /
                port: http-metrics
              initialDelaySeconds: 15
              timeoutSeconds: 1
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
            volumeMounts:
              - name: config
                mountPath: /etc/nginx
              - name: tmp
                mountPath: /tmp
              - name: docker-entrypoint-d-override
                mountPath: /docker-entrypoint.d
            resources:
              {}
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: gateway
              topologyKey: kubernetes.io/hostname
        volumes:
          - name: config
            configMap:
              name: loki-release-gateway
          - name: tmp
            emptyDir: {}
          - name: docker-entrypoint-d-override
            emptyDir: {}

Simple Scalable AWS Kube IRSA Values Scenario-diff-output

Simple Thanos Values Scenario-diff-output

Single Binary Scenario-diff-output

[trevorwhitney]

looks good as long as enterprise still works 👍

[trevorwhitney]

as long as we're sure this doesn't negatively affect enterprise deployments, then I'm good with it