[helm] Fix read-only /tmp in single-binary mode

[MiggL]

What this PR does / why we need it:
When running in single-binary mode, these warnings may appear:
level=warn ts=2022-11-15T08:01:39.777179378Z caller=marker.go:214 msg="failed to process marks" path=/var/loki/compactor/retention/markers/1668476560013344947 err="open /tmp/marker-view-912253853: read-only file system"

This was previously fixed in the old chart: grafana/helm-charts#967

Which issue(s) this PR fixes:
Fixes grafana/helm-charts#609, again.

Checklist

• Reviewed the CONTRIBUTING.md guide

[CLAassistant]

All committers have signed the CLA.

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[stefanandres]

Yes, please merge this. This breaks all retentions for existing installations that upgraded. I had to add the workaround like 10months ago back to our installations:

  # https://github.com/grafana/loki/pull/7695
  extraVolumeMounts:
    - name: tmp
      mountPath: /tmp
  extraVolumes:
    - name: tmp
      emptyDir: {}

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%