[docs] Document Kubernetes standard labels cardinality

[sircthulhu]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.
I'm researching services to handle logs from my kubernetes pods. Loki documentation warns about high cardinality performance issues. I could not find any information if kubernetes standard pod labels (such as pod_name, image and so on) are not too cardinal for storing in Loki index.
Anything I could found is answer by Loki team on Grafana forum: Is labels like instance or k8s pod_name have too high cardinality for Loki

Describe the solution you'd like
I believe there should be a chapter about kubernetes labels in Loki documentation so that newcomers won't need to google this cruical information.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Could there be another solutions? :)

Additional context
Add any other context or screenshots about the feature request here.

[oleksii-boiko-ua]

pod name label would definitely introduce cardinality(depends on number of them of course and how often you deploy things) i would suggest keep unique labels to minimum, we ended up having only namespace, job name, container name as a labels, rest you can now move to unindexed labels(structured metadata)

[Angelin01]

This comment will be slightly off topic, but I'd like to argue that this is not only an issue not only with documentation, but also with the default Helm Chart values from both Loki and Promtail (which is the default go-to collector for Loki).

I ran a quick analysis against our "default setup" helm install Loki and Promtail, this is the result:

/ # logcli series --org-id 1 --analyze-labels '{}'
Total Streams:  387
Unique Labels:  10

Label Name  Unique Values  Found In Streams
filename    365            387
pod         216            387
container   105            387
job         87             387
app         67             387
namespace   31             387
instance    22             162
component   19             93
node_name   7              387
stream      2              387

There's clearly a lot of high cardinality here. filename, pod, etc.

This, coupled with the rather slow default parameters for Loki, hence this long standing issue, makes for quite a bad experience "out of the box". Our S3 bucket contains chunks that are mere kBs in size:

I know at lot of what I am posting is related to Promtail, but at this point I feel like both are very much a coupled experience, most users will simply install both together.

Getting to the point: yes, the documentation could be improved, but I'd much prefer if "saner defaults" were provided instead. People look to default values as a starting point, it's probably the best type of documentation there is: it doesn't rot, and it helps immediately both those that need to customize and those that need to fine tune.

[oleksii-boiko-ua]

pod name label would definitely introduce cardinality(depends on number of them of course and how often you deploy things) i would suggest keep unique labels to minimum, we ended up having only namespace, job name, container name as a labels, rest you can now move to unindexed labels(structured metadata)

[Angelin01]

This comment will be slightly off topic, but I'd like to argue that this is not only an issue not only with documentation, but also with the default Helm Chart values from both Loki and Promtail (which is the default go-to collector for Loki).

I ran a quick analysis against our "default setup" helm install Loki and Promtail, this is the result:

/ # logcli series --org-id 1 --analyze-labels '{}'
Total Streams:  387
Unique Labels:  10

Label Name  Unique Values  Found In Streams
filename    365            387
pod         216            387
container   105            387
job         87             387
app         67             387
namespace   31             387
instance    22             162
component   19             93
node_name   7              387
stream      2              387

There's clearly a lot of high cardinality here. filename, pod, etc.

This, coupled with the rather slow default parameters for Loki, hence this long standing issue, makes for quite a bad experience "out of the box". Our S3 bucket contains chunks that are mere kBs in size:

I know at lot of what I am posting is related to Promtail, but at this point I feel like both are very much a coupled experience, most users will simply install both together.

Getting to the point: yes, the documentation could be improved, but I'd much prefer if "saner defaults" were provided instead. People look to default values as a starting point, it's probably the best type of documentation there is: it doesn't rot, and it helps immediately both those that need to customize and those that need to fine tune.