[Loki Plugin] Allow setting X-Scope-OrgID header dynamically

[marpio]

Is your feature request related to a problem? Please describe.
Similar to the to the situation described in grafana/loki#259 we control both the Loki backend and the fluent-bit installation and are running multiple tenants on Kubernetes (a tenant is a namespace), and would like to set the tenant (controled by X-Scope-OrgID) based on said namespace.

Describe the solution you'd like
I would like to be able to set the X-Scope-OrgID dynamically based on a field (in my case the namespace) provided by the kubernetes filter.

Describe alternatives you've considered
I could put a proxy in front of the Loki API that would inspect the labels and split the logs accordingly.

Additional context
As already mentioned in grafana/loki#259 the scenario where both sides are controlled is pretty common and it would be great if the native Loki plugin would support that just as Promtail and, indirectly (via a reserved __tenant_id__ label)the fluent-bit plugin from Grafana https://grafana.com/docs/loki/latest/clients/fluentbit/ does.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[edsiper]

assigned to @nokute78

[nokute78]

@marpio @edsiper How about add new option like tenant_id_key ?
If the key name is set, fluent-bit set the value of the key as a X-Scope-OrgID.

Example

Incoming data is

{"namespace_name":"DynamicOrgID_12345", "data":"XXXXX"}

Option is

tenant_id_key namespace_name

Then X-Scope-OrgID will be DynamicOrgID_12345.

I'm not familiar with kubernetes, is it meaningful for this issue ?
I think we can get namespace by filter_kubernetes. The new option and filter_kubernetes may help it.
https://docs.fluentbit.io/manual/pipeline/filters/kubernetes
https://rubular.com/r/HZz3tYAahj6JCd

[dbluxo]

@nokute78 It would be great to have a tenant_id_key and be able to set the namespace name/value from filter_kubernetes dynamically 👍

[marpio]

@nokute78 yes, that would be perfect. I'm already using the kubernetes filter and yes, it's possible to extract the name of a namespace. Thanks!

[nokute78]

Up voting and @marpio @dbluxo Thank you!

I sent a patch #3260.
It supports tenant_id_key.

It is a simple implementation, so it may not handle such events.

Disclaimer:
If different values are sent to a loki plugin at once, X-Scope-OrdID will be set the value of last record.

e.g.
These events are sent to same loki plugin instance, X-Scope-OrdID will MyTenant2 using option tenant_id_key namespace_name

{"namespace_name":"MyTenant", "data":"hoge"}
{"namespace_name":"MyTenant2", "data":"hoge"}

[dbluxo]

@nokute78 Thanks for your quick implementation!

Disclaimer: If different values are sent to a loki plugin at once, X-Scope-OrdID will be set the value of last record.

Isn't that a bit dangerous if a user wants to map a multi-tenancy via this option? It certainly happens very often that a request to Loki contains logs from different Kubernetes namespaces, because Pods from different namespaces run on one node.

[nokute78]

@dbluxo Thank you for the comment especially kubernetes.

Hmm, current loki plugin creates 1 http request from N events, this design causes that Disclaimer.
So, we have to change the basic design drastically.
(In this case, fluent-bit should create 1~N http request(s) from N events.)

Hmm...

[marpio]

@nokute78 I have the same concern as dbluxo. Basically fluent-bit, in my environment, is running as a DaemonSet (one instance on each node). Since every node can host multiple pods from many namespaces, it potentially also reads logs from multiple tenants (in my case tenant maps to a namespace). With this implementation, tenant A would be able to see logs from tenant B which would defeat the purpose of this feature.
I haven't read the code of the plugin but maybe a buffer with multiple "buckets" (one per tenant) would be an option here? The buckets could be flushed separately(separate requests) depending on the configured size of the bucket/whole buffer and time.

Thanks again and let me know what you think.

[nokute78]

@marpio Thank you for comment.

What I think is here. Is it meaningful?

1. Loki plugin receives multiple events. Each record may have different namespace.
2. The plugin looks into multiple events and groups into several units by tenant_id_key
3. (Buffering until limit?)
4. The plugin creates requests for each units (and it means for each X-Scope-OrgID) and send them.

e.g.
tenant_id_key is namespace_name and incoming records are 3.

{"namespace_name":"DynamicOrgID_12345", "data":"XXXXX"}
{"namespace_name":"test", "data":"XXXXX"}
{"namespace_name":"DynamicOrgID_12345", "data":"YYYYY"}

The Loki plugin creates 2 units.

Unit A

{"namespace_name":"DynamicOrgID_12345", "data":"XXXXX"}
{"namespace_name":"DynamicOrgID_12345", "data":"YYYYY"}

and Unit B

{"namespace_name":"test", "data":"XXXXX"}

Finally, the plugin sends 2 http requests.
One's X-Scope-OrgID is DynamicOrgID_12345 and the other is test.

[marpio]

@nokute78 yes, this is exactly how it should work

[dbluxo]

@nokute78 Your suggestion is great! I think with the buffer_limit it is also important to consider that if in a certain time the buffer_limit is not reached, the messages are still sent to Loki after a specified time, so that even messages from tenants who log less, do not wait "forever".

[nokute78]

2. The plugin looks into multiple events and groups into several units by tenant_id_key

rewrite_tag filter plugin can serialize by namespace.

I tested this example log. It contains multiple namespace_names.

{"namespace_name":"AAA", "data":"aaaa"}
{"namespace_name":"AAA", "data":"bbbb"}
{"namespace_name":"BBB", "data":"aaaa"}
{"namespace_name":"AAA", "data":"cccc"}
{"namespace_name":"BBB", "data":"bbbb"}
{"namespace_name":"CCC", "data":"aaaa"}
{"namespace_name":"BBB", "data":"cccc"}
{"namespace_name":"AAA", "data":"dddd"}

rewrite_tag plugin serializes and rewrites tags like rewrite. + the value of namespace_name.
e.g.

[0] rewrite.AAA: [1616820351.726180701, {"namespace_name"=>"AAA", "data"=>"aaaa"}]
[1] rewrite.AAA: [1616820351.726191913, {"namespace_name"=>"AAA", "data"=>"bbbb"}]
[2] rewrite.AAA: [1616820351.726201421, {"namespace_name"=>"AAA", "data"=>"cccc"}]
[3] rewrite.AAA: [1616820351.726218374, {"namespace_name"=>"AAA", "data"=>"dddd"}]

and

[0] rewrite.BBB: [1616820351.726197002, {"namespace_name"=>"BBB", "data"=>"aaaa"}]
[1] rewrite.BBB: [1616820351.726205699, {"namespace_name"=>"BBB", "data"=>"bbbb"}]
[2] rewrite.BBB: [1616820351.726214016, {"namespace_name"=>"BBB", "data"=>"cccc"}]

and

[0] rewrite.CCC: [1616820351.726209938, {"namespace_name"=>"CCC", "data"=>"aaaa"}]

Then #3260 will set X-Scope-OrgID and creates http request.

Configuration example

[SERVICE]
    Parsers_File parsers.conf

[INPUT]
    Name tail
    Tag  raw
    Path sample.log
    Parser json
    Read_From_Head true

[FILTER]
    Name  rewrite_tag
    Match   raw
    Rule   $namespace_name ^(.+)$ rewrite.$namespace_name false
    Emitter_Name re_emitted

[OUTPUT]
    name loki
    Match rewrite.*
    tenant_id_key namespace_name

Pipeline

1. in_tail -> filter_rewrite_tag -> (rewrite tag name raw -> rewrite.XXX and serialize by tag) -> (in_emitter)
2. (in_emitter) -> out_loki

[dbluxo]

@nokute78 Do I understand this correctly, by serializing (using rewrite_tag) by namespace for each namespace a separate http request is sent?

What would the configuration have to look like if you wanted to use the kubernetes filter to set the tenant_id_key via the namespace name?

[nokute78]

Do I understand this correctly, by serializing (using rewrite_tag) by namespace for each namespace a separate http request is sent?

Yes. In this case, 3 http requests are sent.
First request has a header X-Scope-OrgID:AAA and it has only records which namespace_name is AAA.
The second is about BBB, the third is about CCC.

(You can check that behavior using current release version except X-Scope-OrgID )

The point is

1. Write filter_kubernetes configuration before rewrite_tag to rewrite after appending namespace_name field.
2. Rewritten tag must not be matched rewrite_tag Match case. If it is matched, it will be infinite loop.

(something input) -> filter_kubernetes (appends namespace_name field ) -> filter rewrite_tag (rewrite tag) -> (in_emitter) -> output loki (tenant_id_key property appends X-Scope-OrgID header using the patch #3260 )

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nokute78]

not stale