loki tenant_id_key eval is too late

[sepich]

Bug Report

Describe the bug
In loki output the remove_keys works ahead of tenant_id_key, making it impossible to only send log message with drop_single_key. The label_keys also strip the key from message before tenant_id_key able to read it.

To Reproduce

• Consider this config:

    # https://github.com/fluent/fluent-bit/issues/2935#issuecomment-808657942
    [FILTER]
        Name  rewrite_tag
        Match kube.*
        Rule  $tenant_id ^(.+)$ requeue.$tenant_id false
        Emitter_Name re_emitted
    [OUTPUT]
        Name loki
        Match requeue.*
        Host loki-write.loki.svc
        label_keys $namespace,$container,$host,$app,$stream
        drop_single_key On
        line_format key_value
        tenant_id_key tenant_id
        remove_keys namespace,container,host,app,stream,tenant_id

I have keys log with log message, and tenant_id with desired id. I want to send event to loki having only clear message, and separate labels to this tenant, that why I'm removing all the keys and use drop_single_key.
Unfortunately this leads to:

[2022/10/14 07:14:43] [ warn] [output:loki:loki.0] the value of tenant_id is missing
[2022/10/14 07:14:43] [error] [output:loki:loki.0] loki-write.loki.svc:3100, HTTP status=401
no org id

• If I drop tenant_id from remove_keys - data is being sent, but in loki it looks like:
  {labels} tenant_id=xxx log=aaa

Expected behavior
Log message above expected to look in loki as:
{labels} aaa

Your Environment

• Version used: cr.fluentbit.io/fluent/fluent-bit:1.9.9

Additional context
I'm trying to use fluent-bit for multitenant loki logs per-namespace in k8s. But unfortunately upstream loki support seems to be too broken for that (see #2935 also).
I've found there is another fluent-bit with loki output: https://grafana.com/docs/loki/latest/clients/fluentbit/#configuration-options
But it only have a static tenant_id configuration (not per-event).
Having 2 different fluent-bit loki implementations seems strange.
Will try to migrate to vector.dev, as per-docs it should be working there.

[nokute78]

I sent a patch #6249. Could you check it ?

[sepich]

Thanks, i've build you branch as sepa/fluentbit image. No more the value of tenant_id is missing errors, I do see logs coming to tenants.