Support dynamic paths based on attributes of logs

[captainfalcon23]

Component(s)

exporter/file

Is your feature request related to a problem? Please describe.

I want to send our kubernetes pod logs to an opentelementry server and then write them to disk to log files based on attributes within the logs. For example: /data/logs/NAMESPACE_NAME/POD_NAME/CONTAINER_NAME/out.log

Describe the solution you'd like

As above, allow the file exporter to be able to read attributes of the logs and dynamically generate the log location.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atingchen]

@captainfalcon23 I think the essence of the problem is to export logs to different paths through specific attributes, thus achieving log isolation. To solve this problem, can we use the existing routingprocessor? The routingprocessor reads specific resource attributes and exports the logs to different fileexporters based on the attribute values.

[captainfalcon23]

So I am starting to look at this, but still don't think it will meet the requirement.
Let's say I get logs from a container with the following details:
Container Name: zabbix-agent-1234567
Pod Name: zabbix-agent
Namespace: foo
Cluster (static value): ABC

In this case, I want to save logs to a location like: /mnt/data/logs/$Cluster/$Namespace/$PodName/$ContainerName.log

If I use routing processor, I think I still need to hard code values, and I have to know exactly where my pods are running and if there's some change, I need to manually update this configuration.

Another example is the host logs (e.g. /var/log/messages. I want these to go to a file which is dynamically named "$hostname.log". But again, it seems impossible?

What's your thoughts @atingchen ?

[atingchen]

If we need to implement dynamic file export paths, I think we need to follow these three steps:

1. We need to add a subdirectory_attribute in the configuration. It will be used to generate the export path.

path: /data
subdirectory_attribute:
  - cluster
  - namespace

2. We need to maintain a mapping between the file export path and the io.writer.

   • If the export path is not in our mapping, we need to create a new io.writer.
   • We also need to consider the eviction mechanism of this map, otherwise, there may be a risk of memory leakage.

3. Before exporting telemetry data each time, obtaining io.writer based on the file export path and exporting it.

If we implement it this way, the existing code needs to be significantly modified. I'd like to hear your thoughts on this. @atoulme @djaglowski Thank you.

[djaglowski]

There does seem to be some valuable utility here but it would certainly add a lot of complexity and responsibility to the component. I'm not opposed to it in theory but don't think I can make it a priority to help with it.

[captainfalcon23]

No worries... I am actually about to start looking at some other solution as I can't get my use-case to work as it stands.

[atoulme]

Same way, I would use the routingprocessor and limit this to a set of known paths.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[captainfalcon23]

Happy for these to be closed if you guys like. I moved to another, more flexible solution (vector.dev)