File exporter group by attr

[adam-kiss-sg]

Description:

Added the option to write telemetry data into multiple files, where the file path is based on a resource attribute.

Link to tracking Issue:

#24654

Testing:

Added tests and benchmark for now functionality.

Documentation:

Updated README.md

[adam-kiss-sg]

I will clean up the commit history tomorrow. Should I also move the restructuring part to a separate pr?

[adam-kiss-sg]

Hi @djaglowski, I have some questions about some implementation details:

Error handling

There are some errors that I'm not sure what's the best approach for handling them (should I simply log them? or return an error?)

• when creating a file fails because of permission issue
• when creating a file fails because the parent directory doesn't exists, and auto_create_directories is set to false
• when creating a directory fails
• when closing a file fails

I'm mostly concerned about cases where a batch of telemetry data contains multiple resources, and some of them are written successfully, while others fail.

Data modification

Should the exporter modify the telemetry data? I use ResourcesX.MoveTo() in my code, because it showed much better performance compared to CopyTo. There is also the delete_sub_path_resource_attribute config which would remove an attribute from the resource. If a pipeline contains multiple exporters, would this modify the data for the other exporters as well? What's the best practice here?

Rotation

I disabled rotation when group_by is active, mainly because I'm concerned about the interaction between rotation (using lumberjack) and the group_by functionality. You can also implement some kind of rotation with group_by by adding a date to the resource attribute, so this didn't seem so much of an issue (although it's time based, not size based, so not exactly the same functionality). What do you think about this?

README.md

I noticed this line in the README.md, should I remove it? This intended for primarily for debugging Collector without setting up backends.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[djaglowski]

@adam-kiss-sg, I know we merged your first PR but where is this at otherwise?