Improve kubernetes_logs source performance

[splitice]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We are collecting logs for an active kubernetes cluster and would like to see improved performance.

~1k lines/sec per 1 vCPU appears to be the max peak

Attempted Solutions

• VRL reduction of the logged line

Proposal

1. Provide an option to reduce the data inserted into the log line at the source level
2. Potentially offer some form of acceleration for the kubernetes object (perhaps caching could be done wile the object is already in json from the cache?)

References

No response

Version

0.24.1

[splitice]

@jszwedko I think this also needs a performance domain tag.

I really hope that some time soon theres a performance targetting milestone. The resource usage of vector within our stack is really starting to get excessive (very comparable to the system it replaced. logstash). I'll be opening a general issue soon too with some performance profiling data.

[splitice]

I created my issue on performance as a discussion as I feel its more appropriate #15977