Implementation detail of kubernetes_logs source based on k8s API

[titaneric]

I would like to dive into the detail of implemetation of #23597 here. Since the CONTRIBUTING.md said that it's discouraged to submit a PR exceeding 300-500 lines of code. I think this feature is too big to fit into one single PR, which means multiple PR might be needed.

I have started a very basic implemtation here, thanks to Copilot, which would print 10 lines of pod's log of each pods, and I wrap these features into a new introduced feature toggle called sources-kubernetes_logs-api.

I want to ask how large the PR should be included in the PR , and what kind of condition should be meet in the content. Before submitting the PR, I want to align the implementation with the maintainer here to avoid any differences between our ideas.

Here are the future TODO:

1. Transform the log into Event and send to output stream
2. Filter pods by label
3. Record log's timestamp and start streaming the log from it
4. Validate the output between original and new implementation
5. Benchmark between them

I would like to hear more discussion about this feature

[pront]

Hi @titaneric, for brand new components it is challenging to keep a PR under 300 lines (even if we excluded the generated CUE file changes). There are techniques we can follow to split this up into multiple PRs but feel free to start with a single PR. We will review and if we think it's too large, we will recommend a better strategy. Try to trim down all optional features that can be easily added later without breaking the config UX.

[titaneric]

Got it, thanks for your quick response. I would submit a PR when I think it's ok with the reasonable changed lines.

For the implementation detail, I would like to follow the way how docker_logs source did. Starting with a event stream builder to stream the logs, decorate them, and send to SourceSender for each containers and pods.