Add node labels as tags on events from the `kubernetes_logs` source

[jszwedko]

Similar to pod labels, we should add node labels to log events from the kubernetes_logs source.

Requested by user: #7480 (comment)

[nabokihms]

A nice to have feature! @jszwedko, could I work on this one? It seems like vector needs to run an additional informer and a node annotator (not a complicated task to write an RFC for it).

[spencergilbert]

My only concern about adding additional Watchers for other resources is increased pressure on the cluster's control plane. Since each instance of Vector is keeping it's own individual state here, every instance would be two (today, Pods + Namespace) additional calls to the apiserver. That obviously scales linearly with the number of Nodes in the cluster and could cause undesirable load (ref: vectordotdev/helm-charts#24 (comment)).

That said, I'm not against this feature - and would be interested to know if you've seen any issues caused by the implementation "in the wild" @nabokihms, as I suspect you may have seen more large scale Agent installations than I have 😄 .

[nabokihms]

I think using a field selector Vector can subscribe only on a single Node object. It will also depend on a node number and affect Kubernetes control-plane resources consumption, but I hope not significantly. I will open a PR with the implementation to discuss it first.

Regarding our Kubernetes installations, there are different clusters with no more than 250 nodes (each with a Vector pod aboard). They may not be as large as other installations, but I believe they are good enough to catch problems/ugly behaviors 😉 .