kubernetes_logs source: Additional information on annotation_failed errors

[spencergilbert]

We're experiencing the same problem. The time when the logs stop shipping correlates with annotation_failed errors:

2022-05-30T04:31:09.092406Z ERROR source{component_kind="source" component_id=kubernetes_logs
component_type=kubernetes_logs component_name=kubernetes_logs}: vector::internal_events::kubernetes_logs:
Failed to annotate event with pod metadata. event=Log(LogEvent { fields: Object({"file":
Bytes(b"/var/log/pods/*redacted*/0.log"), "message": Bytes(b"2022-05-30T11:31:09.04624439+07:00 stderr F
*redacted*"), "source_type": Bytes(b"kubernetes_logs"), "timestamp": Timestamp(2022-05-30T04:31:09.092358889Z)}),
metadata: EventMetadata { datadog_api_key: None, splunk_hec_token: None, finalizers: EventFinalizers([]),
schema_definition: Definition { collection: Collection { known: {}, unknown: None }, meaning: {}, optional: {} } } })
error_code="annotation_failed" error_type="reader_failed" stage="processing"

Originally posted by @aikoven in #12014 (comment)

The kube-rs upgrade looks to have helped with some of the errors, but there seems to be an issue with annotating unrelated to deletion of metadata. Improving the logging here may point us in the right direction.

[kitos9112]

Experiencing this odd behaviour with all my statefulsets every time the pods roll over and reuse the same name. Grateful to have found someone with a similar issue!