Lockup

[splitice]

Sometimes our remote systems stop outputting events. When this happens its fatal until restart.

vector[1972]: 2025-12-05T02:55:24.938488Z  WARN sink{component_kind="sink" component_id=output component_type=vector}:request{request_id=89517}: vector::sinks::util::retries: Internal log [Request timed out. If this happens often while the events are actually reaching their destination, try decreasing `batch.max_bytes` and/or using `compression` if applicable.  ...

This is the sink configuration.

[sinks.output]
type = "vector"
inputs = [ "parse_logs" ]
address = "{{ vector_logging_server }}"
version = "2"
compression = true
request.concurrency = 16
batch.max_bytes = 60000000
  [sinks.output.buffer]
#  max_size = 33_438_953_472
  max_events = 30000
  when_full = "drop_newest"

Other remotes dont usually show any issue at the same time and its very infrequent.

vector_logging_server is a DNS name resolving to multiple records (kubernetes nodeport)

I'm not entirely sure how to troubleshoot this. We recently updated to 0.51.1 to see if the issue would go away, it didnt. Theory is ever a stalled connection or stalled dns request. Is either a likely cause?