Vector pods restart continuously because of "connection refuse" from the target.

[JayJay-K]

Question

Hello.

I injected vector container to the tetragon pod.
(but the situation is same when I install vector pod with daemonset)

The pods restart again and again:

[admin@bastion]$ kubectl get pod
NAME                               READY   STATUS    RESTARTS          AGE
tetragon-jxpv2                     2/3     Running   451 (3m59s ago)   41h

I checked the vector container's log and I could see following logs:

2026-07-15T22:53:58.847021Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111)
2026-07-15T22:54:15.148924Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: tcp connect error: Connection refused (os error 111) error_type="request_failed" stage="processing"
2026-07-15T22:54:15.148967Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111)
2026-07-15T22:54:17.978948Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: Internal log [HTTP error.] is being suppressed to avoid flooding.
2026-07-15T22:54:17.978994Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2026-07-15T22:54:18.758516Z  INFO vector::signal: Signal received. signal="SIGTERM"
2026-07-15T22:54:18.758563Z  INFO vector: Vector is stopping.
2026-07-15T22:54:18.759779Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="tet_out, tet_out2, tet_in" time_remaining="59 seconds left"

I think it might happen because of consecutive "connection refuse".
What is the best way to prevent "RESTART" vector, in case consecutive "connection refuse"?

Thanks.

Vector Config

  sources:
     tet_in:
       type: file
       include:
       - /var/run/cilium/tetragon/tetragon.log
       read_from: "end"

   sinks:
     tet_out2:
       inputs:
       - tet_in

       # Sink to HTTPD
       encoding:
         codec: text
       type: http
       uri: http://[2001:100:100:11::280]:56789

Vector Logs

2026-07-15T22:53:58.847021Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111)
2026-07-15T22:54:15.148924Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: tcp connect error: Connection refused (os error 111) error_type="request_failed" stage="processing"
2026-07-15T22:54:15.148967Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: tcp connect error: Connection refused (os error 111)
2026-07-15T22:54:17.978948Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: Internal log [HTTP error.] is being suppressed to avoid flooding.
2026-07-15T22:54:17.978994Z  WARN sink{component_kind="sink" component_id=tet_out2 component_type=http}:request{request_id=1}: vector::sinks::util::retries: Internal log [Retrying after error.] is being suppressed to avoid flooding.
2026-07-15T22:54:18.758516Z  INFO vector::signal: Signal received. signal="SIGTERM"
2026-07-15T22:54:18.758563Z  INFO vector: Vector is stopping.
2026-07-15T22:54:18.759779Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="tet_out, tet_out2, tet_in" time_remaining="59 seconds left"

[agrovr]

The refused connection is probably not what is restarting Vector. The important lines are:

Signal received. signal="SIGTERM"
Vector is stopping.

Vector's HTTP sink retries failed requests by default, whereas those lines show that something outside Vector asked the process to terminate. In Kubernetes the usual causes are a failing liveness/startup probe, an OOM/resource event, a rollout/eviction, or the lifecycle of the pod that contains the sidecar.

First identify the termination reason for the Vector container (replace the namespace/container name as needed):

kubectl -n <namespace> describe pod tetragon-jxpv2
kubectl -n <namespace> get pod tetragon-jxpv2 -o jsonpath='{range .status.containerStatuses[*]}{.name}{"  restarts="}{.restartCount}{"  reason="}{.lastState.terminated.reason}{"  exit="}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl -n <namespace> logs tetragon-jxpv2 -c <vector-container> --previous

In describe, look especially for Unhealthy, Killing, OOMKilled, eviction, or rollout events. Also inspect the actual pod/DaemonSet manifest for livenessProbe and startupProbe. A liveness probe should test whether Vector itself is alive; it should not fail just because tet_out2 cannot reach its downstream HTTP server. If downstream availability needs to affect pod status, readiness is the safer probe because a readiness failure does not restart the container.

Also check the container command for --require-healthy. Normally a failed sink startup health check is logged and Vector continues, but --require-healthy makes Vector exit when that startup check fails. That would be worth removing (or disabling this sink's startup health check) if the destination is allowed to be temporarily unavailable. It does not appear to explain the later SIGTERM in the log you posted, though.

After the restart cause is fixed, you can make the outage more durable with a disk buffer:

data_dir: /var/lib/vector

sinks:
  tet_out2:
    # existing type, inputs, uri, and encoding
    buffer:
      type: disk
      max_size: 268435488
      when_full: block

Mount data_dir on writable persistent storage. The default memory buffer is lost on a forceful restart; a disk buffer preserves data that has been synced to disk. This mitigates data loss, but it does not remove the external SIGTERM, so I would diagnose the pod events/last termination state first.

References: Vector HTTP sink retries and buffer options, Vector data_dir, and Kubernetes container state/restart diagnosis.