Continuous Failed to make HTTP(S) request Warnings

[twcchu]

Question

We are getting warnings for Failed to make HTTP(S) request for the datadog_metrics sink every few hours. The buffer is not discarding events or filling up according to the metrics. We are using default sink buffer and batch settings. Are there other metrics we can look at to triage this warning or other settings that will keep the connection from resetting?

Vector Config

sources:
  vector_metrics_filebeat:
    type: internal_metrics
    scrape_interval_secs: 10
sinks:
  datadog_metrics_filebeat:
    type: datadog_metrics
    inputs:
      - metrics_fb_tag
      - logs_count
    default_api_key: ${DATADOG_API_KEY}
    site: datadoghq.com

Vector Logs

{"timestamp":"2025-11-12T15:58:46.820474Z","level":"WARN","message":"HTTP error.","error":"connection error: Connection reset by peer (os error 104)","error_type":"request_failed","stage":"processing","target":"vector::internal_events::http_client","span":{"name":"http"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":310843,"name":"request"},{"name":"http"}]}
{"timestamp":"2025-11-12T15:58:46.820541Z","level":"WARN","message":"Retrying after error.","error":"Failed to make HTTP(S) request: Failed to make HTTP(S) request: connection error: Connection reset by peer (os error 104)","internal_log_rate_limit":true,"target":"vector::sinks::util::retries","span":{"request_id":310843,"name":"request"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":310843,"name":"request"}]}
{"timestamp":"2025-11-12T18:23:03.881544Z","level":"WARN","message":"HTTP error.","error":"connection error: Connection reset by peer (os error 104)","error_type":"request_failed","stage":"processing","target":"vector::internal_events::http_client","span":{"name":"http"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":315970,"name":"request"},{"name":"http"}]}
{"timestamp":"2025-11-12T18:23:03.881628Z","level":"WARN","message":"Retrying after error.","error":"Failed to make HTTP(S) request: Failed to make HTTP(S) request: connection error: Connection reset by peer (os error 104)","internal_log_rate_limit":true,"target":"vector::sinks::util::retries","span":{"request_id":315970,"name":"request"},"spans":[{"component_id":"datadog_metrics_filebeat","component_kind":"sink","component_type":"datadog_metrics","name":"sink"},{"request_id":315970,"name":"request"}]}

[thomasqueirozb]

When running Vector in debug mode you are likely to see Sending HTTP request. logs.

vector/src/internal_events/http_client.rs

Lines 31 to 44 in 5c16191

	impl<T: HttpBody> InternalEvent for AboutToSendHttpRequest<'_, T> {
	fn emit(self) {
	debug!(
	message = "Sending HTTP request.",
	uri = %self.request.uri(),
	method = %self.request.method(),
	version = ?self.request.version(),
	headers = ?remove_sensitive(self.request.headers()),
	body = %FormatBody(self.request.body()),
	);
	counter!("http_client_requests_sent_total", "method" => self.request.method().to_string())
	.increment(1);
	}
	}

For information on controlling log verbosity refer to the docs.

This will show you exactly what is being sent to datadog.

However, I don't think that will tell you anything useful. Connection reset by peer (os error 104) usually indicates a connection error originating from the server, and there is little that can be done. (From what I read online it could also be that your system time is incorrect or maybe even a firewall issue but those would need more investigation. Again, it could be that datadog itself is resetting the connection).

FYI: Under the hood we're using hypers http client. Maybe tuning Vector's verbosity to log hyper's trace/debug logs will show you more information.