Q&A - Load Testing Vector Aggregator Pipeline

[syedg1]

Question

I am looking for ways to load test the vector aggregator metrics pipeline we have setup (datadog agent > vector aggregator > datadog api). In some production regions, we receive about 200-300M metric data points per minute, so we would like to test the pipeline against a similar load before rollout to ensure that there is no data loss.

Are there any recommended load testing frameworks?

[thomasqueirozb]

Hi @syedg1, we use https://github.com/DataDog/lading for our own performance tests. You should be able to hit this kind of throughput using lading. If you find any issues running Vector this throughput please follow up with an issue. Thanks!

[thomasqueirozb]

@syedg1 maybe it is that Vector is creating payloads too large to be ingested by that datadog endpoint. This definitely looks like a bug as the sink should drop events that are too large before they're sent into the endpoint. For more info on this see this other discussion

What would be the correct metrics to monitor for each of these components of the pipeline?

This is a bit tricky. While debugging I generally follow this guide.

My intuition tells me to look at the sink and to try to debug this by doing one of the following:

1. hacky but maybe faster: having some other sink and to output all metrics to both datadog and to this sink and then stop as soon as you see the "Payload Too Large" error and then try to identify which event is present in your debug sink but didn't make it to datadog.
2. more complex but more reliable: override the endpoint on the datadog sink to a local address and make a local web server to forward all incoming events to datadog. From your custom local web server you'll be able to grab failed events and output them to a file to get a minimally reproducible example

I think we can close this discussion an create a bug issue for this to consolidate all info there

[thomasqueirozb]

While looking at one of my answers to that discussion, I noticed that we also probably emit GotHttpResponse

vector/src/internal_events/http_client.rs

Lines 52 to 60 in d6421e3

	impl<T: HttpBody> InternalEvent for GotHttpResponse<'_, T> {
	fn emit(self) {
	debug!(
	message = "HTTP response.",
	status = %self.response.status(),
	version = ?self.response.version(),
	headers = ?remove_sensitive(self.response.headers()),
	body = %FormatBody(self.response.body()),
	);

Setting the environment variable VECTOR_LOG=debug will likely output the response along with the 413 status code to stdout, meaning that you can easily see what is going on without the complex solutions I mentioned in my previous comment

[syedg1]

@thomasqueirozb Just to clarify the error shared above is actually emitted by the datadog agent, not vector. This has me a bit confused, because I am not seeing any error logs regarding messages sent from datadog agent to vector, or from vector to datadog sink (apart from a couple sporadic warnings here saying that messages will be retried).

However, I am seeing spikes in datadog.agent.point.dropped metric along with 413 errors when I am running the Lading job. I do have end-to-end acknowledgements enabled, so not sure if it's a matter of backpressure on the datadog agent as well.

[thomasqueirozb]

@syedg1 what you described sounds like an issue with either dd agent or lading, but not with Vector since the error occurs before Vector receives the message. If you'd like you can verify this by running vector tap.

The line you pasted in the comment ago references agent source code:

ERROR | (comp/forwarder/defaultforwarder/transaction/transaction.go:431 in internalProcess) | Error code "413 Payload Too Large" received while sending transaction to "https://7-71-2-app.agent.datadoghq.com./intake/": "Payload too big", dropping it

If lading is producing metrics too big to be ingested from the agent then I'd suggest opening an issue in their repo. You could also try reducing the bytes_per_second value

[syedg1]

@thomasqueirozb currently, I am monitoring the vector.component_events_discarded_total metric to see if vector is dropping any events, but not seeing any spikes there. Are there any other metrics I should be checking to confirm that no metrics are being dropped within vector?

Additionally, I noticed that the datadog.agent.point.dropped metric usually spikes under high volume of metrics when end-to-end acknowledgements is enabled. It seems like end-to-end acknowledgements is causing backpressure on the datadog agent. Is this expected? If so, are there any suggestions on how we can reduce the backpressure?