Q&A configuration size & complexity

[janweinkauff]

Question

I’m currently working on a project that aims to route telemetry data from a large number of tenants to a high number of distinct sinks, ranging from thousands to tens of thousands and beyond.

We’ve begun benchmarking Vector’s behavior with large configurations. Initial results using single configuration files with 10,000 components show startup times (for both vector and vector validate) between 20 seconds (for ~100 components) and 2800 seconds (for ~10,000 components).

Now we’re trying to better understand the sensible limits for configuration files and how to scale Vector effectively in such scenarios. It seems we’ll need to find a strategy for sharding configurations across many Vector instances.

Some questions we’d love feedback on:

• Does it make sense to create configuration files with more than 100, 1,000 or 10,000 components?
• What are your experiences or best practices for splitting large configurations into smaller, manageable pieces?
• Have you conducted any benchmarks to test the impact of configuration file size and complexity on Vector’s performance?

Vector Config

No response

Vector Logs

No response