Add stress testing framework, with basic metrics example to demonstrate.#3241
Add stress testing framework, with basic metrics example to demonstrate.#3241lalitb wants to merge 18 commits intoopen-telemetry:mainfrom
Conversation
[pull] main from open-telemetry:main
✅ Deploy Preview for opentelemetry-cpp-api-docs canceled.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3241 +/- ##
==========================================
- Coverage 90.06% 90.03% -0.02%
==========================================
Files 220 220
Lines 7069 7069
==========================================
- Hits 6366 6364 -2
- Misses 703 705 +2 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a multi-threaded stress testing framework and provides a basic OpenTelemetry metrics example to demonstrate its usage under high-concurrency workloads.
- Adds a reusable C++ stress test library with throughput monitoring
- Implements a metrics stress test example leveraging OpenTelemetry
- Integrates the stress tests into the CMake build
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| stress/common/stress.h | Declares the Stress class and supporting data structures |
| stress/common/stress.cc | Implements thread spawning, monitoring, and graceful shutdown |
| stress/metrics/metrics.cc | Adds a sample metrics stress test using the new framework |
| stress/common/CMakeLists.txt | Builds stress as a static library |
| stress/metrics/CMakeLists.txt | Builds stress_metrics executable and links necessary targets |
| CMakeLists.txt | Hooks the stress directory into the main project build |
Comments suppressed due to low confidence (1)
stress/common/stress.h:72
- The new Stress framework lacks associated unit tests to validate its behavior. Consider adding tests to cover key functionality.
class Stress
| std::vector<std::thread> threads_; // Vector to hold worker threads | ||
| std::vector<WorkerStats> stats_; // Vector to hold statistics for each thread | ||
| const size_t numThreads_; // Number of threads to run | ||
| std::atomic<bool> stopFlag_{false}; // signal to stop the test |
There was a problem hiding this comment.
The member variable stopFlag_ is never used, as the global STOP flag is used instead. Remove stopFlag_ or integrate it into the control flow.
| std::atomic<bool> stopFlag_{false}; // signal to stop the test | |
| // Removed unused stopFlag_ member variable |
| // Global flags | ||
| std::atomic<bool> STOP( | ||
| false); // Global flag to stop the stress test when signaled (e.g., via Ctrl+C) | ||
| std::atomic<bool> READY(false); // Global flag to synchronize thread start |
There was a problem hiding this comment.
The READY flag is declared but never used. Either remove it or use it to synchronize thread start.
| void Stress::monitorThroughput() | ||
| { | ||
| uint64_t lastTotalCount = 0; | ||
| auto lastTime = std::chrono::steady_clock::now(); |
There was a problem hiding this comment.
throughputHistory grows without bound in long-running tests, potentially causing high memory usage. Consider capping its size or computing rolling statistics without storing all entries.
| auto lastTime = std::chrono::steady_clock::now(); | |
| auto lastTime = std::chrono::steady_clock::now(); | |
| const size_t MAX_HISTORY_SIZE = 100; // Maximum number of entries in throughputHistory |
| return attributes_set; | ||
| } | ||
|
|
||
| void InitMetrics(const std::string /*&name*/) |
There was a problem hiding this comment.
The parameter 'name' is unused in InitMetrics. Either remove it or use it to parameterize the meter provider.
| void InitMetrics(const std::string /*&name*/) | |
| void InitMetrics(const std::string &name) |
| { | ||
| std::srand(static_cast<unsigned int>(std::time(nullptr))); // Seed the random number generator | ||
| // Pre-generate a set of random attributes | ||
| size_t attribute_count = 1000; // Number of attribute sets to pre-generate |
There was a problem hiding this comment.
[nitpick] Magic number 1000 used for attribute_count; consider making it a configurable constant or command-line parameter.
| size_t attribute_count = 1000; // Number of attribute sets to pre-generate | |
| size_t attribute_count = kDefaultAttributeCount; // Number of attribute sets to pre-generate |
| uint64_t throughput = currentCount / elapsed; | ||
| throughputHistory.push_back(throughput); | ||
|
|
||
| double avg = 0; |
There was a problem hiding this comment.
avg can be of type uint64_t
| { | ||
| uint64_t lastTotalCount = 0; | ||
| auto lastTime = std::chrono::steady_clock::now(); | ||
| std::vector<uint64_t> throughputHistory; |
There was a problem hiding this comment.
seems not required, a history_counter is enough
| std::this_thread::sleep_for(std::chrono::seconds(SLIDING_WINDOW_SIZE)); | ||
|
|
||
| auto currentTime = std::chrono::steady_clock::now(); | ||
| auto elapsed = std::chrono::duration_cast<std::chrono::seconds>(currentTime - lastTime).count(); |
There was a problem hiding this comment.
seconds could be a bit inaccurate
| CPU_SET(threadIndex % std::thread::hardware_concurrency(), &cpuset); | ||
| pthread_setaffinity_np(pthread_self(), sizeof(cpuset), &cpuset); | ||
| #endif | ||
|
|
There was a problem hiding this comment.
is the READY flag meant to start workers at the same time? Or is it superfluous as copilot mentioned?
3 participants
Changes
This PR adds a basic stress testing framework to validate the scalability and reliability of the functionality under high-concurrency and long-running workloads. Unlike Google Benchmark, which focuses on micro-benchmarking and latency measurements for isolated operations, this framework tries to simulate sustained, multi-threaded workloads to test a given workload. The idea is to complement the existing benchmarks by adding stress-tests to addressing long-duration and high-concurrency use-cases.
This is already implemented for .Net and Rust, and most of the ideas are taken from there. I felt the need for this to test some optimizations I am doing for metrics, but feel to comment if this doesn't seem helpful.
Also added a basic stress-testing example for metrics to demonstrate. Below are the results from the metrics stress test as an example:
It’s still in the early stages and will need further enhancements but should be a good starting point. Future improvements could include adding memory and CPU usage information alongside the existing throughput, as well as refining the initial warm-up period to sustain consistent data collection.
Implementation Details:
Worker Threads:
- The worker threads (default to number of cores) are spawned to execute the workload.
- Each worker thread executes the workload function (func) in a loop until a global STOP flag is set. (ctrl-c)
- Each thread maintains its own iteration count to minimize contention.
Throughput Monitoring:
- A separate controller thread monitors throughput by periodically summing up iteration counts across threads.
- Throughput is calculated over a sliding window (SLIDING_WINDOW_SIZE) and displayed dynamically.
Final Summary:
- At the end of the test, the program calculates and prints the total iterations, duration, and average throughput.
For significant contributions please make sure you have completed the following items:
CHANGELOG.mdupdated for non-trivial changes