Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves the binary search algorithm used in apply-load benchmarks by implementing a statistically-driven approach using t-statistics. The changes ensure more reliable performance measurements by adapting sample sizes based on confidence levels, while requiring a minimum of 30 samples for statistical validity. Additionally, it fixes issues with SAC max TPS benchmarking where the math was incorrectly handling the granularity change from 1s to 50ms.
Changes:
- Implemented new
noisyBinarySearchfunction that uses t-statistics and adaptive sampling with configurable confidence levels - Refactored benchmark functions to return single-ledger timing measurements for use with the new binary search algorithm
- Updated minimum ledger count requirement from >0 to ≥30 and removed the TARGET_CLOSE_TIME_STEP_MS validation constraint
- Added comprehensive test coverage for the noisy binary search algorithm with multiple variance and function types
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/simulation/ApplyLoad.cpp | Implements the new noisy binary search algorithm and refactors findMaxLimitsForModelTransaction and findMaxSacTps to use it |
| src/simulation/ApplyLoad.h | Adds function declarations for the new benchmark functions and exports noisyBinarySearch for testing |
| src/simulation/test/LoadGeneratorTests.cpp | Adds extensive test coverage for the noisy binary search with various scenarios and updates existing test configurations |
| src/main/Config.cpp | Removes the validation requiring APPLY_LOAD_TARGET_CLOSE_TIME_MS to be a multiple of TARGET_CLOSE_TIME_STEP_MS |
| src/main/CommandLine.cpp | Updates validation to require at least 30 ledgers and changes MIN_TPS comparison from >= to > |
| Builds/VisualStudio/stellar-core.vcxproj | Updates Visual Studio project file to include previously missing source files |
| Builds/VisualStudio/stellar-core.vcxproj.filters | Updates Visual Studio project filters to include previously missing source files |
The binary search uses t-statistic to ensure the necessary confidence at each step. This results in needing less samples far away from the true value and more samples around it. We still need at least 30 samples though to keep math simple. This works reasonably well, but still doesn't avoid some fundamental variance issues that we have that result in very different and statistically significant performance difference between different runs. That concern should hopefully be addressed separately. Also fix some SAC max TPS benchmark issues, the math got messed up after I've updated it from 1s granularity to 50ms.
18c5782 to
cea4539
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The binary search uses t-statistic to ensure the necessary confidence at each step. This results in needing less samples far away from the true value and more samples around it. We still need at least 30 samples though to keep math simple. This works reasonably well, but still doesn't avoid some fundamental variance issues that we have that result in very different and statistically significant performance difference between different runs. That concern should hopefully be addressed separately.
Also fix some SAC max TPS benchmark issues, the math got messed up after I've updated it from 1s granularity to 50ms.
Checklist
clang-formatv8.0.0 (viamake formator the Visual Studio extension)