-
Notifications
You must be signed in to change notification settings - Fork 476
test(profiling): unflake some tests #15797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
22ea28e to
fc07442
Compare
304ff10 to
3113856
Compare
Codeowners resolved as |
Performance SLOsComparing candidate kowalski/test-profiling-unflake-some-tests (1afc46c) with baseline main (0e20665) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 17.969µs (SLO: <20.000µs 📉 -10.2%) vs baseline: 📈 +21.0% Memory: ✅ 42.664MB (SLO: <43.250MB 🟡 -1.4%) vs baseline: +5.1% ✅ add_inplace_aspectTime: ✅ 14.901µs (SLO: <20.000µs 📉 -25.5%) vs baseline: -0.4% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.0% ✅ add_inplace_noaspectTime: ✅ 0.340µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +0.8% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ add_noaspectTime: ✅ 0.551µs (SLO: <10.000µs 📉 -94.5%) vs baseline: +0.9% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ bytearray_aspectTime: ✅ 17.914µs (SLO: <30.000µs 📉 -40.3%) vs baseline: -0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ bytearray_extend_aspectTime: ✅ 23.647µs (SLO: <30.000µs 📉 -21.2%) vs baseline: -0.6% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ bytearray_extend_noaspectTime: ✅ 2.749µs (SLO: <10.000µs 📉 -72.5%) vs baseline: -0.1% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ bytearray_noaspectTime: ✅ 1.453µs (SLO: <10.000µs 📉 -85.5%) vs baseline: -0.3% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.2% ✅ bytes_aspectTime: ✅ 16.550µs (SLO: <20.000µs 📉 -17.3%) vs baseline: -0.8% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ bytes_noaspectTime: ✅ 1.415µs (SLO: <10.000µs 📉 -85.8%) vs baseline: -1.3% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ bytesio_aspectTime: ✅ 55.292µs (SLO: <70.000µs 📉 -21.0%) vs baseline: -0.8% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ bytesio_noaspectTime: ✅ 3.282µs (SLO: <10.000µs 📉 -67.2%) vs baseline: +0.2% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.0% ✅ capitalize_aspectTime: ✅ 14.567µs (SLO: <20.000µs 📉 -27.2%) vs baseline: -1.0% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.8% ✅ capitalize_noaspectTime: ✅ 2.543µs (SLO: <10.000µs 📉 -74.6%) vs baseline: -0.9% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ casefold_aspectTime: ✅ 14.600µs (SLO: <20.000µs 📉 -27.0%) vs baseline: -0.6% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ casefold_noaspectTime: ✅ 3.162µs (SLO: <10.000µs 📉 -68.4%) vs baseline: ~same Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.7% ✅ decode_aspectTime: ✅ 15.554µs (SLO: <30.000µs 📉 -48.2%) vs baseline: -0.5% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ decode_noaspectTime: ✅ 1.593µs (SLO: <10.000µs 📉 -84.1%) vs baseline: -1.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ encode_aspectTime: ✅ 17.977µs (SLO: <30.000µs 📉 -40.1%) vs baseline: 📈 +21.4% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ encode_noaspectTime: ✅ 1.443µs (SLO: <10.000µs 📉 -85.6%) vs baseline: -4.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ format_aspectTime: ✅ 171.352µs (SLO: <200.000µs 📉 -14.3%) vs baseline: ~same Memory: ✅ 42.802MB (SLO: <43.250MB 🟡 -1.0%) vs baseline: +5.3% ✅ format_map_aspectTime: ✅ 191.183µs (SLO: <200.000µs -4.4%) vs baseline: +0.1% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +5.1% ✅ format_map_noaspectTime: ✅ 3.816µs (SLO: <10.000µs 📉 -61.8%) vs baseline: +0.9% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ format_noaspectTime: ✅ 3.113µs (SLO: <10.000µs 📉 -68.9%) vs baseline: -1.0% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ index_aspectTime: ✅ 15.208µs (SLO: <20.000µs 📉 -24.0%) vs baseline: -0.6% Memory: ✅ 42.703MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.1% ✅ index_noaspectTime: ✅ 0.464µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.5% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +4.8% ✅ join_aspectTime: ✅ 16.926µs (SLO: <20.000µs 📉 -15.4%) vs baseline: -1.2% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ join_noaspectTime: ✅ 1.542µs (SLO: <10.000µs 📉 -84.6%) vs baseline: -0.3% Memory: ✅ 42.605MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +4.9% ✅ ljust_aspectTime: ✅ 20.701µs (SLO: <30.000µs 📉 -31.0%) vs baseline: -0.7% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +4.5% ✅ ljust_noaspectTime: ✅ 2.654µs (SLO: <10.000µs 📉 -73.5%) vs baseline: -2.4% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.1% ✅ lower_aspectTime: ✅ 17.766µs (SLO: <30.000µs 📉 -40.8%) vs baseline: -1.8% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ lower_noaspectTime: ✅ 2.406µs (SLO: <10.000µs 📉 -75.9%) vs baseline: -0.3% Memory: ✅ 42.585MB (SLO: <43.250MB 🟡 -1.5%) vs baseline: +5.0% ✅ lstrip_aspectTime: ✅ 17.586µs (SLO: <30.000µs 📉 -41.4%) vs baseline: -0.5% Memory: ✅ 42.684MB (SLO: <43.250MB 🟡 -1.3%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 1.847µs (SLO: <10.000µs 📉 -81.5%) vs baseline: -0.4% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.7% ✅ modulo_aspectTime: ✅ 166.424µs (SLO: <200.000µs 📉 -16.8%) vs baseline: -0.2% Memory: ✅ 42.802MB (SLO: <43.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 180.138µs (SLO: <200.000µs -9.9%) vs baseline: +3.2% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ modulo_aspect_for_bytesTime: ✅ 168.623µs (SLO: <200.000µs 📉 -15.7%) vs baseline: -0.3% Memory: ✅ 42.861MB (SLO: <43.500MB 🟡 -1.5%) vs baseline: +4.9% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 171.765µs (SLO: <200.000µs 📉 -14.1%) vs baseline: ~same Memory: ✅ 42.782MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ modulo_noaspectTime: ✅ 3.646µs (SLO: <10.000µs 📉 -63.5%) vs baseline: -1.5% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ replace_aspectTime: ✅ 211.435µs (SLO: <300.000µs 📉 -29.5%) vs baseline: ~same Memory: ✅ 42.625MB (SLO: <44.000MB -3.1%) vs baseline: +4.8% ✅ replace_noaspectTime: ✅ 2.934µs (SLO: <10.000µs 📉 -70.7%) vs baseline: +0.8% Memory: ✅ 42.703MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ repr_aspectTime: ✅ 1.431µs (SLO: <10.000µs 📉 -85.7%) vs baseline: +0.7% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ repr_noaspectTime: ✅ 0.525µs (SLO: <10.000µs 📉 -94.7%) vs baseline: ~same Memory: ✅ 42.743MB (SLO: <43.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ rstrip_aspectTime: ✅ 19.019µs (SLO: <30.000µs 📉 -36.6%) vs baseline: +0.1% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 1.923µs (SLO: <10.000µs 📉 -80.8%) vs baseline: -0.2% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ slice_aspectTime: ✅ 15.760µs (SLO: <20.000µs 📉 -21.2%) vs baseline: -1.1% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ slice_noaspectTime: ✅ 0.601µs (SLO: <10.000µs 📉 -94.0%) vs baseline: +0.4% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ stringio_aspectTime: ✅ 53.889µs (SLO: <80.000µs 📉 -32.6%) vs baseline: -1.0% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 3.615µs (SLO: <10.000µs 📉 -63.8%) vs baseline: ~same Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +4.8% ✅ strip_aspectTime: ✅ 17.452µs (SLO: <20.000µs 📉 -12.7%) vs baseline: -1.6% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ strip_noaspectTime: ✅ 1.841µs (SLO: <10.000µs 📉 -81.6%) vs baseline: -1.6% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.3% ✅ swapcase_aspectTime: ✅ 18.393µs (SLO: <30.000µs 📉 -38.7%) vs baseline: -1.4% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ swapcase_noaspectTime: ✅ 2.811µs (SLO: <10.000µs 📉 -71.9%) vs baseline: +0.8% Memory: ✅ 42.684MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ title_aspectTime: ✅ 18.141µs (SLO: <30.000µs 📉 -39.5%) vs baseline: -1.1% Memory: ✅ 42.684MB (SLO: <43.000MB 🟡 -0.7%) vs baseline: +5.1% ✅ title_noaspectTime: ✅ 2.620µs (SLO: <10.000µs 📉 -73.8%) vs baseline: -1.9% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ translate_aspectTime: ✅ 24.138µs (SLO: <30.000µs 📉 -19.5%) vs baseline: 📈 +16.8% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ translate_noaspectTime: ✅ 4.289µs (SLO: <10.000µs 📉 -57.1%) vs baseline: -0.7% Memory: ✅ 42.644MB (SLO: <43.500MB 🟡 -2.0%) vs baseline: +5.1% ✅ upper_aspectTime: ✅ 17.744µs (SLO: <30.000µs 📉 -40.9%) vs baseline: -1.7% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ upper_noaspectTime: ✅ 2.400µs (SLO: <10.000µs 📉 -76.0%) vs baseline: -0.9% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.125µs (SLO: <10.000µs 📉 -48.7%) vs baseline: 📈 +19.5% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +5.0% ✅ ospathbasename_noaspectTime: ✅ 4.236µs (SLO: <10.000µs 📉 -57.6%) vs baseline: -2.4% Memory: ✅ 42.566MB (SLO: <43.500MB -2.1%) vs baseline: +4.8% ✅ ospathjoin_aspectTime: ✅ 6.217µs (SLO: <10.000µs 📉 -37.8%) vs baseline: -1.6% Memory: ✅ 42.723MB (SLO: <43.500MB 🟡 -1.8%) vs baseline: +5.3% ✅ ospathjoin_noaspectTime: ✅ 6.354µs (SLO: <10.000µs 📉 -36.5%) vs baseline: -0.2% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.1% ✅ ospathnormcase_aspectTime: ✅ 3.544µs (SLO: <10.000µs 📉 -64.6%) vs baseline: -1.2% Memory: ✅ 42.605MB (SLO: <43.500MB -2.1%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 3.581µs (SLO: <10.000µs 📉 -64.2%) vs baseline: -2.8% Memory: ✅ 42.585MB (SLO: <43.500MB -2.1%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.912µs (SLO: <10.000µs 📉 -50.9%) vs baseline: -0.7% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 4.996µs (SLO: <10.000µs 📉 -50.0%) vs baseline: -0.1% Memory: ✅ 42.625MB (SLO: <43.500MB -2.0%) vs baseline: +5.2% ✅ ospathsplitdrive_aspectTime: ✅ 3.776µs (SLO: <10.000µs 📉 -62.2%) vs baseline: ~same Memory: ✅ 42.507MB (SLO: <43.500MB -2.3%) vs baseline: +4.8% ✅ ospathsplitdrive_noaspectTime: ✅ 0.741µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.8% Memory: ✅ 42.664MB (SLO: <43.500MB 🟡 -1.9%) vs baseline: +5.3% ✅ ospathsplitext_aspectTime: ✅ 4.605µs (SLO: <10.000µs 📉 -54.0%) vs baseline: -0.9% Memory: ✅ 42.546MB (SLO: <43.500MB -2.2%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 4.582µs (SLO: <10.000µs 📉 -54.2%) vs baseline: -1.9% Memory: ✅ 42.526MB (SLO: <43.500MB -2.2%) vs baseline: +5.0% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.411µs (SLO: <20.000µs 📉 -82.9%) vs baseline: 📈 +14.6% Memory: ✅ 34.957MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +5.2% ✅ 1-count-metrics-100-timesTime: ✅ 199.813µs (SLO: <220.000µs -9.2%) vs baseline: +0.3% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.8% ✅ 1-distribution-metric-1-timesTime: ✅ 3.360µs (SLO: <20.000µs 📉 -83.2%) vs baseline: +0.6% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +4.7% ✅ 1-distribution-metrics-100-timesTime: ✅ 214.073µs (SLO: <230.000µs -6.9%) vs baseline: -0.4% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.163µs (SLO: <20.000µs 📉 -89.2%) vs baseline: -0.7% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.5% ✅ 1-gauge-metrics-100-timesTime: ✅ 136.462µs (SLO: <150.000µs -9.0%) vs baseline: -0.3% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +4.9% ✅ 1-rate-metric-1-timesTime: ✅ 3.169µs (SLO: <20.000µs 📉 -84.2%) vs baseline: +0.9% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.3% ✅ 1-rate-metrics-100-timesTime: ✅ 213.733µs (SLO: <250.000µs 📉 -14.5%) vs baseline: +0.7% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.6% ✅ 100-count-metrics-100-timesTime: ✅ 19.990ms (SLO: <22.000ms -9.1%) vs baseline: +0.5% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +4.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.210ms (SLO: <2.550ms 📉 -13.3%) vs baseline: -1.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.410ms (SLO: <1.550ms -9.0%) vs baseline: +1.1% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.0% ✅ 100-rate-metrics-100-timesTime: ✅ 2.175ms (SLO: <2.550ms 📉 -14.7%) vs baseline: -0.3% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +4.8% ✅ flush-1-metricTime: ✅ 4.506µs (SLO: <20.000µs 📉 -77.5%) vs baseline: -1.5% Memory: ✅ 35.291MB (SLO: <35.500MB 🟡 -0.6%) vs baseline: +5.2% ✅ flush-100-metricsTime: ✅ 173.922µs (SLO: <250.000µs 📉 -30.4%) vs baseline: ~same Memory: ✅ 35.330MB (SLO: <35.500MB 🟡 -0.5%) vs baseline: +5.3% ✅ flush-1000-metricsTime: ✅ 2.172ms (SLO: <2.500ms 📉 -13.1%) vs baseline: -0.2% Memory: ✅ 36.117MB (SLO: <36.500MB 🟡 -1.0%) vs baseline: +5.4% 🟡 Near SLO Breach (16 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
## Description This improves typing in a test file. Related - #15797
3113856 to
5472e43
Compare
## Description This improves typing in a test file. Related - #15797
Can you please expand why? Also, great variable naming, I had to look it up: penultimate: last but one in a series of things; second last. Unsure why it helps the test tho, lol. |
|
@vlad-scherbich addressed your comments, please take another look 🙇 |
vlad-scherbich
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
99e90eb to
1afc46c
Compare

Description
This makes some tests less flaky. Previously, we would sometimes parse an empty Profile (0 samples in it) and make the test fail. The fix is to, when this happens, ignore the failure and parse/use the previous Profile instead (which we expect to ALWAYS be non-empty).
Related