Report disposable performance in JSON #719
Conversation
ben-grande
force-pushed
the
preload-perf-json
branch
from
4dde06a to
0ccf8af
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #719 +/- ##
==========================================
- Coverage 70.55% 70.49% -0.06%
==========================================
Files 61 61
Lines 13616 13616
==========================================
- Hits 9607 9599 -8
- Misses 4009 4017 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ben-grande
force-pushed
the
preload-perf-json
branch
5 times, most recently
from
38bfc4d to
fb8b907
Compare
|
What is the motivation for the format change? Easier parsing on the reporting side? |
Yes, much easier parsing: Simply with "fedora-42-xfce_vm-gui-concurrent-api": {
"extra_id": "",
"name": "vm-gui-concurrent-api",
"gui": true,
"concurrent": true,
"from_dom0": false,
"preload_max": 0,
"non_dispvm": true,
"admin_api": true,
"template": "fedora-42-xfce",
"api_results": {
"iteration": {
"1": {
"dom": 0.018,
"exec": 2.253,
"total": 2.27
},
"2": {
"dom": 0.083,
"exec": 2.152,
"total": 2.235
},
"3": {
"dom": 0.084,
"exec": 2.197,
"total": 2.281
},
"4": {
"dom": 0.084,
"exec": 2.174,
"total": 2.258
},
"5": {
"dom": 0.03,
"exec": 0.161,
"total": 0.192
},
"6": {
"dom": 0.039,
"exec": 0.158,
"total": 0.197
},
"7": {
"dom": 0.097,
"exec": 0.131,
"total": 0.228
},
"8": {
"dom": 0.086,
"exec": 0.116,
"total": 0.202
},
"9": {
"dom": 0.049,
"exec": 0.195,
"total": 0.245
},
"10": {
"dom": 0.048,
"exec": 0.113,
"total": 0.16
},
"11": {
"dom": 0.094,
"exec": 0.098,
"total": 0.192
},
"12": {
"dom": 0.079,
"exec": 0.082,
"total": 0.161
},
"13": {
"dom": 0.039,
"exec": 0.033,
"total": 0.072
},
"14": {
"dom": 0.044,
"exec": 0.12,
"total": 0.164
},
"15": {
"dom": 0.123,
"exec": 0.051,
"total": 0.174
},
"16": {
"dom": 0.119,
"exec": 0.063,
"total": 0.182
}
},
"stage": {
"dom": {
"values": [
0.018,
0.083,
0.084,
0.084,
0.03,
0.039,
0.097,
0.086,
0.049,
0.048,
0.094,
0.079,
0.039,
0.044,
0.123,
0.119
],
"mean": 0.07,
"median": 0.081
},
"exec": {
"values": [
2.253,
2.152,
2.197,
2.174,
0.161,
0.158,
0.131,
0.116,
0.195,
0.113,
0.098,
0.082,
0.033,
0.12,
0.051,
0.063
],
"mean": 0.631,
"median": 0.126
},
"total": {
"values": [
2.27,
2.235,
2.281,
2.258,
0.192,
0.197,
0.228,
0.202,
0.245,
0.16,
0.192,
0.161,
0.072,
0.164,
0.174,
0.182
],
"mean": 0.701,
"median": 0.195
}
}
},
"iterations": 16,
"mean": 0.179,
"total": 2.86
} |
ben-grande
force-pushed
the
preload-perf-json
branch
from
fb8b907 to
bc6c462
Compare
ben-grande
force-pushed
the
preload-perf-json
branch
10 times, most recently
from
52bc5dc to
005069b
Compare
ben-grande
force-pushed
the
preload-perf-json
branch
from
005069b to
685ccf6
Compare
OpenQA test summaryComplete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025082915-4.3&flavor=pull-requests Test run included the following:
New failures, excluding unstableCompared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025081011-4.3&flavor=update
Failed tests8 failures
Fixed failuresCompared to: https://openqa.qubes-os.org/tests/149225#dependencies 72 fixed
Unstable testsPerformance TestsPerformance degradation:15 performance degradations
Remaining performance tests:159 tests
|
|
Test seems fine: https://openqa.qubes-os.org/tests/150719 Maybe add The results will be added to the tests-data on the other PR: https://openqa.qubes-os.org/tests/150719/file/system_tests-perf_test_results.txt Otherwise, it is quite difficult to evaluate with so many tests and data points. |
The data parser may want to compare how the same hardware performed across different dates as well as how different hardware performs the test. The HCL tests have a prefix to not conflict with the properties keys, as they may have the same name, such as "memory", "kernel". For: QubesOS/qubes-issues#1512
ben-grande
force-pushed
the
preload-perf-json
branch
from
685ccf6 to
cc9ccf7
Compare
|
We discussed "today" about making iterations count configurable. And while it may be a good idea in general, it won't be great for CI, as then reports are harder to compare (especially the "total" values will be different, but even "mean" may be impacted by just iterations count, if for example there are some fluctuations at the start/end). |
For: QubesOS/qubes-issues#1512
Downside of JSON is unique toplevel key, works well of OpenQA and Github reporting, but if you want to measure different runs, output to different file is recommended. I have added some keys containing the date of the run as well as hardware info for better comparison.