Disposable performance: Add drive information to HCL, more details in spec, more tests for line graph #744
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #744 +/- ##
==========================================
- Coverage 70.74% 70.68% -0.06%
==========================================
Files 61 61
Lines 13809 13809
==========================================
- Hits 9769 9761 -8
- Misses 4040 4048 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ben-grande
force-pushed
the
preload-hcl-nvme
branch
from
797cf76 to
6c1206b
Compare
ben-grande
changed the title
tests/dispvm_perf_reader.py
Outdated
| - Global: | ||
| - Date: {} | ||
| - Qubes: {} | ||
| - Xen: {} | ||
| - Global Kernel: {} | ||
| - Template: | ||
| - Name: {} | ||
| - Build time: {} | ||
| - Last update: {} | ||
| - Virtual CPUs: {} | ||
| - Bootstrap memory: {} | ||
| - Maximum memory: {} | ||
| - Kernel: {} | ||
| - Kernel options: {} | ||
| - Hardware: | ||
| - Certified: {} | ||
| - Brand: {} | ||
| - Model: {} | ||
| - CPU: {} | ||
| - RAM: {} MiB | ||
| - BIOS: {} | ||
| - SCSI: {} | ||
| - NVMe: {} |
There was a problem hiding this comment.
That become quite impossible to track, maybe use f-string instead?
There was a problem hiding this comment.
And also, is the intermediate specs dict necessary?
tests/dispvm_perf_reader.py
Outdated
| "hcl-model": data["hcl-model"], | ||
| "hcl-bios": data["hcl-bios"], | ||
| "hcl-cpu": data["hcl-cpu"], | ||
| "hcl-kernel": data["hcl-kernel"], |
There was a problem hiding this comment.
One is the global and the other is the template's kernel.
| TestConfig( | ||
| "dom0-dispvm-preload-5-api", | ||
| preload_max=5, | ||
| admin_api=True, | ||
| extra_id="dom0-dispvm-preload-api", | ||
| from_dom0=True, | ||
| ), | ||
| TestConfig( | ||
| "dom0-dispvm-preload-5-gui-api", | ||
| preload_max=5, | ||
| gui=True, | ||
| admin_api=True, | ||
| extra_id="dom0-dispvm-preload-gui-api", | ||
| from_dom0=True, | ||
| ), | ||
| TestConfig( | ||
| "dom0-dispvm-preload-6-api", | ||
| preload_max=6, | ||
| admin_api=True, | ||
| extra_id="dom0-dispvm-preload-api", | ||
| from_dom0=True, | ||
| ), | ||
| TestConfig( | ||
| "dom0-dispvm-preload-6-gui-api", | ||
| preload_max=6, | ||
| gui=True, | ||
| admin_api=True, | ||
| extra_id="dom0-dispvm-preload-gui-api", | ||
| from_dom0=True, | ||
| ), | ||
| TestConfig( | ||
| "dom0-dispvm-preload-concurrent-api", | ||
| concurrent=True, | ||
| preload_max=MAX_CONCURRENCY, |
There was a problem hiding this comment.
I'm concerned the dispvm_pref job will take even longer. That's no-go. If anything, make it shorter.
There was a problem hiding this comment.
There are 16 concurrent tests, although they are the fastest tests, it doesn't represent user workflows if not leverage by a client built for that. Currently there are no known clients. Securedrop might be one in the future.
There was a problem hiding this comment.
I see you marked some existing tests as optional, but those new for each 1-6 concurrency still execute unconditionally. Does it really make sense to collect this info at every test run? Pick one, or maybe two of them.
There was a problem hiding this comment.
Picked 2 and 4.
- 2 represents what a low end hardware can support
- 4 to get better results hopefully with always preloads ready in queue
ben-grande
force-pushed
the
preload-hcl-nvme
branch
from
c2f6cb5 to
2376284
Compare
|
black complains |
ben-grande
force-pushed
the
preload-hcl-nvme
branch
from
2376284 to
7f1a695
Compare
ben-grande
force-pushed
the
preload-hcl-nvme
branch
2 times, most recently
from
d32bd77 to
d0c8a57
Compare
OpenQA test summaryComplete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025121419-4.3&flavor=pull-requests Test run included the following:
New failures, excluding unstableCompared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025111104-4.3&flavor=update
Failed tests12 failures
Fixed failuresCompared to: https://openqa.qubes-os.org/tests/158999#dependencies 18 fixed
Unstable testsDetailsPerformance TestsPerformance degradation:11 performance degradations
Remaining performance tests:84 tests
|
ben-grande
force-pushed
the
preload-hcl-nvme
branch
2 times, most recently
from
7b495e0 to
4572b75
Compare
|
So, always bothered me that tests are excluded from pylint, and it could have reported the error before being openqaed. I did add some files now, at least the ones I have made a lot of changes too. Some I didn't dare to touch because there were a lot of warnings. |
ben-grande
force-pushed
the
preload-hcl-nvme
branch
2 times, most recently
from
8409998 to
1dcc889
Compare
ben-grande
force-pushed
the
preload-hcl-nvme
branch
2 times, most recently
from
e11d88d to
c0dbeb1
Compare
|
Will finish this tomorrow. |
ben-grande
force-pushed
the
preload-hcl-nvme
branch
3 times, most recently
from
6d976b7 to
5c352a7
Compare
|
Disposable performance tests took 55 minutes. It failed on the graph generator. Using the same results the run 162195, we can get these graphs: Details
|
|
OpenQA this please. |
|
openQArun LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf,system_tests_network,system_tests_network_ipv6,system_tests_network_updates,system_tests_gui_interactive,system_tests_gui_interactive_preloaded,system_tests_backupdispvm |
1 similar comment
|
openQArun LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf,system_tests_network,system_tests_network_ipv6,system_tests_network_updates,system_tests_gui_interactive,system_tests_gui_interactive_preloaded,system_tests_backupdispvm |
Disk information is already being reported for SCSI, but modern laptops may not have ATA and only have PCIe for NVMe. Adding this information to disposable performance report may help understand the execution speed. For: QubesOS/qubes-issues#1512
Also add a common prefix so files for different templates are sorted together on OpenQA. For: QubesOS/qubes-issues#1512
Avoid syntactically wrong tests from reaching OpenQA.
ben-grande
force-pushed
the
preload-hcl-nvme
branch
from
5c352a7 to
e9c2c85
Compare
ben-grande
force-pushed
the
preload-hcl-nvme
branch
from
e9c2c85 to
a3c02a3
Compare
|
|
Closing as superseded by #760 (comment) |
3 participants
Disk information is already being reported for SCSI, but modern laptops may not have ATA and only have PCIe for NVMe.
Adding this information to disposable performance report may help understand the execution speed.
For: QubesOS/qubes-issues#1512
i3 CPU with 16GB RAM is surprising that it could run the full performance test suite. Old disk that had many reads and writes can impact, but not sure it should be in the HCL.
The decision to add more tests to line graph is that 1 preloaded disposable is not being tested as well as no varied amount of GUI preloaded disposable are being tested for the line graph. Hopefully, this will allow an examination closer to user workflows.