Skip to content

Conversation

ben-grande
Copy link
Contributor

for key, value in data.items():
name = value["name"]
mean = value["mean"]
key = name + "(mean:" + mean + ")"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TypeError: can only concatenate str (not "float") to str

name = value["name"]
mean = value["mean"]
key = name + "(mean:" + mean + ")"
result[key] = float(value)
Copy link
Member

@marmarek marmarek Aug 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here value is a dict - float() doesn't work on it

@marmarek
Copy link
Member

And finally, existing past jobs have the old format and the script loads those too for comparison - dropping old format support breaks that.

@marmarek
Copy link
Member

marmarek commented Aug 26, 2025

BTW, you can run the script yourself, it uses public API (except actually posting the comment, but with --show-results-only option it's skipped).
For example for recent PR jobs, comparing to earlier updates job the call would be:

python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --latest --build 2025082613-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only

@ben-grande
Copy link
Contributor Author

For example for recent PR jobs, comparing to earlier updates job the call would be:

Yes, I never actually run this as I didn't knew, I should have asked for help/instructions.

Is the command known to take long (more than 5 min)? I am using verbose and there are network connections for the first minute and then it hangs. Maybe there is something happening but not being logged.

@marmarek
Copy link
Member

It does take a while with the --instability option, but should be quicker without it. But still, it makes quite a few API calls (all results for jobs in at least two test runs).

@ben-grande
Copy link
Contributor Author

I didn't get a report from the public API yet (15 minutes hanging) but did commit on a best effort.

@ben-grande
Copy link
Contributor Author

~/openqa-tests-qubesos$ time python3 utils/github_reporting.py --package-list utils/github_package_mapping.json --latest --build 2025082613-4.3 --flavor pull-requests --version 4.3 --compare-to-build 2025081011-4.3 --show-results-only --verbose
2025-08-27 13:59:17,364 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:19,631 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs?build=2025082613-4.3&version=4.3&flavor=pull-requests HTTP/1.1" 200 11377
2025-08-27 13:59:20,454 root: getting job 150652
2025-08-27 13:59:20,487 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:22,982 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150652/details HTTP/1.1" 200 21376
2025-08-27 13:59:23,593 root: creating orphan job for 150652
2025-08-27 13:59:23,606 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:25,099 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150652/details HTTP/1.1" 200 21376
2025-08-27 13:59:25,510 root: Flushing orphan_job 150652 to the database
2025-08-27 13:59:25,703 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:29,004 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150653/details HTTP/1.1" 200 10895
2025-08-27 13:59:29,514 root: Flushing child_job 150653 to the database
2025-08-27 13:59:29,581 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:30,350 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150654/details HTTP/1.1" 200 64710
2025-08-27 13:59:32,111 root: Flushing child_job 150654 to the database
2025-08-27 13:59:32,195 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:34,376 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150655/details HTTP/1.1" 200 12963
2025-08-27 13:59:36,393 root: Flushing child_job 150655 to the database
2025-08-27 13:59:36,425 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:37,549 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150656/details HTTP/1.1" 200 85498
2025-08-27 13:59:50,956 root: Flushing child_job 150656 to the database
2025-08-27 13:59:50,991 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:53,111 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150657/details HTTP/1.1" 200 79799
2025-08-27 13:59:58,239 root: Flushing child_job 150657 to the database
2025-08-27 13:59:58,273 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 13:59:59,564 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150658/details HTTP/1.1" 200 61593
2025-08-27 14:00:03,363 root: Flushing child_job 150658 to the database
2025-08-27 14:00:03,402 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:06,140 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150659/details HTTP/1.1" 200 19809
2025-08-27 14:00:06,419 root: Flushing child_job 150659 to the database
2025-08-27 14:00:06,449 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:08,184 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150660/details HTTP/1.1" 200 88471
2025-08-27 14:00:18,305 root: Flushing child_job 150660 to the database
2025-08-27 14:00:18,340 urllib3.connectionpool: Starting new HTTPS connection (1): openqa.qubes-os.org:443
2025-08-27 14:00:19,018 urllib3.connectionpool: https://openqa.qubes-os.org:443 "GET /api/v1/jobs/150661/details HTTP/1.1" 200 76813

@marmarek
Copy link
Member

After making all those API calls, it should finish relatively quickly...

value = data[1]
data_dict = dict(v.split("=", maxsplit=1) for v in data[2:])
key = name + "(avg:" + data_dict["average"] + ")"
result[key] = float(total)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

total is not set in this scope... value?

@ben-grande ben-grande force-pushed the preload-perf-json branch 3 times, most recently from 4feb3e6 to b37303b Compare August 27, 2025 14:50
@ben-grande
Copy link
Contributor Author

  • debian-13-xfce_dispvm (mean:6.428): 102.84
  • debian-13-xfce_dispvm-gui (mean:7.359): 117.75 🟢 ( previous job: 119.40, improvement: 98.62%)
  • debian-13-xfce_dispvm-concurrent (mean:3.09): 49.44
  • debian-13-xfce_dispvm-gui-concurrent (mean:3.917): 62.68 🟢 ( previous job: 64.59, improvement: 97.03%)
  • debian-13-xfce_dispvm-dom0 (mean:7.078): 113.25
  • debian-13-xfce_dispvm-dom0-gui (mean:8.054): 128.87 🔺 ( previous job: 127.44, degradation: 101.12%)
  • debian-13-xfce_dispvm-dom0-concurrent (mean:3.384): 54.15
  • debian-13-xfce_dispvm-dom0-gui-concurrent (mean:4.147): 66.36 🔺 ( previous job: 65.60, degradation: 101.16%)
  • debian-13-xfce_dispvm-preload (mean:3.73): 59.68
  • debian-13-xfce_dispvm-preload-gui (mean:4.607): 73.71
  • debian-13-xfce_dispvm-preload-concurrent (mean:2.85): 45.60
  • debian-13-xfce_dispvm-preload-gui-concurrent (mean:3.806): 60.89
  • debian-13-xfce_dispvm-preload-dom0 (mean:3.84): 61.43
  • debian-13-xfce_dispvm-preload-dom0-gui (mean:5.08): 81.29
  • debian-13-xfce_dispvm-preload-dom0-concurrent (mean:3.279): 52.46
  • debian-13-xfce_dispvm-preload-dom0-gui-concurrent (mean:4.037): 64.59
  • debian-13-xfce_dispvm-api (mean:6.876): 110.01
  • debian-13-xfce_dispvm-gui-api (mean:7.965): 127.44 🟢 ( previous job: 127.48, improvement: 99.97%)
  • debian-13-xfce_dispvm-concurrent-api (mean:3.335): 53.36
  • debian-13-xfce_dispvm-gui-concurrent-api (mean:4.171): 66.74 🔺 ( previous job: 65.39, degradation: 102.06%)
  • debian-13-xfce_dispvm-preload-api (mean:3.868): 61.89
  • debian-13-xfce_dispvm-preload-less-api (mean:6.414): 102.62
  • debian-13-xfce_dispvm-preload-more-api (mean:3.938): 63.00
  • debian-13-xfce_dispvm-preload-gui-api (mean:5.488): 87.81
  • debian-13-xfce_dispvm-preload-concurrent-api (mean:3.172): 50.75
  • debian-13-xfce_dispvm-preload-gui-concurrent-api (mean:3.995): 63.92
  • debian-13-xfce_vm (mean:0.043): 0.69
  • debian-13-xfce_vm-gui (mean:0.04): 0.65 🟢 ( previous job: 7.40, improvement: 8.72%)
  • debian-13-xfce_vm-concurrent (mean:0.028): 0.45
  • debian-13-xfce_vm-gui-concurrent (mean:0.026): 0.41 🟢 ( previous job: 7.33, improvement: 5.65%)
  • debian-13-xfce_vm-api (mean:0.038): 0.60
  • debian-13-xfce_vm-gui-api (mean:0.045): 0.72 🟢 ( previous job: 2.17, improvement: 33.26%)
  • debian-13-xfce_vm-concurrent-api (mean:0.029): 0.47
  • debian-13-xfce_vm-gui-concurrent-api (mean:0.032): 0.51 🟢 ( previous job: 1.82, improvement: 27.86%)
  • fedora-42-xfce_dispvm (mean:7.202): 115.24 🔺 ( previous job: 111.99, degradation: 102.90%)
  • fedora-42-xfce_dispvm-gui (mean:8.523): 136.36 🔺 ( previous job: 131.63, degradation: 103.60%)
  • fedora-42-xfce_dispvm-concurrent (mean:3.681): 58.90 🔺 ( previous job: 57.25, degradation: 102.87%)
  • fedora-42-xfce_dispvm-gui-concurrent (mean:4.418): 70.69 🟢 ( previous job: 74.72, improvement: 94.60%)
  • fedora-42-xfce_dispvm-dom0 (mean:7.76): 124.16 🟢 ( previous job: 124.92, improvement: 99.39%)
  • fedora-42-xfce_dispvm-dom0-gui (mean:8.862): 141.79 🟢 ( previous job: 147.17, improvement: 96.35%)
  • fedora-42-xfce_dispvm-dom0-concurrent (mean:3.812): 60.98 🟢 ( previous job: 64.09, improvement: 95.16%)
  • fedora-42-xfce_dispvm-dom0-gui-concurrent (mean:4.669): 74.70 🟢 ( previous job: 75.59, improvement: 98.82%)
  • fedora-42-xfce_dispvm-preload (mean:4.35): 69.59 🟢 ( previous job: 69.72, improvement: 99.81%)
  • fedora-42-xfce_dispvm-preload-gui (mean:4.951): 79.21 🔺 ( previous job: 79.21, degradation: 100.01%)
  • fedora-42-xfce_dispvm-preload-concurrent (mean:3.027): 48.43 🟢 ( previous job: 49.89, improvement: 97.09%)
  • fedora-42-xfce_dispvm-preload-dom0 (mean:4.37): 69.91 🟢 ( previous job: 72.35, improvement: 96.63%)
  • fedora-42-xfce_dispvm-preload-dom0-gui (mean:5.785): 92.55 🔺 ( previous job: 91.39, degradation: 101.27%)
  • fedora-42-xfce_dispvm-preload-dom0-concurrent (mean:3.578): 57.25 🔺 ( previous job: 57.13, degradation: 100.22%)
  • fedora-42-xfce_dispvm-preload-dom0-gui-concurrent (mean:4.329): 69.26 🔺 ( previous job: 67.89, degradation: 102.02%)
  • fedora-42-xfce_dispvm-api (mean:7.992): 127.87 🟢 ( previous job: 128.15, improvement: 99.78%)
  • fedora-42-xfce_dispvm-gui-api (mean:9.206): 147.29 🟢 ( previous job: 149.03, improvement: 98.83%)
  • fedora-42-xfce_dispvm-concurrent-api (mean:3.919): 62.71 🟢 ( previous job: 66.32, improvement: 94.55%)
  • fedora-42-xfce_dispvm-gui-concurrent-api (mean:4.535): 72.56 🟢 ( previous job: 77.33, improvement: 93.84%)
  • fedora-42-xfce_dispvm-preload-api (mean:4.798): 76.78 🔺 ( previous job: 73.95, degradation: 103.82%)
  • fedora-42-xfce_dispvm-preload-less-api (mean:7.434): 118.94 🔺 ( previous job: 116.39, degradation: 102.19%)
  • fedora-42-xfce_dispvm-preload-more-api (mean:4.441): 71.05 🟢 ( previous job: 71.33, improvement: 99.61%)
  • fedora-42-xfce_dispvm-preload-gui-api (mean:5.622): 89.94 🟢 ( previous job: 92.06, improvement: 97.70%)
  • fedora-42-xfce_dispvm-preload-concurrent-api (mean:3.506): 56.09 🟢 ( previous job: 61.60, improvement: 91.05%)
  • fedora-42-xfce_dispvm-preload-gui-concurrent-api (mean:4.226): 67.62 🟢 ( previous job: 77.66, improvement: 87.07%)
  • fedora-42-xfce_vm (mean:0.022): 0.36 🟢 ( previous job: 9.19, improvement: 3.87%)
  • fedora-42-xfce_vm-gui (mean:0.029): 0.46 🟢 ( previous job: 9.01, improvement: 5.09%)
  • fedora-42-xfce_vm-concurrent (mean:0.02): 0.33 🟢 ( previous job: 8.88, improvement: 3.66%)
  • fedora-42-xfce_vm-gui-concurrent (mean:0.021): 0.34 🟢 ( previous job: 9.15, improvement: 3.75%)
  • fedora-42-xfce_vm-api (mean:0.033): 0.53 🟢 ( previous job: 2.24, improvement: 23.76%)
  • fedora-42-xfce_vm-gui-api (mean:0.039): 0.62 🟢 ( previous job: 2.33, improvement: 26.49%)
  • fedora-42-xfce_vm-concurrent-api (mean:0.027): 0.44 🟢 ( previous job: 1.62, improvement: 27.09%)
  • fedora-42-xfce_vm-gui-concurrent-api (mean:0.032): 0.51 🟢 ( previous job: 2.20, improvement: 23.27%)
  • whonix-workstation-17_dispvm (mean:7.641): 122.26 🟢 ( previous job: 123.87, improvement: 98.69%)
  • whonix-workstation-17_dispvm-gui (mean:8.861): 141.78 🟢 ( previous job: 148.68, improvement: 95.36%)
  • whonix-workstation-17_dispvm-concurrent (mean:4.307): 68.92 🟢 ( previous job: 77.00, improvement: 89.51%)
  • whonix-workstation-17_dispvm-gui-concurrent (mean:5.032): 80.51 🟢 ( previous job: 89.18, improvement: 90.27%)
  • whonix-workstation-17_dispvm-dom0 (mean:8.466): 135.46 🔺 ( previous job: 135.24, degradation: 100.16%)
  • whonix-workstation-17_dispvm-dom0-gui (mean:9.852): 157.63 🟢 ( previous job: 159.23, improvement: 99.00%)
  • whonix-workstation-17_dispvm-dom0-concurrent (mean:4.574): 73.18 🟢 ( previous job: 76.91, improvement: 95.16%)
  • whonix-workstation-17_dispvm-dom0-gui-concurrent (mean:5.138): 82.21 🟢 ( previous job: 87.45, improvement: 94.01%)
  • whonix-workstation-17_dispvm-preload (mean:7.882): 126.11 🔺 ( previous job: 124.31, degradation: 101.45%)
  • whonix-workstation-17_dispvm-preload-gui (mean:8.68): 138.88 🔺 ( previous job: 138.71, degradation: 100.12%)
  • whonix-workstation-17_dispvm-preload-concurrent (mean:3.936): 62.98 🟢 ( previous job: 66.00, improvement: 95.41%)
  • whonix-workstation-17_dispvm-preload-gui-concurrent (mean:4.679): 74.87 🟢 ( previous job: 77.46, improvement: 96.65%)
  • whonix-workstation-17_dispvm-preload-dom0 (mean:4.871): 77.94 🟢 ( previous job: 81.24, improvement: 95.93%)
  • whonix-workstation-17_dispvm-preload-dom0-gui (mean:6.24): 99.84 🟢 ( previous job: 102.71, improvement: 97.21%)
  • whonix-workstation-17_dispvm-preload-dom0-concurrent (mean:4.112): 65.80 🟢 ( previous job: 72.84, improvement: 90.33%)
  • whonix-workstation-17_dispvm-preload-dom0-gui-concurrent (mean:5.006): 80.09 🟢 ( previous job: 87.14, improvement: 91.92%)
  • whonix-workstation-17_dispvm-api (mean:8.358): 133.73 🟢 ( previous job: 140.50, improvement: 95.18%)
  • whonix-workstation-17_dispvm-gui-api (mean:9.79): 156.64 🟢 ( previous job: 157.40, improvement: 99.52%)
  • whonix-workstation-17_dispvm-concurrent-api (mean:4.291): 68.65 🟢 ( previous job: 76.06, improvement: 90.26%)
  • whonix-workstation-17_dispvm-gui-concurrent-api (mean:4.921): 78.73 🟢 ( previous job: 87.18, improvement: 90.30%)
  • whonix-workstation-17_dispvm-preload-api (mean:5.07): 81.12 🟢 ( previous job: 84.09, improvement: 96.47%)
  • whonix-workstation-17_dispvm-preload-less-api (mean:7.886): 126.18 🔺 ( previous job: 126.04, degradation: 100.12%)
  • whonix-workstation-17_dispvm-preload-more-api (mean:5.397): 86.35 🟢 ( previous job: 89.12, improvement: 96.90%)
  • whonix-workstation-17_dispvm-preload-gui-api (mean:6.32): 101.12 🟢 ( previous job: 101.72, improvement: 99.41%)
  • whonix-workstation-17_dispvm-preload-concurrent-api (mean:4.176): 66.82 🟢 ( previous job: 71.58, improvement: 93.35%)
  • whonix-workstation-17_dispvm-preload-gui-concurrent-api (mean:5.025): 80.41 🟢 ( previous job: 88.24, improvement: 91.12%)
  • whonix-workstation-17_vm (mean:0.026): 0.41 🟢 ( previous job: 9.27, improvement: 4.48%)
  • whonix-workstation-17_vm-gui (mean:0.025): 0.40 🟢 ( previous job: 9.82, improvement: 4.02%)
  • whonix-workstation-17_vm-concurrent (mean:0.018): 0.29 🟢 ( previous job: 8.93, improvement: 3.29%)
  • whonix-workstation-17_vm-gui-concurrent (mean:0.025): 0.41 🟢 ( previous job: 9.38, improvement: 4.33%)
  • whonix-workstation-17_vm-api (mean:0.043): 0.69 🟢 ( previous job: 2.56, improvement: 27.07%)
  • whonix-workstation-17_vm-gui-api (mean:0.036): 0.58 🟢 ( previous job: 2.53, improvement: 22.89%)
  • whonix-workstation-17_vm-concurrent-api (mean:0.032): 0.50 🟢 ( previous job: 1.81, improvement: 27.83%)
  • whonix-workstation-17_vm-gui-concurrent-api (mean:0.024): 0.39 🟢 ( previous job: 2.57, improvement: 14.96%)

- btrfs-progs
- python3-nose2
- python3-objgraph
- python3-deepmerge
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrap this in {% if grains['osrelease'] >= '4.3' %} (or just != '4.2') - the package is not available in F37 (R4.2 dom0).

@marmarek marmarek merged commit 17a410e into QubesOS:main Sep 5, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants