Move benchmarking workflow cli from testinfra to executorch #10162

Camyll · 2025-04-14T21:51:36Z

Request from Executorch team to run android-perf or apple-perf workflows via commandline

https://github.com/pytorch/executorch/actions/workflows/android-perf.yml
https://github.com/pytorch/executorch/actions/workflows/apple-perf.yml

HOW TO:

Be in repo pytorch/executorch. Navigate to extension/benchmark folder.
Make sure you have your GITHUB_TOKEN set (classic personal access token, not fine-grained) (export GITHUB_TOKEN=<YOUR_TOKEN>)
run python benchmark_workflow_cli.py with either --android or --ios as a required argument
use --help to see other optional arguments (--branch, --models, --devices, --benchmark-configs)

cc @guangy10 @huydhn @kirklandsign @shoumikhin

pytorch-bot · 2025-04-14T21:51:40Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10162

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 24 Unrelated Failures

As of commit cd6b6fe with merge base cb3eba0 ():

NEW FAILURES - The following jobs have failed:

Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
The process '/usr/bin/git' failed with exit code 1
Check Labels / Check labels (gh)
RuntimeError: Error checking labels: PR does not have required labels
Lint / lintrunner / linux-job (gh)
>>> Lint for extension/benchmark/scripts/benchmark.py:
pull / android / run-emulator (gh)
The process '/usr/bin/sh' failed with exit code 255

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / test-eval_llama-mmlu-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
pull / test-eval_llama-wikitext-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-llama_runner_eager-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (add, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (emformer_join, portable, linux.4xlarge.memory) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
pull / test-models-linux (ic3, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (linear, portable, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
pull / test-models-linux (w2l, portable, linux.4xlarge.memory) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux-basic (mv3, portable, cmake, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11... / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux-basic (vit, portable, buck2, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-models-linux-basic (vit, portable, cmake, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-quantized-aot-lib-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-setup-linux-gcc / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / test-static-llama-qnn-linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / unittest / linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
pull / unittest-buck / linux / linux-job (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2025-04-14T21:52:11Z

This PR needs a `release notes:` label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

extension/benchmark/benchmark_workflow_cli.py

guangy10 · 2025-04-14T23:50:28Z

extension/benchmark/benchmark_workflow_cli.py

+        "--models",
+        type=str,
+        required=False,
+        default="llama",


Similar to branch. Users should specify what model(s) they want to benchmark with. Do not provide default seems better to me

extension/benchmark/benchmark_workflow_cli.py

guangy10 · 2025-04-14T23:52:57Z

extension/benchmark/benchmark_workflow_cli.py

+    app_type = parser.add_mutually_exclusive_group(required=True)
+    app_type.add_argument(
+        "--android",
+        action="store_true",
+        required=False,
+        help="run the test on Android",
+    )
+    app_type.add_argument(
+        "--ios",
+        action="store_true",
+        required=False,
+        help="run the test on iOS",
+    )


nit: to be consistent, can we rename it to "platform"?

extension/benchmark/benchmark_workflow_cli.py

guangy10

Do not forget to include the instructions to the README.md

Camyll · 2025-05-21T19:48:56Z

@guangy10 can you remove changes requested as the PR has been updated? Thanks!

huydhn · 2025-05-23T01:22:15Z

There are lots of unrelated failures, so you probably want to rebase it and also fix lint before landing (I could help with that)

move workflow cli from testinfra to executorch

d59abc4

Camyll requested a review from tarun292 as a code owner April 14, 2025 21:51

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 14, 2025

Camyll requested review from huydhn, guangy10, kirklandsign and yangw-dev April 14, 2025 21:52

guangy10 reviewed Apr 14, 2025

View reviewed changes