Skip to content

DevX: Improve error reporting for benchmark jobs. #8125

@guangy10

Description

@guangy10

🐛 Describe the bug

Image

As shown in an example run, there are two issues that make it difficult to understand the expected behavior and locate the exact error:

  1. Unexpected job success status: Two configurations failed to export, yet the benchmark-on-device jobs were still marked as successful (highlighted in the red box). The expected behavior is that benchmark-on-device jobs should either not be scheduled or be marked as skipped/canceled if a dependent job, such as export, fails.
  2. In the export-models job, the error in "Upload artifacts to S3" step is misleading. It should be marked as skipped or canceled, as the "Run script in the container" step failed to generate the .pte file (highlighted in the yellow box).

cc: @huydhn @cbilgin @digantdesai @kimishpatel

Versions

trunk

cc @huydhn @kirklandsign @shoumikhin @mergennachin @byjlw

Metadata

Metadata

Assignees

Labels

enhancementNot as big of a feature, but technically not a bug. Should be easy to fixmodule: benchmarkIssues related to the benchmark infrastructuremodule: user experienceIssues related to reducing friction for userstriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

Projects

Status

Done

Status

Done

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions