Skip to content

Create a script + github action for rebaselining tests #23146

Open
@sbc100

Description

@sbc100

We have a bunch of codesize tests that need to be rebaselined. This need can occur under 2 different circumstances.

  1. A llvm or binaryen commit rolled in that changes the generated code.
  2. An emscripten PR causes changes to the generated code.

I think it would be good if we can avoid mixing (1) and (2). These means that the main branch should always be updated to include changes from llvm/binaryen before the changes for specific PR are landed (as part of that PR).

For (1) procedure I've been using looks something like this:

$ git checkout -b main upstream/main
$ ./emcc --clear-cache
$ test/runner other.*code_size* other.*codesize* --rebase
$ git add -u .
$ git commit -m "Rebaseline codesize expectations. NFC"
$ git show  # check everything looks reasonable
$ git push upsteam # direct push to upstream, no PR

There are several problems with this (as pointed out in #23126 (comment)).

  1. Its a lot of separate steps
  2. It involves pushing directly to repo, without any review
  3. Its hard to see (using git diff) exactly what effect the changes have.

I propose adding a new script that performs most of the step above that will have the following advantages:

  1. It can analyze the results and present a human readable report (e.g. test xxx regressed by 10 bytes (0.1%)).
  2. It can automatically create a PR with the report in the description.
  3. It can be turned into a github action so it can be run on a remote machine (not a users machine where llvm and binaries version might not be exactly at tot).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions