Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run more resource-intensive regression tests somewhere #1623

Open
chameco opened this issue Mar 23, 2022 · 1 comment
Open

Run more resource-intensive regression tests somewhere #1623

chameco opened this issue Mar 23, 2022 · 1 comment
Assignees
Labels
tooling: CI Issues involving CI/CD scripts or processes

Comments

@chameco
Copy link
Contributor

chameco commented Mar 23, 2022

We've historically used our "production" proofs in CI as regression tests. Some of these proofs (BLST, see #1622; maybe BIKE, see #1563) require enough time or memory that they are unwieldy to run in normal CI on every commit. However, it would still be useful to see if future SAW changes negatively impact these proofs. We should run these proofs regularly somewhere (maybe daily on master in Fryingpan?) and make sure that the results are visible.

@RyanGlScott RyanGlScott added the tooling: CI Issues involving CI/CD scripts or processes label Mar 23, 2022
@jldodds
Copy link
Contributor

jldodds commented Apr 4, 2022

Proposed approach, discussed by most caretakers:

  • Add a dev branch with automatic merge to main when full CI succeeds
  • Split CI into integration tests and proof tests
  • Integration tests run on every commit for every branch
  • Proof tests run nightly on dev branch on Galois runners kicked off from actions (could be parallelized to speed up)
  • Social agreement that it is acceptable to roll back dev branch to last fully working commit if breaking commit is blocking progress
  • Apply continual effort to continue to expand integration tests to minimize the need for proof tests

@kquick kquick self-assigned this Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tooling: CI Issues involving CI/CD scripts or processes
Projects
None yet
Development

No branches or pull requests

4 participants