Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

saw should be repeatable, or have a repeatable mode #2090

Open
2 tasks
sauclovian-g opened this issue Aug 13, 2024 · 0 comments
Open
2 tasks

saw should be repeatable, or have a repeatable mode #2090

sauclovian-g opened this issue Aug 13, 2024 · 0 comments
Labels
tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable type: enhancement Issues describing an improvement to an existing feature or capability

Comments

@sauclovian-g
Copy link
Collaborator

sauclovian-g commented Aug 13, 2024

Currently if you run saw multiple times or from different checkouts, the output varies. Ideally the results would be fully repeatable. This is desirable for testing and also for differential analysis of bugs and so forth.

Some solver-based things may not be fully repeatable without significant intervention and that may not be worthwhile (for example, exact counterexamples for verification failures) but there are at least two things that currently vary that could be nailed down quite easily:

  • Timestamps. Right now all outputs are timestamped; this is sometimes useful but should be suppressible.
  • Filenames. Currently saw runs realpath on filenames, which it probably shouldn't at all (see saw inappropriately expands symbolic links #2082). This introduces the current directory (which is often, especially when testing/developing, the path of the current saw-script checkout) into the output. At a minimum, in repeatable mode it ought to prune the current directory from the filenames it prints.
@sauclovian-g sauclovian-g added the tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable label Aug 13, 2024
@sauclovian-g sauclovian-g added type: bug Issues reporting bugs or unexpected/unwanted behavior type: enhancement Issues describing an improvement to an existing feature or capability and removed type: bug Issues reporting bugs or unexpected/unwanted behavior labels Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tooling: test infrastructure Issues involving test infrastructure or test execution, or making SAW more testable type: enhancement Issues describing an improvement to an existing feature or capability
Projects
None yet
Development

No branches or pull requests

1 participant