Skip to content

Fix autoharness termination test & print metadata in alphabetical order #3971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 3, 2025

Conversation

carolynzech
Copy link
Contributor

@carolynzech carolynzech commented Apr 1, 2025

#3968 failed in the merge queue because the cargo_autoharness_termination test assumed that the recursion test wouldn't time out, but it did. This doesn't happen often, since this test has worked up until now, but in the interest of having robust tests, this PR splits the termination test into two:

  • autoharness_timeout, which tests that the default timeout of 60s kicks in
  • autoharness_unwind, which tests that the default unwind bound of 20 kicks in

The unwind test specifies a timeout of 5m to ensure that the unwind bound takes effect before the timeout.

While I was fixing this, I also fixed some inconsistencies with the printed metadata:

  • We use "Chosen Function" in one table and "Selected Function" in another--changed it to "Selected Function" in both to be consistent
  • The functions aren't printed alphabetically in 2/3 tables--fixed so that they're always sorted

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

@github-actions github-actions bot added the Z-EndToEndBenchCI Tag a PR to run benchmark CI label Apr 1, 2025
@tautschnig
Copy link
Member

In some previous PR I had already run into the issue of timeouts occurring before the unwinding limit was reached, at which point I increases the default timeout to 60 seconds instead of 30 seconds. Can we instead adjust the timeout for just this test (via command-line options) to 5 minutes?

@carolynzech
Copy link
Contributor Author

In some previous PR I had already run into the issue of timeouts occurring before the unwinding limit was reached, at which point I increases the default timeout to 60 seconds instead of 30 seconds. Can we instead adjust the timeout for just this test (via command-line options) to 5 minutes?

I thought about doing this, but I thought that was a long time to let a single test run. I suppose it's okay to make one exception though.

@carolynzech carolynzech force-pushed the fix-autoharness-test branch from 9f8ef96 to 7de0c1b Compare April 1, 2025 16:23
@carolynzech carolynzech marked this pull request as ready for review April 1, 2025 16:24
@carolynzech carolynzech requested a review from a team as a code owner April 1, 2025 16:24
@carolynzech
Copy link
Contributor Author

I've split the termination test into two tests, one to test the unwind bound and one for the timeout. The unwind test sets the timeout to 5m, but at least testing locally it never gets close to that amount, so I think we're good.

@carolynzech carolynzech requested a review from tautschnig April 1, 2025 16:27
@tautschnig tautschnig added this pull request to the merge queue Apr 3, 2025
Merged via the queue into model-checking:main with commit 2cfe4dc Apr 3, 2025
24 of 25 checks passed
github-merge-queue bot pushed a commit that referenced this pull request Apr 4, 2025
Bump Kani version to 0.61.0.

Github-generated release notes:

## What's Changed
* Fix CHANGELOG of 0.60.0 by @qinheping in
#3925
* Bump tests/perf/s2n-quic from `d88faa4` to `8670e83` by @dependabot in
#3928
* Update toolchain to 2025-03-04 by @qinheping in
#3927
* Install the right toolchain for HEAD and BASE checks in
`verify-std-check.yml` by @remi-delmas-3000 in
#3920
* Automatic cargo update to 2025-03-10 by @github-actions in
#3926
* Automatic toolchain upgrade to nightly-2025-03-05 by @github-actions
in #3929
* Upgrade toolchain to nightly-2025-03-07 by @tautschnig in
#3931
* Upgrade toolchain to nightly-2025-03-12 by @tautschnig in
#3933
* Automatic toolchain upgrade to nightly-2025-03-13 by @github-actions
in #3934
* Update CBMC dependency to 6.5.0 by @tautschnig in
#3936
* Automatic toolchain upgrade to nightly-2025-03-14 by @github-actions
in #3937
* Automatic toolchain upgrade to nightly-2025-03-15 by @github-actions
in #3938
* Automatic toolchain upgrade to nightly-2025-03-16 by @github-actions
in #3939
* Automatic toolchain upgrade to nightly-2025-03-17 by @github-actions
in #3940
* Automatic cargo update to 2025-03-17 by @github-actions in
#3941
* Autoharness: Don't panic on `_` argument and add `_autoharness` suffix
to GOTO files by @carolynzech in
#3942
* Implement `f16` and `f128` cases in `codegen_float_type` by
@carolynzech in #3943
* Support function implementations of known built-ins by @tautschnig in
#3945
* Autoharness: metadata improvements and enable standard library
application by @carolynzech in
#3948
* Autoharness: `--list` option by @carolynzech in
#3952
* Add support for anonymous nested statics by @carolynzech in
#3953
* Automatic cargo update to 2025-03-24 by @github-actions in
#3954
* Bump tests/perf/s2n-quic from `8670e83` to `324cf31` by @dependabot in
#3955
* Document behavior of checked_size_of_raw and is_inbounds by @rajath-mk
in #3956
* Upgrade toolchain to 2025-03-18 by @zhassan-aws in
#3959
* Remove unstable-features from code formatting script by @zhassan-aws
in #3962
* Remove CI job to update features/verify-rust-std by @tautschnig in
#3963
* Make is_inbounds public by @rajath-mk in
#3958
* Enable Kani to work with a stable toolchain by @zhassan-aws in
#3964
* Automatic cargo update to 2025-03-31 by @github-actions in
#3966
* Add support for struct field accessing in loop contracts by
@thanhnguyen-aws in #3970
* Bump tests/perf/s2n-quic from `324cf31` to `d0aff82` by @dependabot in
#3968
* Clarify `is_inbounds` docs by @carolynzech in
#3974
* Upgrade toolchain to 2025-04-01 by @carolynzech in
#3973
* Remove remaining `--enable-unstable` mentions by @carolynzech in
#3978
* Clean up unused dependencies by @zhassan-aws in
#3981
* Automatic toolchain upgrade to nightly-2025-04-02 by @github-actions
in #3983
* Update dependencies per `cargo-outdated` by @carolynzech in
#3982
* Fix `autoharness` termination test & print metadata in alphabetical
order by @carolynzech in
#3971
* Fix cargo invocations to only use `pkg_args` where appropriate by
@carolynzech in #3984


**Full Changelog**:
kani-0.60.0...kani-0.61.0

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
@carolynzech carolynzech deleted the fix-autoharness-test branch April 11, 2025 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Z-EndToEndBenchCI Tag a PR to run benchmark CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants