Skip to content

Remove non-deterministic tests from test runner#706

Merged
rennergade merged 1 commit intomainfrom
667-deprecate-non-deterministic-test-functionality
Feb 6, 2026
Merged

Remove non-deterministic tests from test runner#706
rennergade merged 1 commit intomainfrom
667-deprecate-non-deterministic-test-functionality

Conversation

@m-hemmings
Copy link
Member

ISSUE

The test runner previously maintained a separate "non-deterministic" test path which is now unused

DESCRIPTION

  • Remove the NON_DETERMINISTIC_PARENT_NAME constant and all routing for non-deterministic/ test folders from scripts/wasmtestreport.py.
  • Delete the test_single_file_non_deterministic wrapper and simplify the unified test function docstring to reflect only deterministic and fail modes.

@m-hemmings m-hemmings linked an issue Feb 6, 2026 that may be closed by this pull request
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

End-to-End Test Report

Test Preview

Test Report

Deterministic Tests

Summary

MetricCount
Total Test Cases142
Number of Successes142
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeOutput
File Tests
chartests.cSuccessNone
Success
chdir_getcwd.cSuccessNone
Success
chmod.cSuccessNone
Success
clock_gettime_highlevel.cSuccessNone
Success
clock_gettime_simple.cSuccessNone
Success
cloexec.cSuccessNone
Success
close.cSuccessNone
Success
doubleclose.cSuccessNone
Success
dup.cSuccessNone
Success
dup2.cSuccessNone
Success
dup3.cSuccessNone
Success
dupwrite.cSuccessNone
Success
fchdir.cSuccessNone
Success
fchmod.cSuccessNone
Success
fcntl.cSuccessNone
Success
fdatasync.cSuccessNone
Success
filetest.cSuccessNone
Success
filetest1000.cSuccessNone
Success
flock.cSuccessNone
Success
fstat.cSuccessNone
Success
fstatfs.cSuccessNone
Success
fsync.cSuccessNone
Success
ftruncate.cSuccessNone
Success
getcwd.cSuccessNone
Success
getrandom.cSuccessNone
Success
ioctl.cSuccessNone
Success
link.cSuccessNone
Success
lseek.cSuccessNone
Success
mkdir_rmdir.cSuccessNone
Success
open.cSuccessNone
Success
pread_pwrite.cSuccessNone
Success
printf.cSuccessNone
Success
read.cSuccessNone
Success
readbytes.cSuccessNone
Success
readlink.cSuccessNone
Success
readlinkat.cSuccessNone
Success
rename.cSuccessNone
Success
sc-writev.cSuccessNone
Success
stat.cSuccessNone
Success
statfs.cSuccessNone
Success
sync_file_range.cSuccessNone
Success
truncate.cSuccessNone
Success
unlink.cSuccessNone
Success
unlinkat.cSuccessNone
Success
write.cSuccessNone
Success
writeloop.cSuccessNone
Success
writepartial.cSuccessNone
Success
writev.cSuccessNone
Success
Memory Tests
brk.cSuccessNone
Success
malloc.cSuccessNone
Success
malloc_large.cSuccessNone
Success
memcpy.cSuccessNone
Success
memory_error_test.cSuccessNone
Success
mmap.cSuccessNone
Success
mmap_aligned.cSuccessNone
Success
mmap_complicated.cSuccessNone
Success
mmap_file.cSuccessNone
Success
mmap_shared.cSuccessNone
Success
mprotect.cSuccessNone
Success
mprotect_boundary.cSuccessNone
Success
mprotect_end_region.cSuccessNone
Success
mprotect_middle_region.cSuccessNone
Success
mprotect_multiple_times.cSuccessNone
Success
mprotect_same_value.cSuccessNone
Success
mprotect_spanning_regions.cSuccessNone
Success
sbrk.cSuccessNone
Success
segfault.cSuccessNone
Success
shm.cSuccessNone
Success
shmtest.cSuccessNone
Success
vtable.cSuccessNone
Success
Networking Tests
dnstest.cSuccessNone
Success
epollcreate1.cSuccessNone
Success
gethostname.cSuccessNone
Success
getsockopt.cSuccessNone
Success
makepipe.cSuccessNone
Success
pipe.cSuccessNone
Success
pipe2.cSuccessNone
Success
pipeinput.cSuccessNone
Success
pipeinput2.cSuccessNone
Success
pipeonestring.cSuccessNone
Success
pipepong.cSuccessNone
Success
pipewrite.cSuccessNone
Success
poll.cSuccessNone
Success
recvfrom-sendto.cSuccessNone
Success
serverclient.cSuccessNone
Success
shutdown.cSuccessNone
Success
shutdown_fork.cSuccessNone
Success
simple-select.cSuccessNone
Success
simple_epoll.cSuccessNone
Success
socket.cSuccessNone
Success
socket_cloexec.cSuccessNone
Success
socketepoll.cSuccessNone
Success
socketpair.cSuccessNone
Success
socketselect.cSuccessNone
Success
tcp_connect_single.cSuccessNone
Success
uds-getsockname.cSuccessNone
Success
uds-socketselect.cSuccessNone
Success
Process Tests
chain_thread.cSuccessNone
Success
exit.cSuccessNone
Success
fork.cSuccessNone
Success
fork2malloc.cSuccessNone
Success
fork_simple.cSuccessNone
Success
fork_syscall.cSuccessNone
Success
forkandopen.cSuccessNone
Success
forkdup.cSuccessNone
Success
forkexecuid.cSuccessNone
Success
forkexecv-arg.cSuccessNone
Success
forkexecv.cSuccessNone
Success
forkfiles.cSuccessNone
Success
forkmalloc.cSuccessNone
Success
forknodup.cSuccessNone
Success
function-ptr.cSuccessNone
Success
getegid_syscall.cSuccessNone
Success
getgid_syscall.cSuccessNone
Success
getpid.cSuccessNone
Success
getpid_syscall.cSuccessNone
Success
getppid.cSuccessNone
Success
getppid_syscall.cSuccessNone
Success
getuid.cSuccessNone
Success
getuid_syscall.cSuccessNone
Success
hello-arg.cSuccessNone
Success
hello.cSuccessNone
Success
longjmp.cSuccessNone
Success
mutex.cSuccessNone
Success
noforkfiles.cSuccessNone
Success
sem_forks.cSuccessNone
Success
setsid.cSuccessNone
Success
template.cSuccessNone
Success
thread.cSuccessNone
Success
wait.cSuccessNone
Success
waitpid_anychild.cSuccessNone
Success
waitpid_syscall.cSuccessNone
Success
waitpid_wnohang.cSuccessNone
Success
Signal Tests
alarm.cSuccessNone
Success
setitimer.cSuccessNone
Success
sigalrm.cSuccessNone
Success
sigchld.cSuccessNone
Success
signal-fork.cSuccessNone
Success
signal-simple.cSuccessNone
Success
signal_fork.cSuccessNone
Success
signal_int_ignored.cSuccessNone
Success
sigprocmask.cSuccessNone
Success
Fail Tests

Summary

MetricCount
Total Test Cases1
Number of Successes1
Number of Failures0
Number of Compilation Failure Native0
Number of Runtime Failure Native0
Number of Segmentation Fault Native0
Number of Timeout During Native0
Number of Lind Wasm Compile Failure0
Number of Lind Wasm Runtime Failure0
Number of Lind Wasm Segmentation Failure0
Number of Timeout During Lind Wasm run0
Number of Unknown Failure0
Number of C Compiler and Wasm Output mismatch0
Number of Fail Test: Native Succeeded (Should Fail)0
Number of Fail Test: Wasm Succeeded (Should Fail)0
Number of Fail Test: Both Native and Wasm Succeeded (Should Fail)0
Number of Fail Test: Native Compilation Failure (Should Succeed)0
Number of Fail Test: Wasm Compilation Failure (Should Succeed)0

Test Results by Category

Test CaseStatusError TypeOutput
Memory Tests
mmap-negative1.cSuccessNone
Success

@rennergade rennergade merged commit a78185a into main Feb 6, 2026
3 checks passed
Yaxuan-w added a commit that referenced this pull request Feb 8, 2026
commit 48e27f5
Author: MHemmings <discount.yoyos@gmail.com>
Date:   Sat Feb 7 08:37:03 2026 -0800

    Add support for flags to be passed to wasmtestreport.py and lind_comp… (#691)

    * Add support for flags to be passed to wasmtestreport.py and lind_compile.sh scripts

    * Test permissions on wasmtestreport.py

    * Correct permissions on other scripts for tests

    * Remove support for json files and hard code math folder as variable

commit a78185a
Author: MHemmings <discount.yoyos@gmail.com>
Date:   Fri Feb 6 12:43:45 2026 -0800

    Remove non-deterministic tests from test runner (#706)

commit e717f4a
Author: Alice Wen <40227173+Yaxuan-w@users.noreply.github.com>
Date:   Thu Feb 5 14:01:59 2026 -0500

    Update gitignore (#702)

    * Update gitignore

    * Update gitignore

commit df0e912
Author: Alice Wen <40227173+Yaxuan-w@users.noreply.github.com>
Date:   Thu Feb 5 11:22:00 2026 -0500

    Add 3i and grate documentation (#493)

    * Add 3i and grate documentation

    * Deploy 3i doc to web

    * Fix type

    * Revise Lind-Wasm documentation for clarity and detail

    Updated the Lind-Wasm documentation to clarify the concepts of cages, grates, and the 3i system. Enhanced explanations of memory safety, control flow integrity, and the roles of Wasmtime and RawPOSIX.

    * Revising 3i doc according to comments

    * Update 3i.md (upper portion)

    I will comment on some of the lower portions which I think are wrong.

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update Wasm Implementation

    * Update wasm implementation

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Fix typo

    * Add 3i API doc + update 3i readme

    * Update 3i.md

    * Update 3i.md

    * Update 3i.md

    * todo

    * Update grate call routine doc and pic

    * Update according to comments

    * Add thread pool explanation

    * Remove ds_store

    * Add link and refine

    ---------

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

commit 44d910e
Author: Rishabh Budhouliya <31189405+rishabhBudhouliya@users.noreply.github.com>
Date:   Wed Feb 4 19:06:04 2026 -0500

    lind_run: re-exec with sudo when not running as root (#688)

    * add sudo if not root for lind_run

    * refactor
Yaxuan-w added a commit that referenced this pull request Feb 8, 2026
* Update Makefile, e2e, compilation, run scripts

* Add chroot fix to lind_run

* Copy to chroot

* Squashed commit of the following:

commit 48e27f5
Author: MHemmings <discount.yoyos@gmail.com>
Date:   Sat Feb 7 08:37:03 2026 -0800

    Add support for flags to be passed to wasmtestreport.py and lind_comp… (#691)

    * Add support for flags to be passed to wasmtestreport.py and lind_compile.sh scripts

    * Test permissions on wasmtestreport.py

    * Correct permissions on other scripts for tests

    * Remove support for json files and hard code math folder as variable

commit a78185a
Author: MHemmings <discount.yoyos@gmail.com>
Date:   Fri Feb 6 12:43:45 2026 -0800

    Remove non-deterministic tests from test runner (#706)

commit e717f4a
Author: Alice Wen <40227173+Yaxuan-w@users.noreply.github.com>
Date:   Thu Feb 5 14:01:59 2026 -0500

    Update gitignore (#702)

    * Update gitignore

    * Update gitignore

commit df0e912
Author: Alice Wen <40227173+Yaxuan-w@users.noreply.github.com>
Date:   Thu Feb 5 11:22:00 2026 -0500

    Add 3i and grate documentation (#493)

    * Add 3i and grate documentation

    * Deploy 3i doc to web

    * Fix type

    * Revise Lind-Wasm documentation for clarity and detail

    Updated the Lind-Wasm documentation to clarify the concepts of cages, grates, and the 3i system. Enhanced explanations of memory safety, control flow integrity, and the roles of Wasmtime and RawPOSIX.

    * Revising 3i doc according to comments

    * Update 3i.md (upper portion)

    I will comment on some of the lower portions which I think are wrong.

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/index.md

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Update Wasm Implementation

    * Update wasm implementation

    * Update docs/internal/3i.md

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

    * Fix typo

    * Add 3i API doc + update 3i readme

    * Update 3i.md

    * Update 3i.md

    * Update 3i.md

    * todo

    * Update grate call routine doc and pic

    * Update according to comments

    * Add thread pool explanation

    * Remove ds_store

    * Add link and refine

    ---------

    Co-authored-by: Justin Cappos <justincappos@gmail.com>

commit 44d910e
Author: Rishabh Budhouliya <31189405+rishabhBudhouliya@users.noreply.github.com>
Date:   Wed Feb 4 19:06:04 2026 -0500

    lind_run: re-exec with sudo when not running as root (#688)

    * add sudo if not root for lind_run

    * refactor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deprecate non-deterministic test functionality

3 participants