Skip to content

Start a 'test' directory and add a bunch of async tests #519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 23, 2025
Merged

Conversation

lukewagner
Copy link
Member

This PR adds a bunch of tests to a new test/async directory, along with a few placeholder tests in test/values and test/resources. The async tests call all the built-ins and exercise a number of the major semantic cases, but of course many more tests are necessary to cover all the semantically interesting cases.

Currently, the tests found a number of small differences between the spec and the implementation in the wasip3-prototyping repo, which have been filed and are in the process of being merged. Once the merging is complete, the tests should all successfully run on the implementation.

If anyone wants to get into writing more wast tests, please jump in! There's a good amount of example material to work off of to get started.

@alexcrichton
Copy link
Collaborator

Nice! And thanks for writing these up!

Do you have ideas on how to improve the longevity of these tests over time? For example ideally we'd run these tests in CI but that's not easily possible today as wasip3-prototyping still needs bug fixes. Even if that weren't the case though it'll always be true that spec things will likely land here first and, in theory, have tests before it's implemented anywhere.

For the time being it seems reasonable to me to land them here (and somehow set up infrastructure to have them work their way over to Wasmtime), and I have no issue with not running them for now. In the long-term though I do think that we'd ideally get to the point of being able to run the tests. The "most viable" path at this current juncture would possibly be something like depending on wasmtime-py as Python-based bindings for core WebAssembly and then using all the existing Python code for lifting/lowering for running components. That's a pretty large project, however, and I could understand how both (a) that wouldn't happen for quite awhile and (b) that may not even be the best choice of implementation primitives.

Anyway, hence leading to the question about thoughts of running these tests in CI here somehow!

@lukewagner
Copy link
Member Author

That's a great question, and I was thinking along the same lines. Ideally, we would simply have a reference interpreter in the same repo and CI would run the test suite on it. #192 actually has a draft reference interpreter, but it's a bit stale now. But, as you said, I have been considering whether I should fill out the existing Canonical ABI Python into a full reference interpreter. It would take some time, though, and getting to a 1.0 feature set has so far been the priority. But once a few more key missing pieces are filled in, I could maybe switch gears and do that.

Copy link
Collaborator

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I meant to approve this as well, and yeah makes sense to me. In the meantime I like the idea of prepping various bits of infrastructure to get these tests plumbed around.

@lukewagner
Copy link
Member Author

Alright, looks like everything succeeds on the current dev release of wasip3-prototyping with the exception of the one "TODO" in partial-stream-copies.wast, so I'll merge for now and we can iterate as things like #520 get implemented.

@lukewagner lukewagner merged commit b6be91e into main May 23, 2025
2 checks passed
@lukewagner lukewagner deleted the add-tests branch May 23, 2025 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants