-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suite of integration tests for Minion #249
Comments
This is probably a long term project, but I strongly believe fuzzing is the only way to get top quality coverage. Once we had a program which could generate valid minion inputs, and tested with those, we found dozens of bugs in minion I'd never have found otherwise. |
I agree that fuzzing will be good and I agree that we are not there yet :) We probably want a native parser first, before we get to fuzzing. |
We can fuzz into rust types fairly easily using the abitrary crate (ie from bytes to structs without a parser) |
See #231 - could do something similar for the conjure oxide ast? |
My plan for minion_rs was to use the standard integration tests for "valid models" and "right solutions" and then test safety and ffi and the c++ by fuzzing over the minion_rs ast. If we fuzz the bindings well, we may not need to fuzz in conjure-oxide itself for Minion stuff, as minion_rs would be more or less safe |
I think long term it would be useful to have both a Rust native parser for Essence and pretty printing of our AST back to essence. Then it would probably be relatively easy to procedurally generate some valid models and pretty print them to an Essence file that we can feed to old conjure to compare results? |
We should definitely work on fuzzing, but maybe it should be a separate issue? I was thinking we could start with some existing Essence files just to quickly find some things that aren't working. (At this stage there will probably be obvious bugs in the rules, or some constraints just not being supported etc) And later, when we have a working base, we can try to test more exhaustively with fuzzing |
We need a large suite of problems to solve with minion and compare our solutions to conjure (see #247).
Having a large test suite would probably flag up some issues (unsupported Essence features, unsupported Minion constraints, issues with the rewriting engine, issues in minion_rs) and hopefully provide more direction for further work
The text was updated successfully, but these errors were encountered: