Skip to content

Dependency solver quickcheck tests #3245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 29, 2016
Merged

Conversation

grayjay
Copy link
Collaborator

@grayjay grayjay commented Mar 29, 2016

I started adding quickcheck tests for the solver. There is currently only one test, which checks that parameters like --reorder-goals do not affect whether the solver finds an install plan.

Here is an example failure: https://gist.github.com/grayjay/86eea907cd08878855ae

I created a new test suite because the test is currently too slow to run as a unit test. The time to find one of the known bugs on master ranges from under a second to over a minute on my computer, with a median of about 6 seconds (10,000 tests). We may want to run it even longer after those bugs are fixed. Another problem is that the test occasionally runs out of memory. I haven't tried to debug it yet.

There are two other issues. I'm not sure how we should print failing test cases in a human-readable format. The PR currently uses the pretty-show package to parse and reformat the results from the derived Show instance, but it requires happy. I'm also not sure how to organize the test suite. I put the new test in the UnitTests directory since it uses the existing DSL module. Another option is to have three directories under tests, one each for unit tests, solver quickcheck tests, and shared modules.

@BardurArantsson
Copy link
Collaborator

Re: Showing output/data: present looks pretty nice, but the GHC version requirement may be too steep :(. (It seems to require GHC 7.10.x.)

@kosmikus
Copy link
Contributor

This looks quite interesting. Regarding the actual failure: can I assume that you only get failures for test cases involving setup dependencies?

@23Skidoo
Copy link
Member

LGTM, I think this can be merged.

Re: dependency on happy, I think it's fine in the case of the test suite.

@grayjay
Copy link
Collaborator Author

grayjay commented Mar 29, 2016

@BardurArantsson I'll continue using pretty-show since the dependency on happy isn't a problem, but thanks for the suggestion.

@kosmikus I just ran the test for over an hour without --independent-goals or setup dependencies, and there were no failures. I also didn't notice any other types of failure while working on the test.

@23Skidoo Thanks! Would you like me to merge it?

@23Skidoo
Copy link
Member

@grayjay Do the honours :-)

@grayjay
Copy link
Collaborator Author

grayjay commented Mar 29, 2016

@23Skidoo Thanks!

@grayjay grayjay merged commit 283f0af into haskell:master Mar 29, 2016
@grayjay grayjay deleted the solver-quickcheck branch April 1, 2016 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants