Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revised benchmarks and made use of QUIT_GAP #966

Merged

Conversation

olexandr-konovalov
Copy link
Member

I have revised five benchmarks that are part of the weekly run test and made use of QUIT_GAP there to improve automated failure detection.

This PR does not change anything that may be tested by automated tests for PRs, and it can be rebased on the master branch.

@olexandr-konovalov olexandr-konovalov added the topic: tests issues or PRs related to tests label Nov 29, 2016
@codecov-io
Copy link

codecov-io commented Nov 29, 2016

Codecov Report

Merging #966 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #966   +/-   ##
=======================================
  Coverage   68.36%   68.36%           
=======================================
  Files         435      435           
  Lines      230679   230679           
=======================================
  Hits       157706   157706           
  Misses      72973    72973

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70e9509...43e33bb. Read the comment docs.

@olexandr-konovalov
Copy link
Member Author

The next update of this PR contains new benchmarks for automorphism groups and group isomorphism. They are based on @hulpke's initial (full) version of tst/teststandard/grpauto.tst from #896 before it was reduced to avoid running out of time/memory. I am testing the benchmarks at the moment - please do not merge yet.

@olexandr-konovalov olexandr-konovalov added the do not merge PRs which are not yet ready to be merged (e.g. submitted for discussion, or test results) label Nov 29, 2016
> 21952);;
gap> IsomorphismGroups(G,H);
fail
gap> IsomorphismGroups(G,PcGroupCode(CodePcGroup(G),Size(G)))=fail;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, I've started GAP with -o 32g test3.g and after 9.5 hours it's in the line 66, using 19.4g memory

# line 60, input:
G:=PcGroupCode(18738408935379049727906755356708168311565445686261463850856,
21952);;
# line 62, input:
H:=PcGroupCode(18738408935359210460657231881739776911013615108923450662760,
21952);;
# line 64, input:
IsomorphismGroups(G,H);
# line 66, input:
IsomorphismGroups(G,PcGroupCode(CodePcGroup(G),Size(G)))=fail;

@hulpke does it sound plausible?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still in line 66 and using now almost 30g (and unlike in files from tst, this uses default assertion level. I wonder whether that test was ever runnable at all. Reminder to myself: next time watch more carefully for PRs with tst files which are not covered by the Travis entry check.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, on my system the line 64 check takes 10 minutes, the line 66 one 30 seconds.

@hulpke
Copy link
Contributor

hulpke commented Nov 30, 2016

@alex-konovalov
I just tried this last line in my work branch (which should be minimally different from the additions branch you merged, and this last example ran through in 50 seconds.

However random choices (e.g. for permutation representations) can come into play, and there is a number of heuristics that decide whether (e.g.) some permutation representation is ``good enough''. This can be play out differently depending on random choices and yes, I only ran individual tests, not the whole file.

I will look at what happens in sequence with the previous tests.

olexandr-konovalov referenced this pull request Dec 1, 2016
Now it runs withing the memory limits in `make teststandard`
(-o 1g) and takes about 2000s with no packages (still with
info message about alternating recognition).
@olexandr-konovalov olexandr-konovalov removed the do not merge PRs which are not yet ready to be merged (e.g. submitted for discussion, or test results) label Mar 1, 2017
@olexandr-konovalov
Copy link
Member Author

I have rebased this onto the current master and now the problematic benchmark/grpauto/hardest.tst runs as well. So I think is is now ready for merge. Three commits are logically independent, so it should be fine to rebase this onto master. I am running these benchmarks weekly with Travis, and this PR will be helpful since now they should properly fail the test if they do not pass.

Alexander Konovalov added 3 commits March 7, 2017 10:58
- Renamed `Test` to `RunAllTests` to avoid name clash
- for d=1, return identity matrix to make inversion test working

Now you can read this file and then call e.g. RunAllTests(someqs,10);
These are based on the extended version of tst/teststandard/grpauto.tst
@ChrisJefferson ChrisJefferson merged commit e1d140b into gap-system:master Mar 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: tests issues or PRs related to tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants