Can we run (some) manual tests on travis?

[fingolfin]

The examples from the manual add a lot of coverage. It would be nice if we could run some or even all of them on travis (I have no idea how expensiv they are).

I guess the first step for this would be to improve the way we handle them. Right now, there is a relatively convoluted rule in the primary Makefile, which runs some GAP scripts (doc/ref/extractexamples.g, doc/ref/runexamples.g, and the same for doc/tut) and restarts GAP for each chapter. It also runs the tests three times, once starting GAP normally, once with option -A, and once with a LoadAllPackage() call added in.

I think we should instead have these scripts in tst; one script which takes one of the manuals (doc/ref, doc/tut, in the future perhaps also doc/dev), and extracts all examples from it into a subdir, say tst/doc/ref or tst/doc-examples/ref (and the same for tut). Initially, this would create one .tst file pr chapter, but of course we could revise it.

Afterwards, we can just use TestDirectory to run the tests. This way, we unify the way the tests are run, and can investigate how long the tests of each chapter take; we could then add one or multiple travis builds which run the examples of different subsets of the chapters (covering as many as possibly, ideally all).

Note: Restarting GAP between chapters might still be advisable, in that case we'd not useTestDirectory, but the rest of what I said would stay valid, I think.

[markuspf]

I am all for it.

Just to clarify: Do you want to extract the manual examples and commit the test files to the git repository, or do you want to extract them when the test is run?

[fingolfin]

The latter, so that they are always in sync with the manual.

[olexandr-konovalov]

I have some thoughts - will write later.

[ChrisJefferson]

I will not have time to work on this, but if any technical issues arise with the profiling itself, let me know and I am happy to add new functionality as required (where possible of course!)

[markuspf]

Doesn't seem to be too hard to do, I'll try myself a PR.

[markuspf]

I might regret asking this, but does there already exist some automation to give filenames to the documentation scraper? At the moment it seems filenames are recorded in makedocreldata.g...

[fingolfin]

@markuspf You mean, like, instead of listing every. single. file. you want to be able to say something like everything in lib? Well, AutoDoc does that for you.... Now, I am not suggesting to use AutoDoc here, but doing this is quite easy, and you could borrow from AutoDoc (code or inspiration).

Or perhaps you meant something else, then please clarify.

[markuspf]

No I mean exactly this, can you point me to the function in autodoc that can inspire me?

[fingolfin]

It's AUTODOC_FindMatchingFiles and I just see that the utils package also has FindMatchingFiles (based on the AutoDoc code)

[markuspf]

I might do a PR that puts this in the library (FindMatchingFiles)?

I shall have a look, it;d be unfortunate if we had this function in 3 places...

[fingolfin]

Well, the FindMatchingFiles in utils clearly is based on an older version of AUTODOC_FindMatchingFiles, which was the result of some refactoring there. As it is, I think FindMatchingFiles in utils is not very generic and hence not useful for most people (note that I did not submit it to utils, that was @sebasguts I think, and @cdwensley added it?)

And I just noticed that the latest AUTODOC_FindMatchingFiles in AutoDoc is also buggy, as it ignores the extensions argument, hardcoding a value instead (not sure how that happened).

Finally, I am still not sure even this version of the function, with bug fix included, is terribly useful for public consumption, so I'd be reluctant to add to GAP as a documented function... and if you add it to GAP as an undocumented function, then I'd still keep my version in AutoDoc, as I don't want to rely on undocumented APIs... :-)

[fingolfin]

Also, I would expect there to be some overlap with TestDirectory, as that also needs to collect all .tst files in a given directory.

I guess what I am saying is: Don't waste too much time polishing this and putting it in "nicely", rather just get something working for now...