Ideas about splitting slow l3build check/doc into smaller even runs

[muzimuzhi]

When using l3build in a CI, users may want to split a slow job running l3build check and/or l3build doc into several smaller even jobs run in parallel, to speed up the CI. The "even" means those smaller jobs are expected to take time as equal as possible.

Splitting by bundles and by config files are natural ways, but they usually result in uneven jobs, and they can't help in splitting a single config further.

To split l3build check, assuming each test takes equal times to check,

• LaTeX2e already used --first <name> and --last <name>, which is far from ideal.
  See commit latex3/latex2e@46f836e (Split LuaTeX main test executions in GH Actions, 2023-08-07) and Speed up automated testing (main.yaml) workflow runs latex2e#1073.
• Some new options like --first-num <number> and --last-num <number> which denotes the number of first and last tests to run seems a little better than --first/--last.
• Options that allow user to run the first third, second third, and last third (in general kth nth where 1 <= k <= n) is even better.

To split l3build doc, assuming user knowns which names are specifically slow, for example the source3 in l3kernel bundle, latex3/latex3 repository,

• A new option --skip <names> would help a lot.
  It will opens the possibility of running l3build doc source3 and l3build doc --skip source3 in parallel jobs.
• My comment in Split CI into smaller jobs latex3#1295 (comment) suggested adding support for glob pattern !, so l3build doc '!source3' would be the same as l3build doc --skip source3.

To split l3build ctan, one possible way is to provide a pair of new options --no-check and --no-doc.

How do other testing tools for general-purpose programming languages handle this task?

[josephwright]

To split l3build ctan, one possible way is to provide a pair of new options --no-check and --no-doc.

This part I think is a definite no: the entire reason that the ctan target is set up the way it is is that we want to ensure that a release build does 'everything' before getting to the .zip file creation.

[josephwright]

* Some new options like `--first-num <number>` and `--last-num <number>` which denotes the number of first and last tests to run seems a little better than `--first/--last`.

* Options that allow user to run the first third, second third, and last third (in general kth nth where 1 <= k <= n) is even better.

Sounds like --test-block=<m>:<n> or similar - 'test the <m>th block when dividing the tests into <n>' blocks?

[josephwright]

Could just be --block, so with a short version -b?

[josephwright]

To split l3build doc, assuming user knowns which names are specifically slow, for example the source3 in l3kernel bundle, latex3/latex3 repository,

* A new option `--skip <names>` would help a lot.
  It will opens the possibility of running `l3build doc source3` and `l3build doc --skip source3` in parallel jobs.

I'd imagine --exclude or --exclude-files, based on other tools: I guess one would likely go for just --exclude as we don't use file names, and this would need to apply to tests and to docs.