Rework test suite to additionally/primarily use captured outputs

[scop]

Describe the feature/solution

Currently we pull a bunch of large containers for running the test suite, and run them against real commands in them. While this is useful in the sense that it tests against the very real things, and also in situations where unexpected errors in commands might occur, and tracks changes in developing distros, it is:

• slow
• limited to what containers we have available
• wasteful, multiple gigabytes pulled from docker registry

It would be great if we had a corpus of command outputs for different command versions on different platforms we would primarily test against. Secondarily we should (perhaps on schedule, bi-weekly or so) run the live tests (i.e. what we have now) at least against developing distros such as Fedora and Alpine bleeding edges, perhaps Debian too.

Maintenance (please complete the following information)

• This is a request for new completion
• Link to upstream project query about shipping the completion:

N/A

[akinomyoga]

I'm interested in how you think this should be achieved. For example, are target commands supposed to be replaced by the function of the same name in the tests? How are we going to manage the "corpus" of the command outputs? For example, if we save them in the git tree, I'm afraid that the repository size would become unreasonably large, so should we manage the "corpus" externally? Maybe we can automatically generate/update the corpus with CI.

[scop]

I was primarily thinking of adding mock executables, but guess functions could work as well. We have a few examples of the former in the tree, ls test/**/bin

Command outputs could be saved as text files, and the mocked executables/functions would basically cat them.

Sure, the tree would become largish, but I'm not sure how much of a problem that would be. But if it is, and an external storage would solve it, I'm all for it.

Generating in CI or some automated way would be nice. Probably need a way to make manual additions work too, though.