Automated plan split into subplans

[lukaszachy]

Use case 1: Plan contain mutually exclusive tests, tmt should run distinct sets of tests separately
Use case 2: Each plan should be run multiple times with slightly modified environment

UC1: Test attributes as following should cause only equivalent tests to be run in same sub plan

• Hardware (min/max memory, CPU count...) requirement conflicts
• Additional repositories for require/recommend containing unsupported/not shipped packages - test contains metadata representing this fact.

UC2: Run plan "normally" as well as with FIPS mode enabled. Set of tests might be further pruned using relevancy.

It would be great if TMT allows sub-plans generation based on various attributes/plugins and each of them could be individually turn on/off.

[pvalena]

The automated part aside which will be deduced from L1 (and therefore kind-of hackish), when gathered the conflicting requirements, there should be some resul / definition for L2, right?.

Manually, If I wanted to say I want to say in my L2 metadata, while trying to be most specfiic, what should be the actual "split"?

Would it look something like this?

prepare:
    split_plan:
        - repo: http://.......
provision:
    split_plan:
        - fips: enabled
        - cpu: [ 1, 2 ]
        - how: [ container, virtual ]     # (just some theoretical example)

So with each entry, the plan would "duplicate" into two, like so:

• with and without the repository enabled
• with and without the FIPS enabled
• with 1 and 2 CPUs
• and maybe with 2 different provisioners

Would that make sense? Or do you have some completely different approach in mind?

How would UC 1 fit into this? Would there be some additional filtering, based on the requirements?

---

Thinking about this I see 2 distinct behaviours: split_plan (UC1: the tests won't be same in the two) and duplicate_plan (UC2: the tests will be completely same). This also affects how the results will be gathered / merged / reported etc..

[psss]

Recently @fernflower shared with us a related use case for splitting plans per test:

• Each test executed separately on a dedicated (clean) guest
• Test results aggregated from all executed plans

I propose to start with covering this use case as the first step for automated plan generation. The config could look like this:

discover:
    how: fmf
    slice: 1

Any better name for the option? The slice is coming from wow syntax. Perhaps use split instead? Or something like split-plan dictionary proposed above by @pvalena?

The question is into which step config it should go. Probably under the discover step as it is closely related to individual tests? Or under execute? Or even introduce a new config section for options which are not exactly related to a single step?

@lukaszachy, @thrix, @pvalena, @sopos, @jscotka, @fernflower, please share your thoughts.

[sopos]

It is still questionable where (in which step) the test set manipulation should be done. I will propagate my earlier idea of adding a new step which would be dedicated to various manipulations.
So in the discover step, there would be tests gathered as usual and them there would be a modular step which would cut it based on various rules, e.g. simply slice, or group by common HW required, ...

[pvalena]

Actually, I think we have the manually enumerated variants solved via inheritance quite well, and I would rather avoid introducing another syntax.

To take the earlier example:

prepare:
    [ . . . ]
execute:
    [ . . . ]

# maybe this isn't valid syntax. Sorry, writing this from the top of my head.
/fips-enabled
    provision:
        - how: virtual
          fips: enabled

/single-cpu
    provision:
        - how: virtual
          cpu: 1

/double-cpu
    provision:
        - how: virtual
          cpu: 2

/container-provision
    provision:
        - how: container

IOW I retract my earlier proposal for lists / split_plan.

---

It doesn't cover slicing (generating multiple plans from tests), but I think that's a separate issue, which requires autogenerating / special handling (theoretically, you could write them manually as well, but I don't think that would be feasible/usable).

+1 for slice for me. I'd prefer having it in execute,which seems kind-of intuitive (like execute tests "in isolation"); and we don't have many options in there currently:), but I would rather avoid creating another "step", unless it can be accomodated in other way (what would it "do", apart from managing other steps?).

[psss]

A couple of thoughts from the implementation point of view as discussed today with @mkluson and @bocekm today:

So the expectation is that there is syntax defined which enables to create a separate plan for each test identified during the discover step. Here's a little chicken-egg problem for the tmt plan ls or tmt plan show command: How to explore and list/show all available plans without actually executing the discover step?

One possible solution is to perform the expansion only when the plan is executed: So we start with a single plan, run the discover step, find out there are 20 tests and create 20 plans for each of them. This will not work with how the Testing Farm is currently implementing the execution, which is roughly this but in parallel:

for plan in $(tmt plan ls --filter enabled:true); do
    provision a guest
    tmt run --all plan --name $plan provision --how connect --guest $guest

@thrix, @happz, @lukaszachy, @FrNecas, any ideas how to handle this? I don't think it would be a good idea to execute discover step for each plan when just exploring available plans in the repository. Especially for tests fetched from remote repositories this would slow down things substantially.

[kkaarreell]

Just an idea. When listing a directory you can see files and subdirectories. You can also list directories recursively. Would such concept make sense in this situation? Can a sub-plan be represented by an object that is listed but not processed as a test case .. and only processed when required (-R)?

[happz]

Sounds like two incompatible cases. As soon as a test and its metadata presence can lead to a new dynamic plan being spawn just for that test, listing plans without test discovery will inherently yield an incomplete list, sooner or later. One way how to get out of this could be a warning and an option: "Be aware that test metadata may affect the final list of plans, especially if tests employ the following metadata fields: .... If you wish to see the final list of plans, run plan ls --be-thorough-and-perform-discovery for the full and complete list."

I'm not sure there's a perfect solution. If performance is the issue here - and I understand cloning remote repositories just to find out they don't change anything could be a problem - I'd be fine with the behavior I described above.

[lukaszachy]

+1 for @happz idea:

tmt p ls to see if there are some plans
tmt p ls --be-thoroug... if you need to know real situation

[kkaarreell]

From a user perspective, I would like to be able to distinguish these to cases in tmt output, i.e. a warning or different form of an output (preferably) would be used only in case there are such nested plans. Personally, I would like to see the nested plan listed in the output, just identified as not-a-test but a subplan, similarly to the ls file/directory example.

[lukaszachy]

From Stakeholders mtg: We should support usage when final plan name stays the same, even though tests require several distinct provision or prepare steps.

E.g. tmt configuration creates plan called e.g. smoke-test. This name will be used in gating.yml.
tmt shouldn't change this name even though there will be tests requiring their own machines, slicing or separate prepare steps.
So in this case tests from single plan will run on several provisioned machines but their results will be reported into single plan.

[lukaszachy]

Summary from stakeholders mtg:

• tmt needs command for listing all SUT (System Under Test: specific provision + prepare combination in which tests are run) for (selected) plans
• split into distinct SUT caused by Test metadata (L1) doesn't change Plan (L2) name just adds one or more SUT
• split into distinct SUT caused by slicing should not change Plan (L2) name neither
• when Plan runs on several SUT final plan report should be aggregated over all SUT it ran on
• tmt needs way to bind specific SUT to the plan
• in the end tmt should be able to take care of whole CI pipeline (from provision to report)

[sopos]

* in the end tmt should be able to take care of whole CI pipeline (from provision to report)

If the provisioning part was done externally tmt should provide uniqe SUT IDs to a provisioner to be able to provision --how connect to the specific SUTs later and run appropriate prepare and execute steps and test sets there.

I mean something like following flow:

1. tmt run provision --export --SUTs
2. provision SUTs for those IDs
3. tmt run provision --how connect --STUs <list of IDs> so it connects each generated ID to the expected SUT. This step should use cached metadata from step 1 to prevent inconsistencies

This way one could use any kind of provisioning method (e.g. beaker) which is even not supported by tmt itself and connect the machines in the expected way.

[psss]

I'll try to summarize the behaviour discussed above from the implementation point of view:

SUT (system under test) is defined by the provision and the prepare step configs. In addition to the explicit definition of SUT by the provision and prepare key in the plan, we would support detecting SUT during the discover step. There are several use cases for which SUT would be detected:

• For slice=1 each test would need a separate SUT
• Hardware requirements specified in L1 metadata would be used to group tests according to the required hardware
• For basic client-server multihost tests we could also create SUT with guest pairs (based on a dedicated L1 attribute)

Each SUT would be given a unique id. Each test would store id of the corresponding SUT on which it should be executed. The following steps would be performed:

• Provision and prepare all SUTs according to the detected configuration
• Execute tests on appropriate SUT
• Aggregate all test results (still under the single plan)

The question is how the detected SUT, that is the provision and prepare config, should relate to the one specified in the plan: Should it override it? Extend it? Or something else? Also we need to clarify how the interaction with the Testing Farm will work for this approach. @thrix, @happz, any thoughts here?

[happz]

Since someone put this issue on this week's hacking meeting, so I poked tmt internals a bit, just to see what would happen. Nothing complex, not even remotely close to tackling all raised use cases, I merely started with --max and especially how would cloning a plan multiple times look like. There will be internal, shared & mutable state we will need to get rid of, better start digging now than later. See #2907 for a very crude version, there will be updates & discussions.