ci: testplan generation cleanup and scope overhaul #65257

nashif · 2023-11-15T18:53:19Z

originally tags were introduced to reduce the number of tests we run, so, if a driver is changed, there was no need to build kernel tests and such, so it was exclusion based. This means that we would end up building loads of irrelevant tests still because tags file does only cover a few areas.
Most importantly it was abstraction based, so it was not about running tests related to what has changed, instead just reduce the number of tests not guaranteeing coverage at all of what the PR is changing.

This new approach is based on the Maintainer file and connecting areas in that file with tests, if a PR changes one area, tests for this specific area is run. There are a few other steps taken to increase coverage and to deal with special cases.

change some code in some subsys or 2, then...
./scripts/ci/test_plan.py -c HEAD~1.. or whatever

for example, changed bindesc subsystem:

i9:zephyr(ci_plans_cleanup_2): ./scripts/ci/test_plan.py -c 6d40dca0323..
INFO: Changed files:
INFO: ///////////////////////////
INFO: subsys/bindesc/CMakeLists.txt
INFO: ///////////////////////////
INFO: -------------------  excludes --------------
INFO: files to be ignored: set()
INFO: not resolved files: ['subsys/bindesc/CMakeLists.txt']
INFO: -------------------  tests --------------
INFO: Remaining files: {'subsys/bindesc/CMakeLists.txt'}
INFO: -------------------  archs --------------
INFO: Remaining files: {'subsys/bindesc/CMakeLists.txt'}
INFO: -------------------  boards --------------
INFO: Remaining files: {'subsys/bindesc/CMakeLists.txt'}
INFO: -------------------  areas --------------
INFO: Remaining files: {'subsys/bindesc/CMakeLists.txt'}
INFO: file: subsys/bindesc/CMakeLists.txt
INFO: area Binary Descriptors changed..
INFO: /home/nashif/zephyrproject/zephyr/scripts/twister -c --test-config custom_config.yaml --level custom --save-tests _test_plan_partial.json --integration
Deleting output directory /home/nashif/zephyrproject/zephyr/twister-out
INFO    - Using Ninja..
INFO    - Zephyr version: zephyr-v3.5.0-1517-gbbbec63eca7c
INFO    - Using 'zephyr' toolchain.
INFO    - Selecting default platforms per test case
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/nashif/zephyrproject/zephyr/twister-out/testplan.json
INFO    - Writing JSON report _test_plan_partial.json
INFO: Added 17 suites to plan.
INFO: -------------------  post filters --------------
INFO: No twister needed or partial twister run only...
INFO: Total tests gathered: 17
INFO: Total tests to be run (after removing duplicates): 17
INFO: Total nodes to launch: 1

fabiobaltieri · 2023-11-17T11:03:51Z

Nice, MAINTAINER file approach seems like a great plan.

Do we really need the tags? Maybe the plan could just scan for sample/test yaml files under the area and add those to the plan regardless on the tag.

Was also thinking, maybe it would make sense to distinguish between platform changes and subystem changes and behave differently?

nashif · 2023-11-17T13:19:22Z

Do we really need the tags? Maybe the plan could just scan for sample/test yaml files under the area and add those to the plan regardless on the tag.

not tags right now, it is using test identifiers. There are other usecases I have in mind which will merge this data with test reports generated by twister to generate something that is feature based (feature = area). I do not want to depend on just the file globs for this, which is not reliable.

Was also thinking, maybe it would make sense to distinguish between platform changes and subystem changes and behave differently?

That is what the script does, no?

fabiobaltieri · 2023-11-20T11:17:08Z

scripts/get_maintainer.py

+                            ", ".join(area.tests),
+                            ", ".join(area.tags),


I'm a bit confused by the tags here, is this a transient? You are adding handling, then adding some tags to the maintainer file, then removing the tag and only using tests or am I missing something?

yes, started with tags but realized those are very random and inconsistent and do not provide direct relation and decided to look into test identifiers instead which are more accurate.

fabiobaltieri · 2023-11-20T11:17:44Z

That is what the script does, no?

Ok fair enough, sorry not familiar with the original code, need to put more time into understanding this, see my question above about tags+tests in the maintainer file.

fabiobaltieri · 2023-11-29T15:08:36Z

scripts/get_maintainer.py

@@ -479,7 +485,7 @@ def ferr(msg):

    ok_keys = {"status", "maintainers", "collaborators", "inform", "files",
               "files-exclude", "files-regex", "files-regex-exclude",
-               "labels", "description"}
+               "labels", "description", "tests", "tags"}


tags in this file can be removed right?

can, but leaving it does not hurt, I might have a usecase for this to improve coverage as we move forward.

fabiobaltieri

Having a hard time fining anything to comment about on the code and the history could be cleaned a bit :-), I think the whole file could use some restructuring, but so it did before, I think this is functionally a great improvement worth giving it a shot as is.

henrikbrixandersen · 2023-12-01T10:55:29Z

@nashif Needs rebasing.

erwango · 2023-12-11T14:10:00Z

MAINTAINERS.yml

@@ -1051,6 +1141,8 @@ Release Notes:
    - doc/hardware/peripherals/flash.rst
  labels:
    - "area: Flash"
+  tests:
+    - drivers.flash


Does it encompass sample.drivers.flash.shell ?

erwango

I like the idea.

MaureenHelm · 2024-02-08T16:28:21Z

@nashif coming back to this?

nashif · 2024-02-08T16:29:42Z

@nashif coming back to this?

yes, working on that now actually, it is not easy :(

Cleanup and refactoring of some functions and arguments. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

This used to make sense at some point when we had minimal emulation platforms and a few platforms. Building each changed tests for > 600 platforms just does not make sense anymore and does not scale. We should focus on emulation and running the tests rather than just randomly build them over and over again, sometimes when no functional changes are being introduced. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

- Do not depend on west.yaml changes - Change tag for cmsis-dsp to match what we have in tests and the module name in west.yaml. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Use Maintainer file for determining what tests to run in CI and stop using the limited tags file. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Create a custom level with all tests identified. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

This is no longer used or needed. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

When no commit are given, work on locally changed files. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

github-actions · 2024-04-13T00:29:38Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

github-actions · 2024-06-13T00:29:53Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

github-actions · 2024-08-18T00:32:06Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

github-actions · 2024-10-20T00:37:00Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

nashif force-pushed the ci_plans_cleanup_2 branch from 6d40dca to 8883123 Compare November 16, 2023 11:31

nashif requested a review from a team November 16, 2023 11:37

nashif force-pushed the ci_plans_cleanup_2 branch 2 times, most recently from a15fe5d to f50935c Compare November 16, 2023 11:40

fabiobaltieri reviewed Nov 20, 2023

View reviewed changes

fabiobaltieri mentioned this pull request Nov 21, 2023

drivers: serial: stm32u5: Serial wakeup is based on autonomous capability #65532

Merged

nashif force-pushed the ci_plans_cleanup_2 branch from f50935c to ac2b34f Compare November 28, 2023 15:17

nashif marked this pull request as ready for review November 28, 2023 15:18

zephyrbot added area: Continuous Integration area: CMSIS-DSP area: Process labels Nov 28, 2023

zephyrbot requested review from galak, MaureenHelm, stephanosio and XenuIsWatching November 28, 2023 15:19

zephyrbot assigned stephanosio Nov 28, 2023

nashif mentioned this pull request Nov 29, 2023

tests: posix: remove unrelated tags #65873

Merged

nashif force-pushed the ci_plans_cleanup_2 branch from ac2b34f to bac5fc0 Compare November 29, 2023 13:01

fabiobaltieri reviewed Nov 29, 2023

View reviewed changes

fabiobaltieri previously approved these changes Nov 29, 2023

View reviewed changes

henrikbrixandersen previously approved these changes Dec 1, 2023

View reviewed changes

erwango reviewed Dec 11, 2023

View reviewed changes

erwango previously approved these changes Dec 11, 2023

View reviewed changes

MaureenHelm previously approved these changes Dec 13, 2023

View reviewed changes

nashif dismissed stale reviews from MaureenHelm and erwango via 02567e9 December 14, 2023 15:04

nashif dismissed stale reviews from henrikbrixandersen and fabiobaltieri via 02567e9 December 14, 2023 15:04

nashif force-pushed the ci_plans_cleanup_2 branch 2 times, most recently from 02567e9 to 71948b1 Compare December 14, 2023 15:19

nashif mentioned this pull request Dec 20, 2023

add more maintainer file coverage #66702

Merged

nashif added 6 commits February 12, 2024 16:06

ci: testplan: cleanup

5d47105

Cleanup and refactoring of some functions and arguments. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

ci: testplan: rename cmsis-dsp tag

4e27da2

- Do not depend on west.yaml changes - Change tag for cmsis-dsp to match what we have in tests and the module name in west.yaml. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

ci: testplan: use MAINTAINERS file

7162768

Use Maintainer file for determining what tests to run in CI and stop using the limited tags file. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

ci: testplan: create custom level based on code changes

d4ce4ce

Create a custom level with all tests identified. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

ci: testplan: remove tag handling

3330b48

This is no longer used or needed. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif force-pushed the ci_plans_cleanup_2 branch from 71948b1 to ddac1db Compare February 12, 2024 21:06

zephyrbot requested a review from kartben February 12, 2024 21:07

ci: testplan: work on locally modified files

b3e624c

When no commit are given, work on locally changed files. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif force-pushed the ci_plans_cleanup_2 branch from ddac1db to b3e624c Compare February 12, 2024 21:09

github-actions bot added the Stale label Apr 13, 2024

nashif removed the Stale label Apr 13, 2024

github-actions bot added the Stale label Jun 13, 2024

nashif removed the Stale label Jun 13, 2024

nashif unassigned stephanosio Jun 18, 2024

github-actions bot added the Stale label Aug 18, 2024

nashif removed the Stale label Aug 20, 2024

github-actions bot added the Stale label Oct 20, 2024

nashif removed the Stale label Oct 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: testplan generation cleanup and scope overhaul #65257

ci: testplan generation cleanup and scope overhaul #65257

nashif commented Nov 15, 2023 •

edited

Loading

fabiobaltieri commented Nov 17, 2023

nashif commented Nov 17, 2023

fabiobaltieri Nov 20, 2023

nashif Nov 20, 2023

fabiobaltieri commented Nov 20, 2023

fabiobaltieri Nov 29, 2023

nashif Nov 29, 2023

fabiobaltieri Nov 29, 2023

fabiobaltieri left a comment

henrikbrixandersen commented Dec 1, 2023

erwango Dec 11, 2023

erwango left a comment

MaureenHelm commented Feb 8, 2024

nashif commented Feb 8, 2024

github-actions bot commented Apr 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Aug 18, 2024

github-actions bot commented Oct 20, 2024

ci: testplan generation cleanup and scope overhaul #65257

Are you sure you want to change the base?

ci: testplan generation cleanup and scope overhaul #65257

Conversation

nashif commented Nov 15, 2023 • edited Loading

fabiobaltieri commented Nov 17, 2023

nashif commented Nov 17, 2023

fabiobaltieri Nov 20, 2023

Choose a reason for hiding this comment

nashif Nov 20, 2023

Choose a reason for hiding this comment

fabiobaltieri commented Nov 20, 2023

fabiobaltieri Nov 29, 2023

Choose a reason for hiding this comment

nashif Nov 29, 2023

Choose a reason for hiding this comment

fabiobaltieri Nov 29, 2023

Choose a reason for hiding this comment

fabiobaltieri left a comment

Choose a reason for hiding this comment

henrikbrixandersen commented Dec 1, 2023

erwango Dec 11, 2023

Choose a reason for hiding this comment

erwango left a comment

Choose a reason for hiding this comment

MaureenHelm commented Feb 8, 2024

nashif commented Feb 8, 2024

github-actions bot commented Apr 13, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Aug 18, 2024

github-actions bot commented Oct 20, 2024

nashif commented Nov 15, 2023 •

edited

Loading