RFC: manifest: Add finegrained group filters

[aescolar]

This is not so much a real PR at this point, as much as a trigger to discuss about better project filtering in the default Zephyr manifest, and a proposal for a possible direction.

---

Several users are not happy about the size of a default
zephyr workspace, and about the means they have
to select which projects they have locally.

Add more fine grained groups (one per project), to enable
them to locally select which they want.

Users will still get the "default" group by default.
But if they want a small subset they can do
west config manifest.group-filter -- "-default, +proj_x, +proj_y"

---

How this commit is, I just left the old groups there just renamed to hal_all, tools_all, fs_all, crypto_all, debug_all, tee_all, but those groupings don't seem specially useful.
It would make sense to have other sets (disabled by default) based on real use-cases, like "I want to develop for NXP platforms", or "for Nordic platforms".

---

Related:

• Provide local project filter in west config west#653

---

This is the size of a workspace today (6a92ebf) after a plain west init + west update

838M	: manifest (zephyr)
802M	: hal_nxp (modules/hal/nxp)
612M	: hal_stm32 (modules/hal/stm32)
372M	: hal_espressif (modules/hal/espressif)
175M	: picolibc (modules/lib/picolibc)
157M	: lvgl (modules/lib/gui/lvgl)
141M	: hal_silabs (modules/hal/silabs)
130M	: openthread (modules/lib/openthread)
106M	: trusted-firmware-m (modules/tee/tf-m/trusted-firmware-m)
99M	: mbedtls (modules/crypto/mbedtls)
93M	: hal_infineon (modules/hal/infineon)
87M	: hal_atmel (modules/hal/atmel)
69M	: trusted-firmware-a (modules/tee/tf-a/trusted-firmware-a)
68M	: hal_nordic (modules/hal/nordic)
67M	: loramac-node (modules/lib/loramac-node)
57M	: thrift (modules/lib/thrift)
48M	: sof (modules/audio/sof)
39M	: hal_st (modules/hal/st)
33M	: hal_ti (modules/hal/ti)
30M	: psa-arch-tests (modules/tee/tf-m/psa-arch-tests)
21M	: tflite-micro (modules/lib/tflite-micro)
19M	: cmsis (modules/hal/cmsis)
17M	: net-tools (tools/net-tools)
17M	: chre (modules/lib/chre)
16M	: hal_gigadevice (modules/hal/gigadevice)
15M	: tf-m-tests (modules/tee/tf-m/tf-m-tests)
14M	: mcuboot (bootloader/mcuboot)
13M	: hal_rpi_pico (modules/hal/rpi_pico)
13M	: hal_microchip (modules/hal/microchip)
9.8M	: nanopb (modules/lib/nanopb)
7.7M	: hal_openisa (modules/hal/openisa)
7.6M	: lz4 (modules/lib/lz4)
6.9M	: hal_nuvoton (modules/hal/nuvoton)
6.1M	: zscilib (modules/lib/zscilib)
4.7M	: TraceRecorderSource (modules/debug/TraceRecorder)
4.0M	: fatfs (modules/fs/fatfs)
3.8M	: uoscore-uedhoc (modules/lib/uoscore-uedhoc)
3.1M	: zcbor (modules/lib/zcbor)
2.9M	: open-amp (modules/lib/open-amp)
2.8M	: hal_xtensa (modules/hal/xtensa)
2.8M	: canopennode (modules/lib/canopennode)
2.7M	: edtt (tools/edtt)
2.5M	: littlefs (modules/fs/littlefs)
2.4M	: hal_ethos_u (modules/hal/ethos_u)
2.3M	: mipi-sys-t (modules/debug/mipi-sys-t)
2.1M	: hal_telink (modules/hal/telink)
1.8M	: liblc3 (modules/lib/liblc3)
1.6M	: libmetal (modules/hal/libmetal)
1.6M	: hal_renesas (modules/hal/renesas)
1.6M	: hal_altera (modules/hal/altera)
1.4M	: nrf_hw_models (modules/bsim_hw_models/nrf_hw_models)
1.2M	: hal_quicklogic (modules/hal/quicklogic)
856K	: hal_wurthelektronik (modules/hal/wurthelektronik)
820K	: tinycrypt (modules/crypto/tinycrypt)
780K	: segger (modules/debug/segger)
420K	: bsim (tools/bsim)
--------------------------------------------------------------------------
4.2G	: total

[mbolivar-nordic]

@marc-hb has previously shown that treeless clones are a good way to decrease the size of the default workspace: zephyrproject-rtos/west#638

If we're going to chase something like this, I think it'd make sense to start there, using features built into git. If those are not good enough for known reasons, then I think it makes sense to think about something like this, or extending west to add other ways to deactivate projects.

[aescolar]

Thanks @mbolivar-nordic. Are you thinking we should guide in the documentation users to do something like west update -n -o=--depth=1 for CI instead of a full west update(?), or a west update -o=--filter=tree:0 , for normal users?
Those would alleviate some the download and disk size.
Though it also seems some people has a basic "I don't want to have all these repos I know I don't need" reaction.

---

A west update -n -o=--depth=1 following a normal west init ends up with a 3.5 GB workspace instead of 4.2 GB

[marc-hb]

I just measured --filter=tree:0 from scratch again. On my system the west update time goes down from 5 minutes to 3 minutes and the disk usage from 4.1G to 3.5G. This makes a very significant improvement for a CI running west update hundreds of times a day with none of the issues that shallow clones have. But for users downloading from scratch much more rarely I don't think it's worth the added documentation complexity, especially not at the very start of a tutorial.

Though it also seems some people has a basic "I don't want to have all these repos I know I don't need" reaction.

Also, west list noise is annoying.

TBH I don't understand why the default manifest has all repos enabled by default, this looks like some (misplaced?) convenience for CI. It should have only the repos needed by the official tutorial(s) enabled by default.

[aescolar]

How the PR is right now (fcdc676)
If, for example, I'm a developer interested only in what I want to work with a Nordic platform, I can do

west init -m git@github.com:aescolar/zephyr.git --mr west_groups zephyr_project
#this previous line would just be `west init`
cd zephyr_project/
west config manifest.group-filter -- "-default, +cmsis, +hal_nordic, +libmetal, +littlefs, +mbedtls, +mcuboot, +mcumgr, +segger, +tinycrypt, +zcbor, +open-amp"
west update

And end up with 1.1 GB instead of 4.2GB, and a workable workspace.

[aescolar]

west now supports project filters, including filtering with regular expressions matching project names. That provides users the type of functionality this hoped for without this kind of workaround.

• manifest.project-filter west#664