llext-edk: rework, add Python export and more optional information

[pillo79]

When building several different EDKs, there's currently no way to avoid mistaking one for another. The goal of this PR is to add metadata to a generated EDK so that the target board / SOC / CPU can be retrieved and checked by downstream projects.

An additional Python-compatible output format is provided for projects that do not use CMake or make.

To implement the above, a number of internal refactors with minimal visible effect have been implemented, with the goal of having more maintainable and extendable code:

• the EDK output commands have been refactored into generic functions that take the target file type as a parameter;
• flag processing has been made target-agnostic by using placeholders that are expanded in the output functions.

LLEXT EDK documentation has been updated accordingly.

EDIT: updated after review notes.

[pillo79]

v2:

• Fixed a copy-paste error in placeholder expansion.

[tejlmand]

I'm not convinced of two things:

• Why should we suddenly start exporting a bunch of extra variables ?
  No detailed explanation is given, and the closest I can see seems to be because of informational use. So where does this end ?
  Are we going to maintain a list in top-level CMakeLists of all Zephyr build settings.
• Exporting to bash.
  Starting to export to specific shells seems like setting a bad precendence.
  We provide CMake and Makefiles, and any downstream build system should be able to cope with those or easily wrap them to set them in their favorite shell instead.

[tejlmand]

this is growing out of hands and easily becomes unmaintainable, not to mention putting undesired restrictions on Zephyr build system internals.

For the first part, starting to have a long list of settings being maintained at the toplevel CMakeLsts.txt is not ok.
What started as a small and maintainable list which is called at a time where things are known to be defined, such as: ZEPHYR_BASE, WEST_TOPDIR was acceptable, but a constant growing list for llext is not.

Secondly, exporting Zephyr internals as-is start to mean that Zephyr cannot control this.
Are we starting to promise that we will never rename those variables because they are being exported ?

Although there is no intention of renaming SOC_NAME, SOC_SERIES, BOARD_REVISION, etc. then if this change is accepted, then it sets a precedence that any variable might enter this list.

I know there is a wish for a more controlled and well-defined way of exporting Zephyr internals where names are promised to be stable, especially for CI and IDE integration.
And for that reason work is ongoing to try and establish a common output file for build info.

Perhaps this can be useful for llext to avoid above bloat in CMakeLists.txt.

[pillo79]

Fair points, I will revert this change and simply add the few missing variables I'd like to have exported to the EDK command line.

work is ongoing to try and establish a common output file for build info.

I am also very interested in this. Any pointer is appreciated 🙂

[tejlmand]

sure, it's now opened: #79118

[tejlmand]

exactly why are all those details needed ?

Shouldn't it be sufficient to have the board target ?

Or are you starting to replicate the Zephyr build system outside of Zephyr ?
If that's the case, then perhaps integrate to sysbuild is a better choice.

[pillo79]

In order of importance, I'd say:

• NORMALIZED_BOARD_TARGET is the really interesting bit, as it is extremely specific to each EDK build. Otherwise, the EDK exports no metadata that can correlate itself with a board target.
• CROSS_COMPILE_TARGET is very useful as well, as it identifies the specific compiler used and whose CFLAGS are exported by the EDK.
• I have also used ARCH, BOARD, and to a lesser extent SOC_NAME, to logically group different EDKs.

The other variables were added only for "completeness"; sorry for not thinking about the maintenance burden.

[tejlmand]

• CROSS_COMPILE_TARGET is very useful as well, as it identifies the specific compiler used and whose CFLAGS are exported by the EDK.

and this exactly proves my worries of exporting those fields like this, because not all toolchains set this variable and we make no promises. But suddenly we could need to fix "bugs" because something which is intended for internal use is being exported.

An example of a toolchain where this variable is not defined is LLVM.

[tejlmand]

• I have also used ARCH, BOARD, and to a lesser extent SOC_NAME, to logically group different EDKs.

Be careful on using arch for grouping, because a board could have multiple SoCs of different archs or a multi-core SoC where the cores are of different arch. Of course this depends on which level the arch is place, but Zephyr has just gone through a rework to avoid arch on high level organization 😉

[tejlmand]

I don't think we should start exporting to specific shells.
This is a build system, and thus exporting CMake and Makefiles seems ok, but starting to export to bash is going to just bloat our interface.

What about bat files ? fish ? or the favorite shell of some other user ?
If an external build tool is not satisfied with CMake or Makefiles then I would leave it for that tool to have a tool which can read the file and convert to its favorite shell.

Only additional file format I consider acceptable is yaml, because that is a file format Zephyr build system already uses for dumping data, for example in the Zephyr meta file creation.

[pillo79]

And to think I was about to add Python export as well... 😅 I'm surprised this would be considered "bloat" - it's only for the EDK, it's a few lines of code and it just makes the life easier for simple downstream projects - but I understand.
I'll remove this and use the current CMake export to generate the same script in my project.

[tejlmand]

I'm surprised this would be considered "bloat"

when I'm saying bloat then i'm thinking in terms of what we promise externally.
If we provide bash here, should we then also provide this elsewhere, and what about xyz.

And to think I was about to add Python export as well...

had you done it the other way, then a Python might have made it through 😆
because we already use Python a lot, and Python is cross-platform (Linux, Windows, MacOS, etc.), but with Python then a yaml file would have been just as easy / nice to use 😉

Only additional file format I consider acceptable is yaml

[pillo79]

The objective of this PR is to add metadata to an EDK build so that the target board / SOC / CPU can be retrieved and checked by downstream projects. In my case I will require building several different EDKs, and I realized there's no way to avoid mistaking one for another.

I understand your concerns, and will amend the proposal accordingly.

[pillo79]

In order of importance, I'd say:

• NORMALIZED_BOARD_TARGET is the really interesting bit, as it is extremely specific to each EDK build. Otherwise, the EDK exports no metadata that can correlate itself with a board target.
• CROSS_COMPILE_TARGET is very useful as well, as it identifies the specific compiler used and whose CFLAGS are exported by the EDK.
• I have also used ARCH, BOARD, and to a lesser extent SOC_NAME, to logically group different EDKs.

The other variables were added only for "completeness"; sorry for not thinking about the maintenance burden.

[pillo79]

Fair points, I will revert this change and simply add the few missing variables I'd like to have exported to the EDK command line.

work is ongoing to try and establish a common output file for build info.

I am also very interested in this. Any pointer is appreciated 🙂

[pillo79]

And to think I was about to add Python export as well... 😅 I'm surprised this would be considered "bloat" - it's only for the EDK, it's a few lines of code and it just makes the life easier for simple downstream projects - but I understand.
I'll remove this and use the current CMake export to generate the same script in my project.

[teburd]

All seems very reasonable, nice!

[pillo79]

v3, incorporating @tejlmand notes:

• Bash is now Python 🙂
• The list of exported variables is static and considerably reduced
• Both the code and the documentation clearly state the added variables reflect Zephyr internals and no stability guarantees are implied.

[pillo79]

@tejlmand can you please revisit this? Thanks!

[tejlmand]

this doesn't look like yaml to me 😉
ref: #78727 (comment)

then a yaml file would have been just as easy / nice to use 😉

Only additional file format I consider acceptable is yaml

remember, yaml can be read directly into Python dict and is much nicer to work with.
We actually have a yaml module in Zephyr CMake now 😄

[tejlmand]

sure, it's now opened: #79118

[tejlmand]

v3, incorporating @tejlmand notes:

* Bash is now Python 🙂

* The list of exported variables is static and considerably reduced

* Both the code and the documentation clearly state the added variables reflect Zephyr internals and no stability guarantees are implied.

Thanks, but before going into details again on this one, then I have opened #79118 as promised.

I know we currently still have the CMake and Makefile export, but I was thinking that it could be beneficial to cover the Python export functionality you need by expanding the schema with the extra info by using the according build_info() function where needed.
This will also move knowledge / ever growing list of settings away from top-level CMakeLists.txt and closer to where the vars are actually defined and used (and thus reducing maintenance on those).

Later we can then update CMake / makefile generation to actually use the information from the build info infrastructure instead of having to keep track of this manually, but that I would take ast a later stage.

Please take a look at #79118 in relation to this PR and give your initial thoughts.
😄