Make CONFIG_USERSPACE=y and CONFIG_CMAKE_LINKER_GENERATOR=y work together #85241

bjorniuppsala · 2025-02-05T16:00:41Z

This makes the CMake linker generator work with CONFIG_USERSPACE=y

It uses the features added in #84635 to make e.g. controlling which passes include what easier (the same changes are incorporated in this PR too).

I have added ways to funnel inputs from e.g. gen_kobject_placeholders.py and gen_app_partitions.py into the linker generator, in the ld-skeletons this is handled by using the the c preprocessor. I have tried to create general and reusable ways of gathering information from the intermediate linking passes easy and natural. Look at the "Functions for input from prev linker pass" commit for details.

I have split cmake/linker_script/arm/linker.cmake into multiple files to reflect how the ld-skeletons are structured.
There are few zephyr_linker_*() calls that have been changed and added to mirror what the ld-skeletons are doing for userspace. Despite this there are a fair number of differences and missing sections for linker generator, that are not related to userspace that remains to be fixed until it is a complete replacement for the skeletons.

This is currently only tested on gcc for cortex_m4, although I think the changes to make other cortex cores and toolchains working should not be too hard. The aim is to provide IAR toolchain support for CONFIG_USERSPACE, once the actual iar toolchain support is in.

Edit:
Note that the changes I propose here does not quite create a fully working solution. There are a number of further changes needed.
Most prominent may be proper handling of MPU_ALIGN(region_size) for sections defined by the cmake linker generator (see #86563), as is you have to use large enough values for e.g. CONFIG_CUSTOM_SECTION_MIN_ALIGN_SIZE or CONFIG_ARM_MPU_REGION_MIN_ALIGN_AND_SIZE to get proper alignment).

In the end this effort aims to provide CONFIG_USERSPACE support for the IAR toolchain

cmake/modules/extensions.cmake

CMakeLists.txt

cmake/linker_script/common/common-noinit.cmake

cmake/linker_script/common/kobject-data.cmake

cmake/linker_script/common/kobject-priv-stacks.cmake

pdgendt · 2025-02-06T10:06:25Z

cmake/linker_script/common/kobject-priv-stacks.cmake

License and copyright.

The copyright is somewhat unclear, since most of this is either refactorings (direct copy from othe rfiles) or, heavily based on the corresponding .ld files (which in turn does not have that info)

Do not resolve the conversation if a resolution is still pending. It's actually up the commenter to resolve the comments as per the guidelines.

cmake/linker_script/common/kobject-priv-stacks.cmake

pdgendt

License headers need to be accompanied with a copyright notice, I think.

CMakeLists.txt

cmake/linker/ld/ld_script.cmake

cmake/linker/xt-ld/target.cmake

cmake/linker_script/common/common-noinit.cmake

cmake/linker_script/common/common-ram.cmake

bjorniuppsala · 2025-02-06T16:51:43Z

License headers need to be accompanied with a copyright notice, I think.

I was afraid of that... The thing is that the files I am adding are either based on ld-skeletons or directly copy-pasted from linker.cmake, and there are missing copyright tags on many of these sources. So it is "hard" to add a fair copyright notice.

There are also many files (e.g. among include/linker/common/*.ld) that have license tag but not copyright.

cmake/linker/linker_script_common.cmake

cmake/linker_script/arm/linker.cmake

cmake/linker_script/common/kobject-data.cmake

cmake/linker_script/common/kobject-priv-stacks.cmake

cmake/linker_script/common/kobject-rom.cmake

tejlmand

Thanks for this, a lot of great work.

First round of review comments ready, though I haven't been through everything in details so I might notice more things later.
I also want to carry out some testing on this.

So far mostly minor observations, where the only main concern found is the use of get_property() which means code becomes CMake configure time process order dependent and thereby prevents application code to customize / extend the linker script generated.

cmake/linker/linker_script_common.cmake

tejlmand · 2025-02-17T11:57:53Z

cmake/modules/extensions.cmake

+#                       You can have mulitple PASS <name> (in which the section will
+#                       be active in all of the phases) or you can have multiple
+#                       PASS NOT <name> in which case the section will be active in
+#                       passes other than any of the <name>.


because I've read the implementation later, then this becomes unclear.

If just reading this description, then multiple PASS <name> could mean this is allowed:
PASS foo PASS bar PASS baz, similar with PASS NOT <name>, however this is not the implementation, which seems to allow multipl <name> or NOT <name>, like this foo bar baz or NOT foo NOT bar NOT baz.

I think we should just allow the forms:

PASS <name> ..., example PASS foo bar baz.

PASS NOT <name> ..., example PASS NOT foo bar baz.

as there seems to be no reason for the redundant NOT.
That will also simplify the implementation check as you only need to check for NOT at the first index in the list.

Absolutely! I changed to follow your proposal.

As I used it I repeated the PASS NOT foo like so:

zephyr_linker_section(... PASS NOT foo PASS NOT bar)

tejlmand · 2025-02-17T11:59:51Z

cmake/modules/extensions.cmake

+# Internal helper that checks if we have consistent PASS arguments.
+# If the first element is NOT, check that all even elements are NOT, otherwise
+# check that no element is NOT
+macro(zephyr_linker_check_pass_param function_name SECTION_PASS)


function_name is unused, but I guess this whole check can be removed / simplified if following the proposal in the function description.

I left the function and removed the parameter. Maybe it is too ambitious to do these checks ?

tejlmand · 2025-02-17T13:51:49Z

cmake/linker/xt-ld/target.cmake

+    # This should probably do something like:
+    # set(cmake_linker_script_settings ${PROJECT_BINARY_DIR}/include/generated/ld_script_settings_${linker_pass_define}.cmake)
+    # zephyr_linker_generate_linker_settings_file(FILE ${cmake_linker_script_settings})
+    # And pass the file to the generator
+


I think a github enhancement issue proposing to switch to zephyr_linker_generate_linker_settings_file() would be better.

Ill do that instead

tejlmand · 2025-02-17T13:57:53Z

cmake/modules/extensions.cmake

 endfunction()

+# Usage:
+#   zephyr_linker_generate_linker_settings_file([FILE <name>] [STRING <string>])


I guess you had one intention when initially creating this function and then later made adjustments to support more use-cases, and during that process didn't take a second look at the function name.

Saving file in the function name is perhaps not optimal when it's possible to get the function to return content in a variable.

Similar with generate, as the string output is actually not generating anything.

Yep, I removed the string thing.

tejlmand · 2025-02-17T14:07:34Z

cmake/modules/extensions.cmake

+# Usage:
+#   zephyr_linker_generate_linker_settings_file([FILE <name>] [STRING <string>])
+#
+# Add the content generated so far by the zephyr_linker_* functions to a file


generated so far is not correct.

When using file(GENERATE ...) then content is not written until CMake generator time, which again means that settings AFTER zephyr_linker_generate_linker_settings_file() is also taken into consideration.

For example

zephyr_linker_generate_linker_settings_file(FILE foo.txt) zephyr_linker_symbol(SYMBOL __foo EXPR "(@__bar@)")

will actually handle the late zephyr_linker_symbol() call also and thus allowing applications to adjust or create symbols and sections as they like.

This is the reason generator expression are used.

Well spotted! And fixed.

cmake/modules/extensions.cmake

tejlmand · 2025-02-17T14:14:04Z

cmake/linker/linker_script_common.cmake

+      # can't warn here because we can't check for what is relevant in this pass
+      # message(WARNING "Missing definition for ${match}")
+    endif()
+    message("Replacing ${match} with ${value}")


I could imagine this would result in a lot of general message output which the user has no idea what is about or what to use with it.

True! I guarded this on CMAKE_VERBOSE_MAKEFILE

cmake/modules/extensions.cmake

CMakeLists.txt

tejlmand · 2025-02-28T13:29:04Z

cmake/modules/kconfig_extensions.cmake

Take a look at the comment here: #85241 (comment)

I will, i will. Take it easy ;)

I will, i will. Take it easy ;)

I wasn't trying to push you, just wanted to put a direct pointer to the related comment for this file, because sometimes when number of review comments grow large then new comments to older discussions are overlooked, so trying to be helpful 🙂

tejlmand · 2025-02-28T14:05:19Z

cmake/linker/linker_script_common.cmake

+    elseif(${INC_FILE} MATCHES "^.*\.cmake$")
+      include(${INC_FILE})
+    elseif(${INC_FILE} MATCHES "^.*\.h$")
+      list(APPEND PREPROCESSOR_FILES ${INC_FILE})


this is hiding what filetypes is actually supported, perhaps it's better to be explicit here:

zephyr/cmake/modules/extensions.cmake

Lines 5314 to 5317 in d86f703

# KCONFIG : Include Kconfig via import_kconfig

#

function(zephyr_linker_include_generated)

set(options "KCONFIG")

So that the developers wanting to use zephyr_linker_include_generated() can actually see directly that Kconfig, c-headers, and cmake files are supported.

Fixed (by replacing the FILE tag with KCONFIG, HEADER or CMAKE)

tejlmand · 2025-02-28T14:07:59Z

CMakeLists.txt

+# The linker generator also needs sections.h to be able to access e.g. _APP_SMEM_SECTION_NAME
+# for linkerscripts that do not support c-preprocessing.
+zephyr_linker_include_generated(FILE ${ZEPHYR_BASE}/include/zephyr/linker/sections.h)


not to be changed in this PR.

I consider that perhaps it's better to have developers being able to describe the content of sections.h in CMake or a template file which CMake can use, and from there generate header files, CMake files, and whatever else is needed by the build system.

CMake is better at generating header files than reading them 😉

Thoughts ?

I agree 100%. It would be a lot cleaner if the sections names and such coul be deifned together with the other linker stuff in cmake, rather than the other way around.

bjorniuppsala · 2025-03-03T13:00:42Z

Going from the current state of this PR there is a "big" question left, how to handle MPU_ALIGN(region_size).

I deferred that to an RFC: #86563, hoping for some more attention from interested parties.

Edit: there is an implementation in bjorniuppsala/iar-zephyr-fixes@953b7bb

tejlmand

Thanks for the update. Looks good and close to +1 .

Some minor observations, mainly related to the zephyr_linker_check_pass_param().
A function which we may even be able to replace with a very simple list() commadn and if() check.
Feel free to reply to observations.

tejlmand · 2025-03-13T14:48:17Z

cmake/modules/extensions.cmake

+  list(FIND SECTION_PASS "NOT" NOT_AT)
+  if(NOT("${NOT_AT}" STREQUAL "-1"))


can be simplified:

Suggested change

list(FIND SECTION_PASS "NOT" NOT_AT)

if(NOT("${NOT_AT}" STREQUAL "-1"))

if("NOT" IN_LIST SECTION_PASS)

tejlmand · 2025-03-13T14:58:41Z

cmake/modules/extensions.cmake

+  if(NOT SECTION_PASS)
+    message(FATAL_ERROR "At least one pass name required")


Usually in CMake it's acceptable to not provide a value when specifying an argument.

For example, like this:

zephyr_linker_include_generated(... PASS)

Reason for that is that it allows to simply do something like this:

zephyr_linker_include_generated(... PASS ${SOME_VAR})

without having to worry about whether SOME_VAR is defined or not.

With the proposed check, then you force callers into doing:

if(DEFINED SOME_VAR) zephyr_linker_include_generated(... PASS ${SOME_VAR}) else() zephyr_linker_include_generated(...) endif()

So unless there are specific reasons why values must be given, then I propose to skip this check.

I agree with the cmake-tradition/ease of use argument, but I think allowing an empty list of PASSes (due to undefined/empty arguments) might be inviting surprising results.

Logically this should mean that the section (or whatever) would not be included in any pass, which goes against the whole meaning of having the call. Making it mean the section is in all passes (as if it had no PASS parameter) is even more surprising.

E.g. for userspace there parts that are only enabled in a single pass. If those thing were to be included in all passes due to a case like you describe here the build will likely end up with duplicate definitions, silently wasted space, "or worse" :)

The risk of silently wasting space is key I think.

Those are fair points, and if checking for values to PASS argument is a conscious choice, then I approve the decision.

But please make this restriction clear in the description of the function.

In the end I think we'll stick with your proposal. PASS means not active anywhere. This should be "safe" from a waste perspective...

cmake/modules/extensions.cmake

tejlmand · 2025-03-13T15:28:58Z

cmake/modules/extensions.cmake

+endfunction()
+
+# Usage:
+#   zephyr_linker_generate_linker_settings_file([FILE <name>])


You are not checking for GEN_FILE before calling file(GENERATE ...) below, which I guess means FILE argument is now mandatory.

Suggested change

# zephyr_linker_generate_linker_settings_file([FILE <name>])

# zephyr_linker_generate_linker_settings_file(FILE <name>)

tejlmand

LGTM, no blocking comments.

But please take a look at this comment: #85241 (comment)

Make it possible to have multiple PASS parameters to zephyr_linker_section() and zephyr_linker_section_configure() sections (oring them) OR to have multiple PASS NOT p options (in which case the sections applies in neither of the passes) Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

Add MIN_SIZE and MAX_SIZE options to zephyr_linker_section() and zephyr_linker_section_configure(). This allows padding (for MIN_SIZE) and link-time checking (for MAX_SIZE) of sections (and parts of sections). Clarify comments for zephyr_linker_section_configure Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

Implement the MIN_SIZE and MAX_SIZE options for the ld linker file generator Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

Move creation of the generator arguments file from each target.cmake into a function (zephyr_linker_generate_linker_settings_file). Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

@foo

Add methods for the linker-script generator to pick up information from previus linkerpasses: * zephyr_linker_include_generated(FILE f.cmake [PASS X] [KCONFIG]) cmake-includes f.cmake in the relevant PASSes. This allows scripts to generate any content (e.g. sections or section configs) as if it was generated by the main cmake machiery. This is intended to cover the case of ld-script snippets generated by e.g. gen_app_partitions.py It can also parse KCconfig files to capture @CONFIG_FOO@ variables * @foo[,undef:v]@ in an value or expression in the zephyr_linker_* inputs. This looks for a defintion of FOO (se below), and uses its value if found. If FOO is not defined, the whole @@ thing is left untouched, unless ,undef: is used, in which case v is used instead. * zephyr_linker_include_generated(FILE f.h [PASS x]) greps for f.h for #define FOO value to be accessed as @foo@ * zephyr_linker_include_var(VAR FOO VALUE V) allows the main cmake-script to set a @foo@ variable. Note that the #define support is VERY basic, and does not use a proper pre-processor. It works for the current use of files generated by e.g. gen_kobject_placeholders.py but is not a general solution. Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

Tweak gen_app_partitions.py to be able to generate cmake linker generator output as well as ld-script fragments. Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

All in all this makes cmake linker generator able to work for at least most of the kernel tests and samples, on cortex m4. Make the cmake linker generator have a file-structure more similar to the ld-skeletons. Add or edit groups, sections and logic to make the generator reflect what the ld-skeletons do, esp. for kobjects and APP_SMEM, nonint and some other details are also effected. Use the new zephyr_linker_include_generated() and zephyr_linker_include_var() functions to to handle ${DEVICE_API_LINKER_SECTIONS_CMAKE}, the kobject-prebuilt-*.h files and APP_SMEM partition. Essentially the output from gen_app_partitions.py, gen_kobject_placeholders.py. Add ALIGN_WITH_INPUT on sections being put into DATA_REGION. This makes the init layout work for ld. This leverages the updates in gen_app_partitions.py to generate its output as cmake linker generator sections too, and puts them into a group defined in linker.cmake Setup generator variables for alignment of APP_SMEM. Note that this does not yet handle MPU_ALIGN which depends on the size of the section... Fix broken k_object_assignment iterable section Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

It seems as if ld_script.cmake hasnt handled zephyr_linker_section_configure( ALIGN ...) due to a typo in a variable name, Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

For CONFIG_USERSPACE the input from gen_app_partitions.py there is a need to be able to specify input files as well as input sections patterns for zephyr_linker_section_configure(). This is used for app partitions from libraries (which generate input patterns like foo.a:*(.data*)). This adds documentation to zephyr_linker_section_configure() to clarify what INPUT allows, and also adds some parsing tricks to ld_script.cmake to properly add the file patterns when they are missing. Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

bjorniuppsala · 2025-03-20T09:21:39Z

I would appreciate input or approvals.

RobinKastberg

LGTM

bjorniuppsala · 2025-03-21T14:26:07Z

With CONFIG_CMAKE_LINKER_GENERATOR=y test/kernel/threads/tls is still broken due to missing alignment of _t{data,bss}_size.
There is currently no way to express the end-alignment that works for ld.

Given that we lack MPU_ALIGN anyways, I'll work on a followup as a separate PR.

zephyrbot added area: Userspace Userspace area: Debugging area: Build System area: Toolchains Toolchains labels Feb 5, 2025

zephyrbot requested review from 57300, ceolin, dcpleung, jeremybettis, nashif, nordic-krch, nordicjm, stephanosio and tejlmand February 5, 2025 16:01

zephyrbot assigned tejlmand Feb 5, 2025

bjorniuppsala force-pushed the linker_generator_vs_userspace branch 3 times, most recently from d248306 to 368d77e Compare February 5, 2025 16:37

This was referenced Feb 6, 2025

tests: kconfig: cmake: Disable USERSPACE when using LINKER_GENERATOR #85075

Closed

USERSPACE and CMAKE_LINKER_GENERATOR #85277

Closed

pdgendt reviewed Feb 6, 2025

View reviewed changes

bjorniuppsala force-pushed the linker_generator_vs_userspace branch from 368d77e to 088b51b Compare February 6, 2025 15:41

zephyrbot added the area: Linker Scripts label Feb 6, 2025

pdgendt reviewed Feb 6, 2025

View reviewed changes

bjorniuppsala force-pushed the linker_generator_vs_userspace branch 3 times, most recently from dcb3723 to c8ee809 Compare February 7, 2025 08:58

pdgendt reviewed Feb 7, 2025

View reviewed changes

bjorniuppsala force-pushed the linker_generator_vs_userspace branch from c8ee809 to fd14acd Compare February 7, 2025 15:30

tejlmand requested changes Feb 17, 2025

View reviewed changes

bjorniuppsala force-pushed the linker_generator_vs_userspace branch 3 times, most recently from e574dae to d86f703 Compare February 28, 2025 09:40

tejlmand reviewed Feb 28, 2025

View reviewed changes

bjorniuppsala force-pushed the linker_generator_vs_userspace branch 2 times, most recently from 3260fd4 to 8f6de57 Compare March 4, 2025 10:32

bjorniuppsala requested review from pdgendt and tejlmand March 12, 2025 11:54

tejlmand reviewed Mar 13, 2025

View reviewed changes

tejlmand previously approved these changes Mar 14, 2025

View reviewed changes

bjorniuppsala added 9 commits March 14, 2025 13:42

cmake: linker generator: ld: Implement MIN/MAX_SIZE for sections in ld

ca1d3c3

Implement the MIN_SIZE and MAX_SIZE options for the ld linker file generator Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

cmake: linker generator: Refactor generator input file creation

c478c22

Move creation of the generator arguments file from each target.cmake into a function (zephyr_linker_generate_linker_settings_file). Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

scripts: Make gen_app_partitions.py cmake linker generator savvy

fcde63a

Tweak gen_app_partitions.py to be able to generate cmake linker generator output as well as ld-script fragments. Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

cmake: linker generator: ld: Fix alignment bug

55c3678

It seems as if ld_script.cmake hasnt handled zephyr_linker_section_configure( ALIGN ...) due to a typo in a variable name, Signed-off-by: Björn Bergman <bjorn.bergman@iar.com>

bjorniuppsala dismissed tejlmand’s stale review via cbe9308 March 14, 2025 12:50

bjorniuppsala force-pushed the linker_generator_vs_userspace branch from 8f6de57 to cbe9308 Compare March 14, 2025 12:50

bjorniuppsala requested a review from tejlmand March 14, 2025 13:59

bjorniuppsala mentioned this pull request Mar 20, 2025

cmake linker generator - multi pass and min max size for linker sections #84635

Closed

RobinKastberg approved these changes Mar 21, 2025

View reviewed changes

tejlmand approved these changes Mar 27, 2025

View reviewed changes

kartben merged commit fe0b165 into zephyrproject-rtos:main Mar 27, 2025
31 checks passed

bjorniuppsala deleted the linker_generator_vs_userspace branch March 28, 2025 09:19

	# KCONFIG : Include Kconfig via import_kconfig
	#
	function(zephyr_linker_include_generated)
	set(options "KCONFIG")

		list(FIND SECTION_PASS "NOT" NOT_AT)
		if(NOT("${NOT_AT}" STREQUAL "-1"))

	list(FIND SECTION_PASS "NOT" NOT_AT)
	if(NOT("${NOT_AT}" STREQUAL "-1"))
	if("NOT" IN_LIST SECTION_PASS)

		if(NOT SECTION_PASS)
		message(FATAL_ERROR "At least one pass name required")

	# zephyr_linker_generate_linker_settings_file([FILE <name>])
	# zephyr_linker_generate_linker_settings_file(FILE <name>)

Make CONFIG_USERSPACE=y and CONFIG_CMAKE_LINKER_GENERATOR=y work together #85241

Make CONFIG_USERSPACE=y and CONFIG_CMAKE_LINKER_GENERATOR=y work together #85241

Uh oh!

Conversation

bjorniuppsala commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pdgendt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bjorniuppsala commented Feb 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tejlmand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorniuppsala Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjorniuppsala commented Feb 5, 2025 •

edited

Loading

bjorniuppsala Feb 18, 2025 •

edited

Loading

bjorniuppsala commented Mar 3, 2025 •

edited

Loading

bjorniuppsala Mar 14, 2025 •

edited

Loading