Allow CONFIG_POSIX_API on CONFIG_ARCH_POSIX

[cfriedt]

Is your enhancement proposal related to a problem? Please describe.
Problem:

1. CONFIG_ARCH_POSIX and CONFIG_POSIX_API should not be mutually exclusive
2. the POSIX arch should provide a scalable, portable, and maintainable abstraction layer

Intent:
With the POSIX arch, we want to abstract away the host OS system calls (and parts of the system C library) in order to

1. efficiently test Zephyr code paths in CI
2. provide devs with a native environment for prototyping and debugging new features that use Zephyr code paths

For further details on intent see the docs

Clarification on CONFIG_POSIX_API:

1. This is the implementation of a subset of the POSIX API to support Zephyr's needs
   i. not strictly adhering the the embedded POSIX profile
2. The Zephyr application should interface with the Zephyr-provided implementation (with suitable platform-level abstractions)
3. In the case of CONFIG_ARCH_POSIX, The Zephyr application should not interface directly to the host OS system calls / or libc implementations

The native_posix port used to rely on posix_cheats.h to perform some questionable preprocessor redefinitions of standard function names in order to avoid conflicts in the C global symbol namespace. This was obviously not portable and somewhat fragile.

After some time, it was decided that CONFIG_ARCH_POSIX and CONFIG_POSIX_API were mutually exclusive options (see #13054, #13540).

This proposal is intended to show that CONFIG_ARCH_POSIX and CONFIG_POSIX_API do not need to be (and really should not be) mutually exclusive, and that functionality can be achieved with conventional OOP methods and very mature, standard APIs.

Moreover, the proposed solution is completely scalable, maintainable, and portable to any OS (likely even including non-POSIX operating systems).

Describe the solution you'd like

1. Provide a formal definition of abstractions required today to support native_posix
   i. for now this can be limited to C function & symbol definitions
2. An abstraction that works for Linux (and most any other OS) that uses
   i. portable and standard C
   i. standard POSIX APIs (e.g. dlopen(), dlsym(), can be extended to e.g. Windows API)
   i. conventional OOP methods (encapsulation / abstraction)
3. An (optional) secure method for loading host libc symbols
   i. leverage Kconfig to (optionally) provide the name or path of the host libc
   ii. leverage Kconfig to (optionally) require a strong checksum for the host libc

W.r.t. host libc name, on GNU systems, the default fault is obtainable programmatically with #include <gnu/lib-names.h> and LIBC_SO. However, it should be possible to link to other libc's as well.

It would probably make sense to encapsulate this abstraction on a per-OS bases by having separate compilation units (e.g. posix-arch-if-linux.c, posix-arch-macos.c).

Describe alternatives you've considered
Proposal in #24685

• both dynamic and static proposals rely on visibility attribute (ELF-specific)
• requiring a non-standardized feature increases technical debt
• immediately precludes macOS support for native_posix
• AMP is thrown into the mix (somewhat conflates intent)

There may be other RFCs on this subject

Additional context
This enhancement is a requirement / blocks #43987 which

• drastically simplifies using POSIX calls within Zephyr
• already demonstrates that CONFIG_POSIX_API and CONFIG_ARCH_POSIX are not mutually exclusive
• makes Zephyr more compatible for 3rd-party library code
• already has support from HAL / modules / networking
• provides a clear path forward for zephyr-namespaced implementations / POSIX wrappers

PR on its way

[aescolar]

For background, this topic was discussed, between others, here: #13054, and the PR that disabled the combination was #13540

About the purposes of native_posix: those are listed in the documentation. The "run Zephyr inside of a POSIX process in a production system (†)" is not one, and I very strongly discourage it for several reasons. @cfriedt, please, remove that from the description as that may mislead those who read it.

Please note that there is a (very common) misconception in the description that would be good fixing: "native_posix" is a target board, which is not the same (although it uses) the "posix architecture". It would be nice if you corrected it.

This matter does in any case not block the other PR you refer to, as you can simply leave the combination of POSIX ARCH and POSIX API shim disabled.
"already demonstrates that CONFIG_POSIX_API and CONFIG_ARCH_POSIX are not mutually exclusive" this was demonstrated long ago by the fact that it was enabled on tree for a long while. Please refer to the linked RFC for the motivation for disabling it.

About the "both dynamic and static proposals rely on visibility attribute (ELF-specific)", I did explain that this is not the case for the static library version proposal, but you seem to reiterate on it being so, so I would hope you can either back your statement or remove it. Regarding your other listed concerns, I do not agree with them: The 2nd line is not pointing at anything in particular. About the 3rd point , I do not see why the static library version would have per se any problem with macOS, but maybe you'd be so kind as to elaborate. Regarding AMP on simulated targets, that is something that some people have interest on, and that is enabled/made easier by that proposal; you may not have no interest in it, but I hope it does not prevent from understanding the proposal in itself, which can be applied fully ignoring the AMP use case.

[aescolar]

CC @pfalcon @daor-oti

[cfriedt]

For background, this topic was discussed, between others, here: #13054, and the PR that disabled the combination was #13540

@aescolar Thanks for the links - I'll add them to the summary.

About the purposes of native_posix: those are listed in the documentation. The "run Zephyr inside of a POSIX process in a production system (†)" is not one, and I very strongly discourage it for several reasons.

I strongly discourage it as well. It has been suggested verbally in the past, which is why I mentioned it as a hypothetical.

@cfriedt, please, remove that from the description as that may mislead those who read it.

Done.

This matter does in any case not block the other PR you refer to, as you can simply leave the combination of POSIX ARCH and POSIX API shim disabled.

I'll consider it. I would rather just solve this issue here and then allow the other PR to move forward.

About the "both dynamic and static proposals rely on visibility attribute (ELF-specific)", I did explain that this is not the case for the static library version proposal, but you seem to

I re-reviewed the static proposal and it still mentions "localizing hidden symbols (those not declared with default visibility)". Symbol locality, prelinking, objdump capabilities. A lot of that is ELF-specific.

anything in particular. About the 3rd point , I do not see why the static library version would have per se any problem with macOS, but

For the reasons mentioned above.

[aescolar]

About the "both dynamic and static proposals rely on visibility attribute (ELF-specific)", I did explain that this is not the case for the static library version proposal, but you seem to

I re-reviewed the static proposal and it still mentions "localizing hidden symbols (those not declared with default visibility)". Symbol locality, prelinking, objdump capabilities. A lot of that is ELF-specific.

In the static case the output of the "pre-link" stage is an object file. Local symbols are just not global symbols, meaning static int a; in translation unit scope is a "local" symbol. The idea of that proposal is that the {Zephyr OS + embOS + app + possible embedded libC} (without the native_posix + SOC + arch) are pre-linked together, (calls to the arch, soc and board are left as unresolved, everything else that is internal to those is already resolved in that link stage), then all global symbols (which were not otherwise marked to be kept as global) are made local so they don't conflict with further linking steps. The {native_posix + SOC + arch} are pre-linked together, and then linked to the previous pre-linked "embedded" code.

[cfriedt]

I re-reviewed the static proposal and it still mentions "localizing hidden symbols (those not declared with default visibility)". Symbol locality, prelinking, objdump capabilities. A lot of that is ELF-specific.

In the static case the output of the "pre-link" stage is an object file. Local symbols are just not global symbols,

By default, symbols are global unless declared static, in terms of plain-old C. Symbol "visibility" (at the compiler level) is an ELF construct (it is not available for Mach-O targets, iirc).

However, Mach-O does support tooling to categorize symbols as "exported" and "unexported" and the object format also supports differentiating between global and local.

#define	N_GSYM	0x20
#define	N_LSYM	0x80

https://opensource.apple.com/source/xnu/xnu-2050.18.24/EXTERNAL_HEADERS/mach-o/stab.h

meaning static int a; in translation unit scope is a "local" symbol. The idea of that proposal is that the {Zephyr OS + embOS + app + possible embedded libC} (without the native_posix + SOC + arch) are pre-linked together, (calls to the arch, soc and board are left as unresolved, everything else that is internal to those is already resolved in that link stage), then all global symbols (which were not otherwise marked to be kept as global) are made local so they don't conflict with further linking steps. The {native_posix + SOC + arch} are pre-linked together, and then linked to the previous pre-linked "embedded" code.

The rest of this does sound very application-specific. A similar concept that went in recently (or will be going in recently) is that of multi-target images - building a firmware image that includes e.g. application core, radio, or coprocessor firmware, all in one output file.

The AMP aspect of what you've proposed sounds like it has the same goals. E.g. with a bootloader deciding how to load code for each (co-) processor.

I think for the purposes of this issue, I would prefer to focus on just the Zephyr aspect of running CONFIG_POSIX_ARCH and CONFIG_POSIX_API in the same C namespace.