POSIX Roadmap for LTSv3

[cfriedt]

Introduction

POSIX libraries and applications should Just Work™️ in 90% of the cases when being ported to Zephyr. Why? Because we want to enable our users to launch products and applications to the best of our ability, and we want to lower the barrier to entry by enabling users to leverage portable APIs.

This is an umbrella issue that will outlines work to be done in the POSIX API prior to LTSv3 (Ideally complete by v3.5).

Problem description

As discussed at EOSS 2023 and EOSS 2024, Zephyr's POSIX API layer needs some improvements. These can be broken down into the following:

1. Maintainability
2. Application / libc / toolchain Interface
3. Application / library Portability

Proposed change

Implement all Option Requirements of POSIX Subprofiling Option Groups required for PSE51, PSE52, and most of PSE53 compatibility.

Update the POSIX areas at docs.zephyrproject.org. For reference, the top level page of Zephyr's POSIX documentation.

https://docs.zephyrproject.org/latest/services/portability/posix/index.html

Maintainability addresses the needs of the maintainer, contributors, and (to some extent) the users.

Interface addresses the needs of

• the user (users develop POSIX code to the interface, not the implementation) (i.e. _POSIX_C_SOURCE, _XOPEN_SOURCE)
• the C Library maintainers - via Option Requirements (e.g. _POSIX_TIMERS)
• users - we do not need to re-doc the standardized POSIX API (i.e. via doxygen). However, users should rightfully expect to be able to find links to the standards that Zephyr conforms to, as well as the conformance gaps.

Portability addresses the needs of the users who should (rightfully) expect that the POSIX API layer behaves according to the specification.

Detailed RFC

Proposed change (Detailed)

Maintainability

• Support toolchain / libc overrides for POSIX Option Groups
  • posix: allow for external implementation of option groups #74184
• Abstract certain POSIX structures as integers
  • posix: use uint32_t instead of void* for pthread_t #51781
  • include: posix: move pthread impl detail to posix_internal.h #52003
  • posix: abstract pthread_mutex_t as uint32_t #52087
  • posix: abstract pthread_cond_t as uint32_t #52173
  • posix: abstract pthread_key_t as uint32_t #52313
  • posix: headers: fix remaining newlib picolibc zephyr pthread inconsistencies #52316
• Re-use Zephyr primitives inside of the POSIX API layer rather than dogfooding POSIX types
  • posix: pthread: leverage zephyr's synchronization and threading api #58924
  • posix: pthread: race condition between pthread_create() and pthread_join() #56163
  • posix: pthread: barrier functions should not directly invoke ksched.h API #58856
• Remove layering violations / cyclic dependencies between subsys/net, subsys/fs, and lib/posix
  • sockets/posix: poll()/select() implementation should be factored out from sockets subsys into POSIX subsys #24966
  • posix: standardize kconfig options #73047
  • posix: fd_mgmt: implement dup(), dup2(), fseeko(), and ftello() #74096
  • posix: device_io: implement fdopen(), fileno(), pread(), pselect(), pwrite() #73978
• Leverage a more standardized and compatible set of POSIX headers and Kconfig options
  • PicoLibC (base on Newlib) headers already exist in Zephyr SDK
  • Must copy-in for 3rd-party toolchains / libcs that do not support POSIX, so that Zephyr can still provide POSIX support
  • #include_next is required for the above
  • Insert link to PR for pulling in PicoLibc POSIX headers

• The POSIX Architecture and POSIX Subsystem are currently at odds
  • CONFIG_ARCH_POSIX and CONFIG_POSIX_API are mutually exclusive
  • this is primarily due to conflicting symbols in the C global namespace
  • [RFC] POSIX subsys vs libc: playing along or against each other #13787
  • [RFC] New host simulation target: Native simulator #58305
  • Allow CONFIG_POSIX_API on CONFIG_ARCH_POSIX #45100
  • posix: add kconfig and cmake for specifying application conformance #67181
• Remove CONFIG_NET_SOCKETS_POSIX_NAMES
  • introduced prior to CONFIG_POSIX_API in spite of the fact that the BSD Sockets API is part of POSIX
  • TODO insert PR for removing net sockets posix names

The layering issues that exist between subsys/net and lib/posix can be described as a cyclic dependency. The issue can be resolved via mutual dependency.

I.e. Currently, we have:

┌───────────┐       ┌───────────┐
│           ├──────►│           │
│   POSIX   │       │   NET     │
│           │◄──────┤           │
└───────────┘       └───────────┘

But we need

          ┌───────────┐
          │           │
          │   ZVFS    │
          │           │
          └▲────────▲─┘
           │        │
           │        │
┌──────────┴┐       ├───────────┐
│           │       │           │
│   POSIX   ├───────►   NET     │
│           │       │           │
└───────────┘       └───────────┘

Diagrams made with https://asciiflow.com/

I think we may have settled on the name "ZVFS" in the above diagram.

Interface

Tip

Check the Current status of Suprofiling Option Group support from docs to verify that the items below are up to date

• Include a clear reference to the POSIX standard in Zephyr's documentation
  • doc: posix: structural reorganization of posix docs #64540
• Plan for formally supporting Real-Time and Embedded Profiles of IEEE Std 1003.13-2003 (a.k.a. PSE52)
  • [RFC] Proposed development plan for Zephyr's POSIX subsystem #17706
  • Conformance Test Suites:
    • VSX4 POSIX IEEE 1003.1 (Note: test is designed for IEEE 1003.1-2001)
    • VSTH Threads
    • VSRT Real-Time
• With CONFIG_POSIX_API, Ensure POSIX headers can be included as #include <poll.h> rather than #include <zephyr/posix/poll.h>
  • posix: header fixups #51771
• POSIX_SINGLE_PROCESS
  • posix: implement confstr() #66866
  • posix: implement environ #66861
  • posix: implmement getenv() #66862
  • posix: implement setenv() #66863
  • posix: implement sysconf() #56670
  • posix: implement uname() #59924
  • posix: implement unsetenv() #66864
  • posix: env: support for environ, getenv(), setenv(), unsetenv() #66762
• POSIX_SIGNALS
  • posix: implement sigaction() #59925
  • posix: implement sigaddset() #59927
  • posix: implement sigdelset() #59928
  • posix: implement sigemptyset() #59929
  • posix: implement sigfillset() #59931
  • posix: implement sigismember() #59932
  • posix: implement sigpending() #59933
  • posix: implement alarm() #66923
  • posix: implement kill() #66924
  • posix: implement pause() #66925
  • posix: implement sigprocmask() #66928
  • posix: implement sigsuspend() #66929
  • posix: implement sigwait() #66930
• POSIX_DEVICE_IO
  • posix: device_io: implement fdopen() #66932
  • posix: device_io: implement fileno() #66938
  • posix: device_io: implement pread() #66946
  • posix: device_io: implement pselect() #66947
  • posix: device_io: implement pwrite() #66948
• POSIX_FD_MGMT
  • posix: fd_mgmt: implement dup() #74097
  • posix: fd_mgmt: implement dup2() #74098
  • posix: fd_mgmt: implement fseeko() #74099
  • posix: fd_mgmt: implement ftello() #74100
• POSIX_SEMAPHORES
  • posix: implement sem_close() #66955
  • posix: implement sem_open() #66956
  • posix: implement sem_unlink() #66957
• _POSIX_MESSAGE_PASSING
  • posix: implement mq_notify() #66958
• _POSIX_PRIORITY_SCHEDULING
  • posix: implement sched_getparam() #66959
  • posix: implement sched_getscheduler() #66960
  • posix: implement sched_rr_get_interval() #66961
  • posix: implement sched_setparam() #66962
  • posix: implement sched_setscheduler() #66963
• _POSIX_READER_WRITER_LOCKS
  • posix: implement pthread_rwlockattr_getpshared() #66964
  • posix: implement pthread_rwlockattr_setpshared() #66965
• _POSIX_THREAD_PRIORITY_SCHEDULING
  • posix: implement pthread_attr_getinheritsched() #66966
  • posix: implement pthread_attr_getscope() #66967
  • posix: implement pthread_attr_setinheritsched() #66968
  • posix: implement pthread_attr_setscope() #66969
• _POSIX_TIMEOUTS
  • posix: implement mq_timedreceive() #66970
  • posix: implement mq_timedsend() #66971
  • posix: implement posix_trace_timedgetnext_event() #66972
• _XOPEN_STREAMS
  • posix: implement fattach() #66973
  • posix: implement fdetach() #66974
  • posix: implement getmsg() #66975
  • posix: implement getpmsg() #66976
  • posix: implement isastream() #66977
  • posix: implement putmsg() #66978
  • posix: implement putpmsg() #66979
• _POSIX_SPIN_LOCKS (mandatory for the base specification)
  • kernel + posix: add k_spin_trylock(), pthread_spin_lock(), etc #59911
  • posix: implement pthread_spin_destroy()
  • posix: implement pthread_spin_init()
  • posix: implement pthread_spin_lock()
  • posix: implement pthread_spin_trylock()
  • posix: implement pthread_spin_unlock()
  • posix: pthread: support POSIX spinlocks #59910
• _POSIX_THREADS
  • posix: implement pthread_atfork() #59934
  • posix: implement pthread_barrierattr_destroy() #59935
  • posix: implement pthread_barrierattr_init() #59936
  • posix: implement pthread_barrierattr_getpshared() #59937
  • posix: implement pthread_barrierattr_setpshared() #59939
  • posix: implement pthread_cleanup_pop() #59940
  • posix: implement pthread_cleanup_push() #59941
  • posix: implement pthread_equal() #59942
  • posix: implement pthread_kill() #59943
  • posix: implement pthread_sigmask() #59944
  • posix: implement pthread_setcanceltype() #59945
  • posix: implement pthread_testcancel() #59946
  • kernel: support dynamic thread stack allocation #44379
  • kernel: userspace: dynamic thread stack support #59773
  • lib: posix: dynamic stack support for pthread_create #44727
  • lib: posix: dynamic stack support for pthread_create() #25973
• _POSIX_CLOCK_SELECTION
  • posix: implement pthread_condattr_getclock() #59947
  • posix: implement pthread_condattr_setclock() #59948
  • posix: implement clock_nanosleep() #59949
• _POSIX_SHARED_MEMORY_OBJECTS
  • posix: implement mmap() #59950
  • posix: implement munmap() #59951
  • posix: implement shm_open() #59952
  • posix: implement shm_unlink() #59953
• _POSIX_CPUTIME
  • posix: implement clock_getcpuclockid() #59954
• _POSIX_TIMERS
  • posix: implement clock_getres() #59955
  • posix: implement timer_getoverrun() #59956
  • RFC: rework of high resolution timer, sleep, and posix clock APIs and subsystems #40099
  • New timeout API #17155
• _POSIX_REALTIME_SIGNALS
  • posix: realtime signals: implement sigqueue() #74993
  • posix: realtime signals: implement sigtimedwait() #74994
  • posix: realtime signals: implement sigwaitinfo() #74995
• _POSIX_SYNCHRONIZED_IO
  • implement fsync()
  • implement msync()
  • posix: implement fdatasync() #74455
• _POSIX_THREAD_PRIO_PROTECT
  • posix: implement pthread_mutex_getprioceiling() #74456
  • posix: implement pthread_mutex_setprioceiling() #74457
  • posix: implement pthread_mutexattr_getprioceiling() #74458
  • posix: implement pthread_mutexattr_setprioceiling() #74459
  • posix: implement pthread_mutexattr_setprotocol() #74460
• _POSIX_THREAD_SAFE_FUNCTIONS
  • posix: implement flockfile() #74461
  • posix: implement funlockfile() #74462
  • posix: implement ftrylockfile() #74463
  • posix: implement putc_unlocked() #74464
  • posix: implement putchar_unlocked() #74465
  • posix: implement getc_unlocked() #74466
  • posix: implement getchar_unlocked() #74467
  • posix: implement asctime_r() #74468
  • posix: implement ctime_r() #74469
  • posix: implement localtime_r() #74470
  • posix: implement getgrgid_r() #74471
  • posix: implement getgrnam_r() #74472
  • posix: implement getpwnam_r() #74473
  • posix: implement getpwuid_r() #74474
  • posix: implement readdir_r() #74475

• Address regular bugs and enhancement requests
  • No tests/samples covering socket read()/write() calls #25407
  • POSIX API is not portable across arches #51776
  • samples/net/sockets/socketpair does not run as expected #52360
  • lib: posix: sleep() does not return the number of seconds left if interrupted #52517
  • lib: posix: usleep() does not follow the POSIX spec #52518
  • posix: getopt: cannot use getopt() in a standard way #52749
  • posix: fnmatch: fix known bugs #55186
  • net: socketpair: fcntl non-functional with socketpair-provided sockets on posix arch #54996
  • POSIX stat function returns filesystem statistics, not file statistics #58911
  • net: socketpair: fcntl non-functional with socketpair-provided sockets on posix arch #54996
  • minimal libc + posix: add strerror and perror functions #46101
  • posix: some lib and test file renames. canonical header order. split kconfig into smaller pieces #60081

Dependencies

Concerns and Unresolved Questions

Alternatives

There was a previous massive PR that attempted to address a number of the goals at once (#43987), but it is impossible to review in the current state and needs to be broken down and simplified in separate individual PRs.