posix: add support for mmap, memlock, shared memory, and mprotect

[cfriedt]

• add support for _POSIX_MAPPED_FILES
• add support for _POSIX_MEMLOCK_RANGE
• add support for _POSIX_MEMLOCK (currently unimplemented / fails with ENOSYS)
• add support for _POSIX_SHARED_MEMORY_OBJECTS
• add support for _POSIX_MEMORY_PROTECTION (currently unimplemented / fails with ENOSYS)

Shared Memory Objects and Memory-Mapped Files

On systems that support mapping anonymous memory (i.e. have an MMU and lots of RAM), shared memory objects are implemented using k_mem_map() and friends (i.e. a shared memory object is comprised of a number of blocks of virtual memory that are CONFIG_MMU_PAGE_SIZE in size). On systems that do not have an MMU, shared memory objects are implemented with heap allocation.

Note: it is up to the particular vtable implementor, to support the ZFD_IOCTL_MMAP operation. However, there would still need to be some integration with msync() in order to synchronize writes to e.g. a regular file. From that perspective, shared memory objects were an easy prototyping option.

A potential future improvement on shared memory objects would be to support a separate ioctl() to attach statically allocated memory.

z_finalize_typed_fd()

A particular change to highlight in this case though, is adding a mode field to struct fd_entry, which allows fdtable code to identify which subsystem or API a file descriptor belongs to, and to react accordingly. The z_finalize_typed_fd() function allows us to differentiate between fifo, char device, message queue, directory, semaphore, block device, shared memory, regular file, symbolic link, and socket.

In fact, the same 32-bit mode field should also have the file access mode tied to it, rather than the physical file (which is currently the case now for the FS subsystem, and should be considered a bug). It's trivial to consider a case where one file descriptor is used for writing to a file and another file descriptor is used for reading from a file. Both file descriptors point to the same physical file, but each has different access modes and offsets (addressed below).

Per-file Offsets

Another similar adjustment to struct fd_entry is to have a per-fd (per FILE*) offset, which is just consistent with the C/C++, POSIX (all?) programming models of "files".

A few drive-by fixes aside from that, shuffling things between zvfs, posix, and fs, as-needed.

Doc Preview

Fixes #59950
Fixes #59951
Fixes #59952
Fixes #59953

[cfriedt]

• switched back to checking if a page is mapped via arch_page_phys_get() prior to calling k_mem_unmap() due to lack of a better alternative

Still working on my desk

twister -T tests/posix/shm/
...
INFO    - Total complete:   40/  40  100%  skipped:   15, failed:    0, error:    0

[cfriedt]

re-ping @andyross @vaishnavachath @ycsin @dkalowsk

[npitre]

It's not the user's responsibility to manage system resources, it's the
kernel's responsibility.

Sorry, but the POSIX layer you're creating (and that's highly appreciated
by the way) is not user code. You are managing many different resources
already on behalf of code calling the POSIX interface. That code is the user
not the interface.

Knowing how many times a particular object has been mapped and unmapped is
the POSIX layer's responsibility, just like the refcounting you already do
elsewhere between malloc() and free(). You wouldn't free the same memory
twice, right? And you're not calling into the low-level heap allocator to test
if a particular address is still allocated before calling free() either.
The assertion in k_mem_unmap() is there to catch the equivalent of a
double-free, which is a bug and why it shouldn't be removed.

So it should be no different with k_mem_map() and k_mem_unmap().
The first caller to increment the refcount does the map, subsequent callers
only increment the refcount and get the same already mapped address, and the
last to decrement the count to 0 does the unmap.

Am I missing something that would prevents such implementation?

[nashif]

z_finalize_typed_fd()

@cfriedt
dang, we still have those z_ prefixed public APIs, we should have addressed those long time ago :(
Any chance you can address this in a followup PR after this get merged?

@nashif - Sure - I was planning on using zvfs, but would you prefer k?

No k_, Ideally this should be the new prefix we discussed in the f2f (#64627).

Let me open a bug about this and bring the prefix discussion back so we can do something about it.

[nashif]

No k_, Ideally this should be the new prefix we discussed in the f2f (#64627).

Let me open a bug about this and bring the prefix discussion back so we can do something about it.

actually you can use zvfs which is being used already, better than z_ for sure.

[cfriedt]

z_finalize_typed_fd()

@cfriedt
dang, we still have those z_ prefixed public APIs, we should have addressed those long time ago :(
Any chance you can address this in a followup PR after this get merged?

@nashif - Sure - I was planning on using zvfs, but would you prefer k?

No k_, Ideally this should be the new prefix we discussed in the f2f (#64627).

Let me open a bug about this and bring the prefix discussion back so we can do something about it.

@nashif - I think a k_ prefix is fine for things like z_finalize_fd(), etc, but for consistency, that should likely just be a wrapper around a zvfs_ prefixed function (mutual dependency library of net, fs, posix, libc, etc, where common socket / fs / fd stuff was heading since 2022, #51211). There will be a fair number of zvfs_ prefixed functions, so I wouldn't necessarily want to flood the k_ namespace with them.

[nashif]

I think a k_ prefix is fine for things like z_finalize_fd()

k_ should be reserved for kernel/ stuff only, let not mix things up.

[andyross]

Still haven't combed through as well as I'd like, but +1 to everything I saw earlier. And it seems like this has excellent review coverage from others anyway

[dkalowsk]

Still thinking this is better than what we have today. Reviewing I don't see anything obvious standing out.

[npitre]

Am I missing something that would prevents such implementation?

Of course I am. munmap() doesn't pass a fd or any identifier other than
the actual address and length.

But you have another problem here. k_mem_map_phys_bare() should be paired
only with k_mem_unmap_phys_bare() otherwise bad things will happen.
I'd suggest you use k_mem_map_phys_guard() here instead, as it is
much closer to be compatible with k_mem_unmap() aka
k_mem_unmap_phys_guard().

And yet k_mem_unmap_phys_guard() wants to know if this is about anonymous
mem or not which can probably be determined programmatically (whether there
is a corresponding page frame or backing store entry for it) but that is not
implemented now.

So to say that MAP_FIXED is not implementable at the moment due to those
details and it might make sense to simply return ENOTSUP for the time being.