FCNTL F_SETFL Ignore creation flags

[MousseARaser06]

This pull request aims at fixing #4352, it proceeds by having a list of platform-allowed flags (bits) for that operation (F_SETFL), and performing a bitwise AND assignment to exclude all of the flags that aren't allowed.
Allowed flags change from platform to platform

[rustbot]

Thank you for contributing to Miri! A reviewer will take a look at your PR, typically within a week or two.
Please remember to not force-push to the PR branch except when you need to rebase due to a conflict or when the reviewer asks you for it.

[RalfJung]

Thanks for the PR :)

We'll get to reviewing it eventually, but there are other PRs in the queue that will be reviewed first.

[RalfJung]

Thanks for the PR! I have a concern regarding the fundamental approach.

View changes since this review

[RalfJung]

For Linux, we have docs saying "this operation can change only the O_APPEND, O_ASYNC, O_DIRECT, O_NOATIME, and O_NONBLOCK flags". Do we have anything similar for other OSes? If not, it seems kind of risky to just ignore everything we don't know about. The safer approach is to only ignore what we know can definitely be ignored.

[MousseARaser06]

I searched the other docs (the ones in that match statement)
and they all more or less have the same ignored flags in their man pages
the only OS I'm not quite sure is Solaris and Illumos, but then if I don't include them inside the match statement, the test fails
I will re-check everything just to make sure, but I'm 95% sure about this

[RalfJung]

Well, the test should match reality. :)

Overall I think the easier approach is to just only ignore flags we are sure are ignored everywhere.

[MousseARaser06]

MacOs's man pages states the following

The flags for the F_GETFL and F_SETFL commands are as follows:

       O_NONBLOCK   Non-blocking I/O; if no data is available to a read
                    call, or if a write operation would block, the read or
                    write call returns -1 with the error EAGAIN.

       O_APPEND     Force each write to append at the end of file; corre-sponds corresponds
                    sponds to the O_APPEND flag of [open(2)](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/open.2.html#//apple_ref/doc/man/2/open).

       O_ASYNC      Enable the SIGIO signal to be sent to the process
                    group when I/O is possible, e.g., upon availability of
                    data to be read.

FreeBSD's man pages states the following

The flags for the F_GETFL and F_SETFL commands are as follows:

   O_NONBLOCK   Non-blocking I/O; if no data is  available	to  a  [read(2)](https://man.freebsd.org/cgi/man.cgi?query=read&sektion=2&apropos=0&manpath=FreeBSD+15.0-RELEASE+and+Ports)
	    system  call,  or if a [write(2)](https://man.freebsd.org/cgi/man.cgi?query=write&sektion=2&apropos=0&manpath=FreeBSD+15.0-RELEASE+and+Ports) operation would block, the
	    read or write call returns -1 with the error EAGAIN.

   O_APPEND	    Force each write to	append at the end of file; corresponds
	    to the O_APPEND flag of [open(2)](https://man.freebsd.org/cgi/man.cgi?query=open&sektion=2&apropos=0&manpath=FreeBSD+15.0-RELEASE+and+Ports).

   O_DIRECT	    Minimize or	eliminate the cache  effects  of  reading  and
	    writing.   The  system  will  attempt to avoid caching the
	    data you read or write.  If	it cannot  avoid  caching  the
	    data,  it  will  minimize  the  impact the data has	on the
	    cache.  Use	of this	flag can  drastically  reduce  perfor-
	    mance if not used with care.

   O_ASYNC	    Enable  the	 SIGIO	signal to be sent to the process group
	    when I/O is	possible, e.g.,	upon availability of  data  to
	    be read.

   O_SYNC	    Enable synchronous writes.	Corresponds to the O_SYNC flag
	    of [open(2)](https://man.freebsd.org/cgi/man.cgi?query=open&sektion=2&apropos=0&manpath=FreeBSD+15.0-RELEASE+and+Ports).	 O_FSYNC is an historical synonym for O_SYNC.

   O_DSYNC	    Enable   synchronous  data	writes.	  Corresponds  to  the
	    O_DSYNC flag of [open(2)](https://man.freebsd.org/cgi/man.cgi?query=open&sektion=2&apropos=0&manpath=FreeBSD+15.0-RELEASE+and+Ports).

Illumos/Solaris's docs are very unclear, therefore only the most "common" flags are allowed (O_NONBLOCK, O_APPEND, O_DIRECT)

And Linux's man pages states the following

F_SETFL
Set the file status flags to the value specified by arg.
File access mode (O_RDONLY, O_WRONLY, O_RDWR) and file
creation flags (i.e., O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC)
in arg are ignored. On Linux, this operation can change
only the O_APPEND, O_ASYNC, O_DIRECT, O_NOATIME, and
O_NONBLOCK flags. It is not possible to change the O_DSYNC
and O_SYNC flags; see BUGS, below.

[RalfJung]

Okay. So sounds like ignoring O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC will work everywhere while also avoiding the risk of ignoring the wrong thing on new platforms / when new flags are added.

[MousseARaser06]

That sounds about right

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[rustbot]

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[MousseARaser06]

@rustbot review