seccomp filter should return ENOSYS for unknown syscalls

[jethrogb]

Currently, the seccomp filter installed on Linux returns EPERM even for system calls that are unknown. This is problematic when new system calls are added by Linux. Programs wishing to use the new system call will try to call it, and will implement a fallback mechanism when ENOSYS is returned (indicating the kernel doesn't support the call). However, when using containers, it will likely receive EPERM instead, failing instead of trying the fallback path.

In addition to the list of acceptable syscalls, the container definition should include a maximum known syscall number. The seccomp filter should be configured such that calls above the maximum return ENOSYS. When new syscalls are added, the maximum can be increased after the seccomp policy is updated.

[eero-t]

The inherent fragility of seccomp(): https://lwn.net/Articles/738694/

The discussion on the article is very instructive. Basically whatever you do with seccomp, there are potential future landmines. And those article comments didn't even go into kernel syscall & libc differences between HW architectures.

[richfelker]

EPERM just should not be used here. For a very large number of syscalls, EPERM has a well-defined meaning the application/library code making the call knows how to react to, and "blocked by container policy" is usually not in the scope of that meaning. For specific syscalls where the container policy does match the specified EPERM failure case, EPERM could reasonably be kept, but the default for all blocked syscalls should be ENOSYS.

[richfelker]

Note that this is breaking running of musl 1.2.0+ binaries - see docker/for-win#8326 for another example - due to inability to perform the correct meaningful fallback. We can't just treat EPERM as cause for fallback too, since EPERM is also a meaningful error (and fallback to an old time32 syscall when seeing it would have TOCTOU bugs if the cause of EPERM went away between the two calls, making a race window where the Y2038-unsafe syscall gets used).

[cyphar]

On the other hand, if we return ENOSYS for something like clone(CLONE_NEWUSER) then glibc may get very confused ("clone isn't supported? oh i guess we're on Linux 1.2..."). IMHO we should split the current policy into "pretend the syscall doesn't exist" and "tell the program it's not allowed to do dodgy things". I agree that EPERM for unknown syscalls isn't ideal.

[richfelker]

I don't think glibc actually does that, but if you're blocking specific functionality of a syscall rather than the whole syscall, the choice of error code should be made on a per-syscall basis to match existing error semantics. For example CLONE_NEWUSER should do whatever the kernel does when the kernel is configured without user namespace support or with it restricted to use by root. But as-yet-unknown syscalls should all error with ENOSYS.

[fweimer]

@cyphar For unrecognized flag arguments, most system calls use an EINVAL error. EPERM is not correct there, either. (I think clone is an outlier and used to ignore invalid flags in the kernel, but that's been fixed on the kernel side, unlike open, unless you count openat2.)

[fweimer]

Note that this issue is now somewhat urgent because Linux has added an faccessat2 system call, and many applications use it today, by calling glibc's faccessat function. glibc 2.32 and later implement faccessat using faccessat2, and perform fallback to the old emulation code only on ENOSYS errors.

[richfelker]

@fweimer Thanks for adding that. I really don't want this to remain a game of whack-a-mole where exceptions get added for each syscall once breakage is found. Upstream should do the right thing and stop producing EPERM entirely so fallbacks can work as intended. Having support from the glibc side in getting this handled right is much appreciated!

[cyphar]

I want to point out that if you want to change the seccomp filter in Docker, you'll need to make an issue in the Docker repo. We don't control the seccomp filter that Docker uses. Until recently the runtime-spec didn't support custom return values, but that has changed so Docker will need to update their default seccomp profile.

As I said, returning ENOSYS for syscalls deliberately blocked is less than ideal so I'd suggest instead there should be a new seccomp rule added which encodes the highest-known syscall number and return ENOSYS if the requested syscall has a higher number. This isn't fool-proof (several architectures have gaps in their syscall tables for historical reasons) but with the new unified syscall number work it's incredibly unlikely this will cause significant issues in the future.

[jethrogb]

@cyphar First, runc needs to add the ability to specify two different "defaults": one for known but not specifically specified syscalls, and one for unknown syscalls. Currently, non-specified calls all return the same defaultAction.

[richfelker]

OK, can someone familiar with Docker and how that works open an issue on their side and link to this one?

[thaJeztah]

the default seccomp profile in docker is in https://github.com/moby/moby/blob/master/profiles/seccomp/default_linux.go
a similar profile is also in containerd; https://github.com/containerd/containerd/blob/master/contrib/seccomp/seccomp_default.go

and the runtime spec; https://github.com/opencontainers/runtime-spec/blob/f1164e526717e7d0b6e5ac24b05cb4b7401b0a98/config-linux.md#seccomp

[cyphar]

@jethrogb I don't think runc is in a position to do that -- what set of syscalls are "known" is a property of the profile being written, not the container runtime. If you write a profile today and 50 syscalls get added next week, runc (or rather libseccomp) will know about those syscalls but the old profile will not.

Since Docker is the thing generating these profiles (and accepting user-specified profiles too), you would want the user-specified profile to say "this is the latest syscall at the time when I wrote this profile". The Docker-generated profile will then have a default action of SECCOMP_RET_ERRNO with errno EPERM (or whatever it is now) and then a rule will be added which checks whether the syscall number is larger than $NEWEST_SYSCALL and return ENOSYS in that case.

[jethrogb]

I don't think runc is in a position to do that

Not today no, as you explain the profile description needs to be expanded to convey that information.

[thaJeztah]

Currently, runc also needs to know about the syscalls if they're included in the profile (opencontainers/runtime-spec#1071), so if a profile specifies syscalls that runc doesn't know about, the container fails to start (see moby/moby#41562).