Cannot mount devpts or sysfs with a user namespace (as of v0.0.3)

[wking]

While adding user namespaces to my bundle, I had to drop the devpts and sysfs mounts from runC's default config, and I also had to drop ro from runC's default cgroup-mount options. Restoring them to my config.json.template leads to:

# runc config.json
Timestamp: 2015-08-24 16:47:34.679951846 -0700 PDT
Code: System error

Message: invalid argument

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@37

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@242

---
3: init·1
Package: main
File: run.go@21

---
4: init
Package: main
File: utils.go@177

---
5: main
Package: runtime
File: proc.go@58

---
6: goexit
Package: runtime
File: asm_amd64.s@2232WARN[0000] signal: killed
FATA[0000] Container start failed: [8] System error: invalid argument
Makefile:11: recipe for target 'run' failed
make: *** [run] Error 1

with the devpts entry,

# runc config.json
Timestamp: 2015-08-24 16:46:33.526868472 -0700 PDT
Code: System error

Message: operation not permitted

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@37

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52

---
WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: operation not permitted 
Makefile:11: recipe for target 'run' failed
make: *** [run] Error 1

with the sysfs entry, and:

# runc config.json
Timestamp: 2015-08-24 16:51:02.5931295 -0700 PDT
Code: System error

Message: operation not permitted

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@37

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@242

---
3: init·1
Package: main
File: run.go@21

---
4: init
Package: main
File: utils.go@WARN[0000] signal: killed                               
FATA[0000] Container start failed: [8] System error: operation not permitted 
Makefile:11: recipe for target 'run' failed
make: *** [run] Error 1

with ro in the cgroups entry. I haven't been able to figure out why I'm getting these mount errors. If I drop my user namespacing, the errors go away. Is this a runC bug? A user-namespace limitation? A bug in my config template? I'm happy to help with further digging, but I could use a few hints pointing me in a useful direction. Perhaps this is what @LK4D4 was thinking about when he mentioned reconsidering default mounts for unprivileged functionality.

[mrunalp]

@wking Thanks for creating the issue. @estesp reported this a few days back on IRC. This is most likely related to a kernel change. I will look into this.

[wking]

On Tue, Aug 25, 2015 at 08:10:59PM -0700, Mrunal Patel wrote:

@estesp reported this a few days back on IRC.

Hmm, which channel/day? The most recent estesp comments in
#opencontainers look like Hangout linking after our last meeting
(2015-08-12).

This is most likely related to a kernel change. I will look into
this.

I'm running 4.1.0, if that helps.

[estesp]

This is the kernel patch which seems to be the culprit for the issue: http://www.spinics.net/lists/linux-fsdevel/msg86256.html

It went into one of the Linux 4.2-rc's, but Ubuntu recently brought it into their 3.16.0-45 kernel update, which is how I happened to catch it. @wking where are you sourcing your 4.1 kernel from? If from a distro maybe they backported this patch into your kernel? Here is a direct link to the upstream commit: torvalds/linux@1b852bc#diff-3fbed1fd4d15699b74b30cabf5be8133 which shows the 4.2 tag/lineage.

[wking]

On Tue, Aug 25, 2015 at 08:42:58PM -0700, Phil Estes wrote:

@wking where are you sourcing your 4.1 kernel from?

I built it locally from 1, tag v4.1, so I don't have
torvalds/linux@1b852bceb or any FS_USERNS_VISIBLE in my source.

If we suspect a kernel issue, I can try bisecting to actually find the
culprit, but that may take a while and I'm happy to leave it to
someone who's got a spare box and more practice with kexec ;).

[mrunalp]

I tested with the kernel 4.2 on Fedora rawhide. There are two issues:

1. ro needs to be removed from the cgroups mount as @wking mentioned earlier. (probably remount isn't permitted; looking further).
2. Take out gid=5 from the devpts mount options.

With the above two changes to the config.json, I was able to bring up a container in the busybox rootfs.
I will look into this further. @estesp could you try with these two changes and see if it works for you ?

[wking]

On Mon, Aug 31, 2015 at 03:39:47PM -0700, Mrunal Patel wrote:

1. Take out gid=5 from the devpts mount options.

This works for me on my vanilla 4.1.

[estesp]

Works for me with those changes as well in runC. The original report I got was someone testing my Docker-based user namespace patchset before and after the Ubuntu kernel update I mentioned, so it might be trickier to get a test scenario there, although I could provide a modified default_template.go for the native execdriver to change the ro mount option, and gid=5 setting for devpts.

[crosbymichael]

This has been resolved.