mknod with S_IFIFO fails to create FIFO pipe, function call return success does not throw any error

[anjalirai-intel]

Description of the problem

mknod with S_IFIFO fails to create the FIFO pipe in gramine, and also does not throw any error, function call return success.

I have attached one sample code which create FIFO PIPE using mknod with S_IFIFO flag, and performing stat operation on the FIFO pipe. the generated FIFO file is not deleted at the end, so we should be able to see the file at the end

Linux Output: We can notice that mknod syscall returns 0

$ gcc mknod_fifo.c -o mknod_fifo
$ chmod +x ./mknod_fifo

$ ./mknod_fifo
mknod return value is 0
stat return value is 0

Observation: A new FIFO file mknod_fifo_test has been created

$ ls -al
total 44
drwxrwxr-x 3 intel intel  4096 Jun  5 13:07 .
drwxrwxr-x 8 intel intel  4096 Jun  5 13:03 ..
drwxrwxr-x 2 intel intel  4096 Jun  5 12:26 etc
-rw-rw-r-- 1 intel intel  1351 Jun  5 12:58 Makefile
-rwxrwxr-x 1 intel intel 16200 Jun  5 13:07 mknod_fifo
-rw-rw-r-- 1 intel intel   728 Jun  5 13:05 mknod_fifo.c
-rw-rw-r-- 1 intel intel  3808 Jun  5 12:28 mknod_fifo.manifest
-rw-rw-r-- 1 intel intel  2833 Jun  5 12:24 mknod_fifo.manifest.template
p--------- 1 intel intel     0 Jun  5 13:07 mknod_fifo_test

mknod_fifo.zip

Steps to reproduce

Download and extract the attached zip file
make SGX=1
gramine-sgx mknod_fifo

Expected results

I have copied the expected results above from Linux native

Actual results

gramine-sgx mknod_fifo: mknod call was successful but stat call failed. Further more we can see that FIFO file is not present

Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sys.experimental__enable_flock = true      (flock syscall is enabled; still under development and may contain bugs)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

[P1:T1:] error: Mounting file:/dev/cpu_dma_latency may expose unsanitized, unsafe files to unsuspecting application. Gramine will continue application execution, but this configuration is not recommended for use in production!
[P1:T1:] error: Mounting file:/proc may expose unsanitized, unsafe files to unsuspecting application. Gramine will continue application execution, but this configuration is not recommended for use in production!
mknod return value is 0
stat return value is -1
stat() failed -1

Observation: We don't see mknod_fifo_test being present

intel@intel-M50CYP2SBSTD:~/anjali/ltp_mknod_fifo$ ls -al
total 60
drwxrwxr-x 3 intel intel  4096 Jun  5 13:10 .
drwxrwxr-x 8 intel intel  4096 Jun  5 13:03 ..
drwxrwxr-x 2 intel intel  4096 Jun  5 12:26 etc
-rw-rw-r-- 1 intel intel  1351 Jun  5 12:58 Makefile
-rwxrwxr-x 1 intel intel 16152 Jun  5 13:10 mknod_fifo
-rw-rw-r-- 1 intel intel   728 Jun  5 13:05 mknod_fifo.c
-rw-rw-r-- 1 intel intel  3816 Jun  5 13:10 mknod_fifo.manifest
-rw-rw-r-- 1 intel intel  5032 Jun  5 13:10 mknod_fifo.manifest.sgx
-rw-rw-r-- 1 intel intel  2841 Jun  5 13:10 mknod_fifo.manifest.template
-rw-rw-r-- 1 intel intel  2832 Jun  5 13:10 mknod_fifo.o
-rw-rw-r-- 1 intel intel  1808 Jun  5 13:10 mknod_fifo.sig

Gramine commit hash

929bb9d

[dimakuv]

So this issue has two sub-issues:

1. Why does stat() not work?
2. Why is the FIFO pseudo-file (named pipe) is not seen on the host (i.e. in the ls output), as other regular files?

The second sub-issue is simple: FIFOs are not normal files, they are actually pipes and thus must be transparently encrypted in Gramine. Since this encryption is per-enclave, there is no sense to expose the FIFO pseudo-file to the host -- this FIFO won't be readable/writable by any other process, other than this SGX enclave. Therefore, Gramine developers made a conscious choice to hide the FIFO pseudo-file from the host, for security reasons (and because functionality-wise exposing FIFOs would be useless anyway).

Now for the first issue: why doesn't stat() work? That's because FIFO emulation in Gramine doesn't have stat() and hstat() callbacks. You can observe this in the list of callbacks:

• gramine/libos/src/fs/pipe/fs.c

  Lines 286 to 290 in 929bb9d

  	static struct libos_fs_ops fifo_fs_ops = {
  	.read = &pipe_read,
  	.write = &pipe_write,
  	.setflags = &pipe_setflags,
  	};

• gramine/libos/src/fs/pipe/fs.c

  Lines 292 to 296 in 929bb9d

  	static struct libos_d_ops fifo_d_ops = {
  	.open = &fifo_open,
  	.icheckpoint = &fifo_icheckpoint,
  	.irestore = &fifo_irestore,
  	};

Note the missing stat() and hstat() fields. For comparison, these are the callbacks for regular (chroot) files:

• gramine/libos/src/fs/chroot/fs.c

  Line 427 in 929bb9d

  .hstat = &generic_inode_hstat,

• gramine/libos/src/fs/chroot/fs.c

  Line 438 in 929bb9d

  .stat = &generic_inode_stat,

So in the end, when the application calls stat() or hstat(), it ends somewhere around here with an EACCES error:

gramine/libos/src/sys/libos_stat.c

Lines 24 to 25 in 929bb9d

	if (!fs || !fs->d_ops || !fs->d_ops->stat)
	return -EACCES;

I can cook a quick patch that adds these callbacks to FIFOs, in a dummy emulated way. I see no reason why not to allow this, and it will unblock applications like the one attached in this issue.