Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-95023: Added os.setns and os.unshare to easily switch between namespaces on Linux #95046

Merged
merged 48 commits into from
Oct 20, 2022
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
0bdd8bc
gh-95023: added os.setns and os.unshare for namespaces switching on L…
noamcohen97 Jul 20, 2022
3685a27
remove gil release
noamcohen97 Jul 20, 2022
8f22740
better setns, unshare doc
noamcohen97 Jul 21, 2022
5115c8a
added example to setns
noamcohen97 Jul 21, 2022
8845a86
added note about fileno for setns
noamcohen97 Jul 21, 2022
d7bb582
added see also section for os.unshare
noamcohen97 Jul 21, 2022
987613a
Update configure.ac
noamcohen97 Jul 26, 2022
4c91413
Revert "remove gil release"
noamcohen97 Jul 26, 2022
7d23963
better docs
noamcohen97 Jul 26, 2022
b7abf20
assume Linux platform has a working readlink
noamcohen97 Jul 26, 2022
af74db5
change ifdefs
noamcohen97 Jul 26, 2022
5181126
added missing `CLONE_*` consts
noamcohen97 Jul 26, 2022
3a37ac2
run unshare in a different process
noamcohen97 Jul 26, 2022
1386833
reformat doc
noamcohen97 Jul 26, 2022
ae4b661
indent test code
noamcohen97 Jul 26, 2022
57b2c84
Apply suggestions from code review
noamcohen97 Jul 26, 2022
b2df7f7
better doc
noamcohen97 Jul 26, 2022
b4a68b6
Merge branch 'main' into namespaces
noamcohen97 Jul 26, 2022
dc51d01
remove whitespaces from doc
noamcohen97 Jul 26, 2022
5a3cff5
fix NEWS entry
noamcohen97 Jul 26, 2022
1d45196
move test code to another file
noamcohen97 Jul 26, 2022
84c4b8c
add glibc requirements to doc
noamcohen97 Jul 26, 2022
b9d3a34
Revert "move test code to another file"
noamcohen97 Jul 26, 2022
51c60d4
proofreaders review fixes
noamcohen97 Jul 26, 2022
225e06b
fix doc whitespaces
noamcohen97 Jul 26, 2022
fb64bb7
better documentation
noamcohen97 Jul 27, 2022
01d4af4
fix docs
noamcohen97 Jul 27, 2022
15e6d8b
remove bitwise or explanation
noamcohen97 Jul 27, 2022
375165b
Apply suggestions from code review
noamcohen97 Jul 28, 2022
7e2b44c
fix typo
noamcohen97 Jul 28, 2022
3ae952c
ref
noamcohen97 Jul 28, 2022
432d274
fix doc whitespace
noamcohen97 Jul 28, 2022
afa9a00
Merge remote-tracking branch 'upstream/main' into namespaces
noamcohen97 Jul 28, 2022
54c7bce
Merge remote-tracking branch 'upstream/main' into namespaces
noamcohen97 Jul 28, 2022
01713ec
regen posixmodule
noamcohen97 Jul 28, 2022
a6bb345
add name to `Misc/ACKS`
noamcohen97 Jul 28, 2022
1c0fb83
Merge remote-tracking branch 'upstream/main' into namespaces
noamcohen97 Aug 17, 2022
dac402a
fix doc availability
noamcohen97 Aug 17, 2022
ff7f961
regen global strings
noamcohen97 Aug 17, 2022
cc44f01
Merge remote-tracking branch 'upstream/main' into namespaces
noamcohen97 Oct 7, 2022
58bc8a6
Merge branch 'main' into namespaces
noamcohen97 Oct 7, 2022
b14396e
Merge remote-tracking branch 'upstream/main' into namespaces
noamcohen97 Oct 7, 2022
50c4809
Merge branch 'main' into namespaces
noamcohen97 Oct 14, 2022
de7dd3d
do not use stdout for test
noamcohen97 Oct 17, 2022
5c1bbdd
move all test login to subprocess
noamcohen97 Oct 19, 2022
29129d9
Update Lib/test/test_posix.py
noamcohen97 Oct 20, 2022
94883a4
add comments to tests
noamcohen97 Oct 20, 2022
3e365d2
bugfix in test while handling `ENOSPC`
noamcohen97 Oct 20, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions Doc/library/os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -569,6 +569,32 @@ process and user.
See the documentation for :func:`getgroups` for cases where it may not
return the same group list set by calling setgroups().

.. function:: setns(fd, nstype=0)

Reassociate thread with a namespace, see the :manpage:`setns(2)` man page for more details.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

If *fd* refers to a ``/proc/[pid]/ns/`` link, ``setns()`` reassociates the
erlend-aasland marked this conversation as resolved.
Show resolved Hide resolved
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
calling thread with the namespace associated with that link, subject to any
constraints imposed by the *nstype* argument (or any if ``0``).
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
Since Linux 5.8, *fd* may refer to a PID file descriptor obtained from
:func:`~os.pidfd_open`. In this case ``setns()`` reassociates the calling thread
erlend-aasland marked this conversation as resolved.
Show resolved Hide resolved
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
into one or more of the same namespaces as the thread referred to by *fd*
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
subject to any constraints imposed by the *nstype*, which is
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
a bit mask specified by combining one or more of the ``CLONE_NEW*`` constants,
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
e.g. ``setns(fd, os.CLONE_NEWUTS | os.CLONE_NEWPID)``.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
The callers memberships in unspecified namespaces are left unchanged.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
*fd* can be any object with a :meth:`fileno` method, or a raw file descriptor.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

This example reassociates the thread with the ``init`` process' network namespace::
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

fd = os.open("/proc/1/ns/net", os.O_RDONLY)
os.setns(fd, os.CLONE_NEWNET)
os.close(fd)

.. availability:: Linux 3.0 or newer.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

.. versionadded:: 3.12

noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
.. function:: setpgrp()

Call the system call :c:func:`setpgrp` or ``setpgrp(0, 0)`` depending on
Expand Down Expand Up @@ -732,6 +758,42 @@ process and user.
The function is now always available and is also available on Windows.


.. function:: unshare(flags)

Disassociate parts of the process execution context, see the :manpage:`unshare(2)`
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
man page for more details.
The *flags* argument is a bit mask combining zero or more of the ``CLONE_*``
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
constants using ``|`` (bitwise or), that specifies which parts of the execution
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
context should be unshared.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
If *flags* is specified as zero, no changes are made to the calling process'
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
execution context.

.. availability:: Linux 2.6.16 or newer.

.. versionadded:: 3.12

.. seealso::

The :func:`~os.setns` function.

noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
Flags to the :func:`unshare` function, if the implementation supports them.
See the Linux manual for the exact effect and availability.
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

.. data:: CLONE_FILES
CLONE_FS
CLONE_NEWCGROUP
CLONE_NEWIPC
CLONE_NEWNET
CLONE_NEWNS
CLONE_NEWPID
CLONE_NEWTIME
CLONE_NEWUSER
CLONE_NEWUTS
CLONE_SIGHAND
CLONE_SYSVSEM
CLONE_THREAD
CLONE_VM

.. _os-newstreams:
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved

File Object Creation
Expand Down
21 changes: 21 additions & 0 deletions Lib/test/namespaces-test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import os
import sys


def main():
fd = os.open('/proc/self/ns/uts', os.O_RDONLY)
try:
print(os.readlink('/proc/self/ns/uts'))
os.unshare(os.CLONE_NEWUTS)
print(os.readlink('/proc/self/ns/uts'))
os.setns(fd, os.CLONE_NEWUTS)
print(os.readlink('/proc/self/ns/uts'))
except OSError as e:
sys.stderr.write(str(e.errno))
sys.exit(2)
finally:
os.close(fd)


if __name__ == '__main__':
main()
37 changes: 37 additions & 0 deletions Lib/test/test_posix.py
Original file line number Diff line number Diff line change
Expand Up @@ -2172,6 +2172,43 @@ def test_utime(self):
os.utime("path", dir_fd=0)


class NamespacesTests(unittest.TestCase):
"""Tests for os.unshare() and os.setns()."""

@support.requires_subprocess()
def subprocess(self, file_path):
import subprocess
with subprocess.Popen((sys.executable, file_path),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
encoding="utf-8"
) as p:
p.wait()
return (
p.returncode,
tuple(p.stdout),
tuple(p.stderr),
)

@unittest.skipUnless(hasattr(os, 'unshare'), 'needs os.unshare()')
@unittest.skipUnless(hasattr(os, 'setns'), 'needs os.setns()')
@unittest.skipUnless(os.path.exists('/proc/self/ns/uts'), 'need /proc/self/ns/uts')
@support.requires_linux_version(3, 0, 0)
def test_unshare_setns(self):
rc, out, err = self.subprocess(support.findfile("namespaces-test.py"))

if rc == 2:
e = int(err[0])
self.assertIn(e, (errno.EPERM, errno.EINVAL, errno.ENOSPC, errno.ENOSYS))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to move these checks inside the child process code. I don't think that skipping the test if unshare() or setns() fails matters. Just exit the child process early with a success (exit code 0, well, just exit) if you get one of these errors, no? If the child process must always succeed, you can use test.suppot.script_helper.assert_python_ok() which captures stdout and stderr and raises an exception with all data if the process fails.

Maybe it would make sense to add a except PermissionError: instead. Would you mind to add a comment explaining why/how PermissionError can happen?

ENOSPC is expected on unshare() if we exceed some limits on namespaces? If it can only happen on unshare(), maybe add a try/except only on this function call?

Why is EINVAL expected? Can it be raised if there is a bug in the test? Or can it be raised depending on the kernel configuration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! did not know about assert_python_ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EINVAL could be raised the kernel was not configured with the CONFIG_UTS_NS option.

self.skipTest(f"could not call os.unshare / os.setns [Errno {e}].")

self.assertEqual(rc, 0)
self.assertEqual(err, ())
original, new, back = out
self.assertNotEqual(original, new)
self.assertEqual(original, back)
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved


def tearDownModule():
support.reap_children()

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Implement :func:`os.setns` and :func:`os.unshare` for Linux. Patch by Noam Cohen.
105 changes: 104 additions & 1 deletion Modules/clinic/posixmodule.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

107 changes: 107 additions & 0 deletions Modules/posixmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -8588,6 +8588,64 @@ os_pidfd_open_impl(PyObject *module, pid_t pid, unsigned int flags)
#endif


#ifdef HAVE_SETNS
/*[clinic input]
os.setns
fd: fildes
A file descriptor to a namespace.
nstype: int = 0
Type of namespace.

Move the calling thread into different namespaces.
[clinic start generated code]*/

static PyObject *
os_setns_impl(PyObject *module, int fd, int nstype)
/*[clinic end generated code: output=5dbd055bfb66ecd0 input=42787871226bf3ee]*/
{
int res;

Py_BEGIN_ALLOW_THREADS
res = setns(fd, nstype);
Py_END_ALLOW_THREADS

if (res != 0) {
return posix_error();
}

Py_RETURN_NONE;
}
#endif


#ifdef HAVE_UNSHARE
/*[clinic input]
os.unshare
flags: int
Namespaces to be unshared.

Disassociate parts of a process (or thread) execution context.
[clinic start generated code]*/

static PyObject *
os_unshare_impl(PyObject *module, int flags)
/*[clinic end generated code: output=1b3177906dd237ee input=9e065db3232b8b1b]*/
{
int res;

Py_BEGIN_ALLOW_THREADS
res = unshare(flags);
Py_END_ALLOW_THREADS

if (res != 0) {
return posix_error();
}

Py_RETURN_NONE;
}
#endif


#if defined(HAVE_READLINK) || defined(MS_WINDOWS)
/*[clinic input]
os.readlink
Expand Down Expand Up @@ -14933,6 +14991,8 @@ static PyMethodDef posix_methods[] = {
OS__ADD_DLL_DIRECTORY_METHODDEF
OS__REMOVE_DLL_DIRECTORY_METHODDEF
OS_WAITSTATUS_TO_EXITCODE_METHODDEF
OS_SETNS_METHODDEF
OS_UNSHARE_METHODDEF
{NULL, NULL} /* Sentinel */
};

Expand Down Expand Up @@ -15378,6 +15438,53 @@ all_ins(PyObject *m)
#ifdef SCHED_FX
if (PyModule_AddIntConstant(m, "SCHED_FX", SCHED_FSS)) return -1;
#endif

/* constants for namespaces */
#if defined(HAVE_SETNS) || defined(HAVE_UNSHARE)
#ifdef CLONE_FS
if (PyModule_AddIntMacro(m, CLONE_FS)) return -1;
#endif
#ifdef CLONE_FILES
if (PyModule_AddIntMacro(m, CLONE_FILES)) return -1;
#endif
#ifdef CLONE_NEWNS
if (PyModule_AddIntMacro(m, CLONE_NEWNS)) return -1;
#endif
#ifdef CLONE_NEWCGROUP
if (PyModule_AddIntMacro(m, CLONE_NEWCGROUP)) return -1;
#endif
#ifdef CLONE_NEWUTS
if (PyModule_AddIntMacro(m, CLONE_NEWUTS)) return -1;
#endif
#ifdef CLONE_NEWIPC
if (PyModule_AddIntMacro(m, CLONE_NEWIPC)) return -1;
#endif
#ifdef CLONE_NEWUSER
if (PyModule_AddIntMacro(m, CLONE_NEWUSER)) return -1;
#endif
#ifdef CLONE_NEWPID
if (PyModule_AddIntMacro(m, CLONE_NEWPID)) return -1;
#endif
#ifdef CLONE_NEWNET
if (PyModule_AddIntMacro(m, CLONE_NEWNET)) return -1;
#endif
#ifdef CLONE_NEWTIME
if (PyModule_AddIntMacro(m, CLONE_NEWTIME)) return -1;
#endif
noamcohen97 marked this conversation as resolved.
Show resolved Hide resolved
#ifdef CLONE_SYSVSEM
if (PyModule_AddIntMacro(m, CLONE_SYSVSEM)) return -1;
#endif
#ifdef CLONE_THREAD
if (PyModule_AddIntMacro(m, CLONE_THREAD)) return -1;
#endif
#ifdef CLONE_SIGHAND
if (PyModule_AddIntMacro(m, CLONE_SIGHAND)) return -1;
#endif
#ifdef CLONE_VM
if (PyModule_AddIntMacro(m, CLONE_VM)) return -1;
#endif
#endif

#endif

#ifdef USE_XATTRS
Expand Down
Loading