Test failure with "shutil.rmtree()" in isolated build environments (1.5.3.rc1 in Fedora 41)

[hannes101]

I was trying to build backintime for fedora 41. I am getting the following test failures. I could of course drop these tests, but I thought it might be worth reporting during this time of the development cycle.

build.log

The build can be found here.
https://koji.fedoraproject.org/koji/taskinfo?taskID=125095920

After I removed the the above tests, it fails at a later one:
https://kojipkgs.fedoraproject.org//work/tasks/6177/125096177/build.log
Uploading build_2.log…

[buhtz]

Hello Johannes,

thank you very much for trying that RC and reporting the problem.

Problem 1 shutil.rmtree()

Mhm.... Here is an example of one failed test:

__________________________ General.test_ctor_defaults __________________________
self = <test.test_uniquenessset.General testMethod=test_ctor_defaults>
    def test_ctor_defaults(self):
        """Default values in constructor."""
>       with TemporaryDirectory(prefix='bit.') as temp_name:
test/test_uniquenessset.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.13/tempfile.py:946: in __exit__
    self.cleanup()
/usr/lib64/python3.13/tempfile.py:950: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
/usr/lib64/python3.13/tempfile.py:930: in _rmtree
    _shutil.rmtree(name, onexc=onexc)
/usr/lib64/python3.13/shutil.py:763: in rmtree
    _rmtree_safe_fd(stack, onexc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
stack = []
onexc = <function TemporaryDirectory._rmtree.<locals>.onexc at 0xffffb39bc860>
    def _rmtree_safe_fd(stack, onexc):
        # Each stack item has four elements:
        # * func: The first operation to perform: os.lstat, os.close or os.rmdir.
        #   Walking a directory starts with an os.lstat() to detect symlinks; in
        #   this case, func is updated before subsequent operations and passed to
        #   onexc() if an error occurs.
        # * dirfd: Open file descriptor, or None if we're processing the top-level
        #   directory given to rmtree() and the user didn't supply dir_fd.
        # * path: Path of file to operate upon. This is passed to onexc() if an
        #   error occurs.
        # * orig_entry: os.DirEntry, or None if we're processing the top-level
        #   directory given to rmtree(). We used the cached stat() of the entry to
        #   save a call to os.lstat() when walking subdirectories.
        func, dirfd, path, orig_entry = stack.pop()
        name = path if orig_entry is None else orig_entry.name
        try:
            if func is os.close:
                os.close(dirfd)
                return
            if func is os.rmdir:
                os.rmdir(name, dir_fd=dirfd)
                return
    
            # Note: To guard against symlink races, we use the standard
            # lstat()/open()/fstat() trick.
>           assert func is os.lstat
E           AssertionError
/usr/lib64/python3.13/shutil.py:663: AssertionError

It looks like that there is a problem when cleaning up the temporary directory. shutil.rmtree() raise the error. But I don't understand the details.

Might it be the case that this is related to some "special attributes" of your build environment (chroot magic or something like that stuff I don't understand)?

Can you run the test in a regular fedora setup?

EDIT: I asked at python-llist and at the mock upstream project about the assert-error from shutil.rmtree().

Problem 2: User not in second usergroup

Moved to #1912

[hannes101]

Yes, you are right that it's a mock environment. I am not really sure how to properly debug something in a mock environment.
But checking the root.log of a build it seems that the user mockbuild gets created and is only added to one group.
root.log

DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['userdel', '-f', 'mockbuild', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['groupdel', 'mock', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['groupadd', 'mock', '-g', '135', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:608: Child return code was: 0
DEBUG util.py:634: child environment: None
DEBUG util.py:556: Executing command: ['useradd', 'mockbuild', '-o', '-u', '1001', '-g', '135', '-N', '-d', '/builddir', '--prefix', '/var/lib/mock/fedora-41-x86_64/root'] with env {'TERM': 'vt100', 'SHELL': '/bin/sh', 'HOME': '/builddir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'LANG': 'C.UTF-8'} and shell False
DEBUG util.py:459: useradd: warning: the home directory /builddir already exists.
DEBUG util.py:459: useradd: Not copying any file from skel directory into it.
DEBUG util.py:459: Creating mailbox file: File exists
DEBUG util.py:608: Child return code was: 0

[buhtz]

About Problem 1: I asked at python-list and at the mock upstream project about the problem.

About Problem 2: Are you experienced enough with mock to configure it that way that you can add the user to a second usergroup? Do you think this would be an appropriate solution?

[xsuchy]

During the build in Mock the build is started as:

<mock-chroot> sh-5.2# id
uid=0(root) gid=0(root) groups=0(root)

and then dropped privs to:

<mock-chroot> sh-5.2$ id
uid=1000(mockbuild) gid=135(mock) groups=135(mock)

The environment is very minimalistic on purpose. I recommend to skip the test if the len(GROUPS) < 2.

[buhtz]

Problem about usergroup separated into its own issue #1912.

[buhtz]

FYI: The shutil problem might also be related to a specific Python version. It is reported with Python 3.13. I was not able to find the code location in the shutil.py file of Python 3.11.2.

This is the commit where this assert line was introduce in Python.

python/cpython@53b1981

The related PR

python/cpython#119808

Not sure if it is appropriate to contact the committing person about the problem? Maybe he can give us a hint what is going on?

EDIT: Python 3.13 on TravisCI (Ubuntu Jammy) pass all BIT tests.