Use `statx(2)` system call on Linux for extended `os.stat` information

[alexander255]

BPO	39533
Nosy	@vstinner, @tiran, @Alexander255, @scrool
PRs	bpo-39533: Use statx on more recent Linux to expose st_flags and st_btime on all platforms #19125

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-03.04:26:02.819>
labels = ['type-feature', 'library', '3.9']
title = 'Use `statx(2)` system call on Linux for extended `os.stat` information'
updated_at = <Date 2020-05-17.19:40:44.290>
user = 'https://github.com/alexander255'

bugs.python.org fields:

activity = <Date 2020-05-17.19:40:44.290>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-02-03.04:26:02.819>
creator = 'ntninja'
dependencies = []
files = []
hgrepos = []
issue_num = 39533
keywords = ['patch']
message_count = 4.0
messages = ['361265', '369128', '369149', '369150']
nosy_count = 5.0
nosy_names = ['vstinner', 'christian.heimes', 'ntninja', 'slow franklin', 'scrool']
pr_nums = ['19125']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39533'
versions = ['Python 3.9']

Linked PRs

• gh-83714: Use statx on Linux 4.11 and later in os.stat #136334

[alexander255]

Background: For a long time several Linux filesystems have been tracking two extra bits of information. The file attributes bits1 and the file creation time (aka crtime aka btime aka birthtime)2. Before Linux 4.11 accessing these required secret knowledge (ioctl numbers) or access to unstable interfaces (debugfs). However since that version the statx(2) system call3 has finally been added (it has a long history), which exposes these two fields adds (struct) space for potentially more.

Since CPython already exposes st_birthtime on FreeBSD and friends, I think it would be fair to also expose this field on Linux. As the timestamp value is only available on some file systems and configurations it is not guaranteed that the system call will return a value for btime at all. I suppose the field should be set to None in that case. In my opinion it should also become a regular field (available on all platforms) since, with this addition, we now have a suitable value to return on every major platform CPython targets: stx_btime on Linux, st_birthtime on macOS/FreeBSD and st_ctime on Windows.

stx_attributes could be exposed as a new st_attributes flag specific to Linux as there is no equivalent on other platforms to my knowledge (Window's st_file_attributes is similar in some aspects but has a completely different format and content).

There is a Python script I created, that calls statx(2) using ctypes here: https://github.com/ipfs/py-datastore/blob/e566d40a8ca81d8628147e255fe7830b5f928a43/datastore/filesystem/util/statx.py
It may be useful as a reference when implementing this in C.

[tiran]

The statx call was introduced by Kernel 4.11 in 2017. Major LTS Linux distributions like Debian 9, Ubuntu 16.04, and CentOS 7 use older Kernels like Linux 4.9 LTS or 3.10 LTS.

In general we try to support older Kernel ABIs even when Python is compiled on a system with more recent ABI. This means you have to perform a runtime feature detection and fall back to old stat when the syscall fails.

[alexander255]

I thought this might be the case, I'll look into adapting the patch accordingly then.

[tiran]

You can find an example in Python/bootstrap_hash.c that deals with getrandom syscall.

[ntninja]

I have rebased the associated PR for Python 3.12-dev and added backward-compatible support for new statx features introduced in Linux 5.8+. Is the stat fallback still needed?

Also: What about linking with the statx C library symbol on systems that have it? Should it use a weak reference or dlopen instead to ensure that the Python build will work with an older C library? Or should it only unconditionally use the syscall function (in which case all the header definitions will need to be duplicated as well or it would still require a newer system to build)?

[hydrargyrum]

I'm interested in this, is there some progress since?

[vstinner]

I'm interested in this, is there some progress since?

The PR is outdated. Someone has to update it.

[jbosboom]

I'm interested in working on this. I've posted a draft PR as earnest effort (in the sense of earnest money), but we should agree on the answers to these questions before serious review.

Do we fall back to stat?

The previous PR gh-19125 did not fall back to stat if statx was unavailable. A comment in this issue objected to this because statx was relatively new at the time. statx was introduced in Linux 4.11, released April 2017. The oldest currently-supported upstream LTS branch is 5.4, released November 2019. If I am reading the RHEL life cycle and RHEL release dates correctly, the oldest supported release is RHEL8, based on kernel 4.18. It is likely that all mainstream Linux distributions ship kernels supporting statx. On the other hand, there are probably embedded environments stuck on ancient kernels; do their users want to run the newest Python?

A complicating factor is default-deny seccomp syscall filters that have not been updated to allow statx. If they deny with ENOSYS and we fall back to stat, Python will keep working under those filters; if we don't fall back, Python will almost certainly fail during startup.

The code to fall back is simple enough, but testing the fallback path would require running on an old kernel or under a seccomp filter.

My draft PR has a fallback, but no tests for the fallback.

How do we call statx?

In order of increasing reliance on the build environment, here are the ways we can call statx:

1. Hardcode the syscall numbers for architectures we care about, provide our own definition of struct statx and related flags, and call via syscall(). Build is self-sufficient.

2. Use syscall numbers from sys/syscall.h, definitions of struct stat and flags from linux/stat.h, provide our own definitions of flags newer than statx itself, call via syscall(). Build requires Linux kernel userspace API headers >= 4.11. (New userspace API headers can be used on old kernels; they're distinct from the headers used to build kernel modules.)

3. Call using libc's statx wrapper (weakly linked), provide our own definitions of flags newer than statx itself. Requires glibc version 2.28 (August 2018) at build time and run time (but can fall back to stat if runtime libc is old).

My draft PR is option 3.

How do we indicate missing information?

We won't always get everything we ask for in struct statx, depending on the kernel, filesystem type, and filesystem instance (e.g., ext4 supports btime, but really old ext4 filesystems may not). For btime specifically, st_birthtime has never been exposed on Linux, so it's tempting to set it to None when it isn't available. Code that checks the platform before using st_birthtime and code using getattr(st, 'st_birthtime', None) will work, but code that relies on catching AttributeError may not expect st_birthtime to be None -- and there's an example of the latter in test_os.py. So we have to fill it in, either with 0 or with whatever statx returned (not necessarily 0).

The basic stats (also in struct stat) are always initialized, but their bits may not be set in stx_mask. linux/stat.h gives CIFS as an example that might clear STATX_UID/GID if they're being overridden client-side, though I haven't observed this. I have observed a CIFS mount on Linux 6.15.4 that doesn't set STATX_ATIME. We obviously can't set st_atime/uid/gid to None.

I think we should indicate to the application what fields are and aren't reliable, but I don't know how. Exposing stx_mask isn't extensible to other operating systems. We could make up our own mask and flags. We could have a set-like member for tests of the form 'st_birthtime' in st.present, though I don't know how to implement that performantly (and stat is likely hot in some programs). We could introduce int/float subclasses with an extra 'present' member, but I'm sure there are programs that expect exactly int/float. I don't like any of these options.

My draft PR does nothing here.

Which mount ID do we want?

There are two flags to ask for mount IDs, STATX_MNT_ID and STATX_MNT_ID_UNIQUE. The non-unique ID indexes /proc/self/mountinfo and is subject to reuse race conditions; the unique ID can be used with the statmount system call, which requires CAP_SYS_ADMIN and does not currently have a glibc wrapper. Unfortunately they both initialize stx_mnt_id, so we can only get one or the other (unless we call statx twice). Setting both flags is not an error, but I do not think the result is defined; my experiments have always obtained the unique ID.

My draft PR provides the non-unique ID because I think it will be more useful to general Python programs.

How do we test the new members?

ntfs-3g and cifs allow setting btime via the user.ntfs/cifs.creationtime magic xattr. I don't know offhand if ntfs-3g reports btime (being a FUSE filesystem). cifs does, but that requires a server to be available. We could also just create a file, wait, modify the file, and check its btime differs from its mtime and ctime.

We can test stx_mnt_id by looking for it in /proc/self/mountinfo.

Only btrfs and bcachefs currently support stx_subvol. bcachefs has an uncertain future, so we'd want btrfs.

Of the stx_attributes, only STATX_ATTR_MOUNT_ROOT is easily testable, by opening /proc/self/mountinfo and checking the flag is set on mount points. This flag should also be independent of both filesystems.

My draft PR adds no new tests.

Do we want to map attributes to st_flags?

The documentation for the os module claims st_flags "may be available on some Unix systems (such as Linux)", but as far as I can tell it has never been exposed on Linux. The previous PR gh-19125 mapped both Windows and Linux attribute bits to st_flags. Do we want that? Do we want to map st_flags back to attribute bits? (I don't know which systems currently have st_flags or how they are used. Many of them are macOS-specific. The os.chflags documentation says "Availability: Unix, not WASI" but it's not present on Linux. BSD?)

My draft PR doesn't do this.

What do we name stx_attributes?

My PR names it st_attributes, but maybe st_linux_attributes is safer.

Future work

statx supports the AT_STATX_FORCE_SYNC and AT_STATX_DONT_SYNC flags to ask for up-to-date or cached information. (AT_STATX_SYNC_AS_STAT is 0.) I'd like to expose this as a sync=True/False/None argument to os.stat, but I need to see what other operating systems offer first. In particular, the statx flags are hints; if cached info is not available, statx will hit the network/tape/etc. I can imagine that other operating systems might have a flag to say "cached if possible, else fail, never block", which would have to be handled differently.