GC performance regression in free threaded build

[gptsarthak]

Bug report

Bug description:

I've identified a significant performance regression when using Python's free-threaded mode with shared list appends. In my test case, simply appending to a shared list causes a 10-15x performance decrease compared to normal Python operation.

Test Case:

import itertools
import time

def performance_test(n_options=5, n_items=5, iterations=50):
    list = []
    
    def expensive_operation():
        # Create lists of tuples
        data = []
        for _ in range(n_options):
            data.append([(f"a{i}", f"b{i}") for i in range(n_items)])
        
        # Generate all combinations and create result tuples
        results = []
        for combo in itertools.product(*data):
            result = tuple((x[0], x[1], f"name_{i}") for i, x in enumerate(combo))
            results.append(result)
        
        # Commenting the following line solves the performance regression in free-threaded mode
        list.append(results)
        return results

    start = time.time()
    for _ in range(iterations):
        result = expensive_operation()
    
    duration = time.time() - start
    print(f"n_options={n_options}, n_items={n_items}, iterations={iterations}")
    print(f"Time: {duration:.4f}s, Combinations: {len(result)}")
    return duration

if __name__ == "__main__":
    print("Python Performance Regression Test")
    print("-" * 40)
    performance_test()

Results:

• Standard Python3.13: 0.1290s
• Free-threaded Python3.13t: 2.1643s
• Free-threaded Python 3.14.0a7: 2.1923s
• Free-threaded Python3.13t with list.append commented out: 0.1332s

The regression appears to be caused by contention on the per-list locks and reference count fields when appending to a shared list in free-threaded mode.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-132917: Check resident set size (RSS) before GC trigger. #133399
• gh-132917: Use RSS + swap for estimate of process memory usage #133464
• gh-132917: Fix data race detected by tsan #133508
• gh-132917: Use /proc/self/status for mem usage info. #133544
• [3.14] gh-132917: Use /proc/self/status for mem usage info. (GH-133544) #133718

[colesbury]

Yes, this is expected and not a bug.

[corona10]

Closing this issue since this is expected.

[colesbury]

To add a bit more context: if you have an inherently serial operation, like incrementing a counter or appending to a list, it's going to be slower when run with multiple threads on multiple CPUs than if you have some sort of coarse grained serialization (like the GIL) or run it on a single CPU.

[srinivasreddy]

@colesbury I have been reading GIL removal pep. And this issue has been closed as "not planned" , Do we have any plans to fix performance partially at least, especially in the cases of list.append(results) ?

[colesbury]

No

[terryjreedy]

Is this issue specific to .append or does it apply to .extend and slice insertions?

Is/should this be documented, warned against, somewhere other than here?

[colesbury]

I don't think we need to specifically warn against it any more than we would warn against building a list via successive list.insert(0, item).

This is an inherently serial operation. If you try to perform a serial operation from multiple CPUs, you introduce lots of extra communication and everything gets slower. That's normal and not really specific to Python, just like quadratic behavior of some data structures is normal and not specific to Python.

[gptsarthak]

@colesbury Thank you for the response. I understand the general principle about serial operations being slower with multiple threads. However, I want to clarify that my MRE doesn't use any threading at all - it's entirely single-threaded code.

The performance regression I'm observing is occurring in single-threaded code simply by running it in free-threaded Python mode. This suggests that there's a significant overhead for thread safety mechanisms even when no actual threading is used.

This seems important to document, since while experienced concurrent programmers will understand the underlying reasons, many Python users might not expect their unthreaded code to suddenly run 10-15x slower.

Would it be worth mentioning this behavior in the free-threading documentation - that even single-threaded code using mutable shared data structures may experience performance regressions due to the thread safety overhead?

Also, is there any scope for optimizing free-threaded mode under these conditions in future releases?