gh-112087: Make __sizeof__ and listiter_{len, next} to be threadsafe

[corona10]

• Issue: Make list objects thread-safe in --disable-gil builds #112087

[corona10]

@colesbury

• listiter_reduce_general looks safe, so I think that we don't have to touch it.
• The rest of the APIs can be migrated once the gh-114329: Add PyList_GetItemRef function #114504 is merged.

[colesbury]

Regarding iterators: list (and dict) iterators current clear it->it_seq field when the iterator is exhausted. This will make it harder to make the iterators thread-safe because it_seq is mutable. In the free-threaded build, I think we should avoid clearing the it_seq field until the iterator is deallocated.

[colesbury]

The name is a bit confusing to me -- I'm not sure what AS_POSSIBLE means here. Maybe just LOAD_SSIZE(). It's not very descriptive, but at least it's short.

The style I've seen more commonly in CPython looks like:

#ifdef Py_GIL_DISABLED
#  define LOAD_SSIZE_ATOMIC_AS_POSSIBLE(value) _Py_atomic_load_ssize_relaxed(&value)
#endif

[corona10]

Regarding iterators: list (and dict) iterators current clear it->it_seq field when the iterator is exhausted. This will make it harder to make the iterators thread-safe because it_seq is mutable. In the free-threaded build, I think we should avoid clearing the it_seq field until the iterator is deallocated.

@colesbury
I will update the implementation through this PR after #114916 is merged :)

[corona10]

This test was added from #101769, not sure that it will be okay to change the implementation detail.

[carljm]

I think it is OK, see my other comment.

[corona10]

@colesbury
#114843 (review)
I addressed your code review, including the change of implementation details when the iterator is exhausted.

Since it touched on the implementation details, I think that we should listen to other core developers' opinions, too.

1. Change it from CPython 3.13 for the default build and free-threaded build
2. Only change it for a free-threaded build (Be more complex but possible option)

cc @carljm @JelleZijlstra

@gvanrossum
I only touched the tier 1 opcode, but if we need to touch the tier 2 opcode too, please let me know.
The refleak test for tier 2 of test_list and test_iter had already failed in the main branch.

Tier 1

$> ./python.exe -m test test_list test_iter -R 3:3
Using random seed: 1891590703
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 5.97 Run 2 tests sequentially
0:00:00 load avg: 5.97 [1/2] test_list
beginning 6 repetitions
123456
......
0:00:01 load avg: 5.73 [2/2] test_iter
beginning 6 repetitions
123456
......

== Tests result: SUCCESS ==

All 2 tests OK.

Total duration: 1.9 sec
Total tests: run=114
Total test files: run=2/2
Result: SUCCESS

Tier 2

$ ./python.exe -Xuops -m test test_list test_iter -R 3:3
Using random seed: 542131965
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 6.22 Run 2 tests sequentially
0:00:00 load avg: 6.22 [1/2] test_list
beginning 6 repetitions
123456
......
test_list leaked [4, 3, 2] references, sum=9
test_list leaked [6, 4, 4] memory blocks, sum=14
0:00:01 load avg: 6.22 [2/2/1] test_iter -- test_list failed (reference leak)
beginning 6 repetitions
123456
......
test_iter leaked [8, 9, 6] references, sum=23
test_iter leaked [12, 11, 8] memory blocks, sum=31
test_iter failed (reference leak)

== Tests result: FAILURE ==

2 tests failed:
    test_iter test_list

Total duration: 2.0 sec
Total tests: run=114
Total test files

[carljm]

This is the change causing the behavior difference in the test. Previously, an empty iterator would always reduce to the iter builtin, now it will reduce to reversed if it was a reverse iterator. IMO the new behavior is more correct (consistent with the non-empty behavior) and we can consider this a bugfix.

[corona10]

Okay in that case, we need to create backport patches for this.

[corona10]

@gvanrossum @colesbury gentle ping :)

[colesbury]

I think _GUARD_NOT_EXHAUSTED_LIST also needs to be adjusted for the free-threaded build because seq == NULL no longer signifies an exhausted iterator for the free-threaded build. Like with _ITER_JUMP_LIST, I think the deopt can be reduced to:

DEOPT_IF((size_t)it->it_index >= (size_t)PyList_GET_SIZE(seq));

[colesbury]

The indentation I've seen more commonly in CPython looks like:

#ifdef Py_GIL_DISABLED
#  define LOAD_SSIZE(value) _Py_atomic_load_ssize_relaxed(&value)
...
#endif

[gvanrossum]

@gvanrossum @colesbury gentle ping :)

I trust Sam. My queue is very long and my proessing capacity is reduced.

[colesbury]

LGTM