nogil set `clear` causes concurrent `__str__` to print as empty dict

[luisggpina]

Bug report

Bug description:

Hi,

We're a research group focused on testing concurrent runtimes. Our work-in-progress prototype found that the current nogil build __str__ can return "{}" (empty dict) instead of the expected "set()" when there is a concurrent clear() operation. The program below shows the wrong behavior:

import threading
import sys
def t0(b1,s,res):
    b1.wait()
    s.clear()

def t1(b1,s,res):
    b1.wait()
    res.append(s.__str__())

def Test():
  s =  {17, 18, 'a', 'b', 'c', 'd', 'e'}
  threads=[]
  barrier = threading.Barrier(2)
  res = []
  threads.append(threading.Thread(target= t0, args=(barrier, s,res)))
  threads.append(threading.Thread(target= t1, args=(barrier, s,res)))
  for i in range(0, len(threads)):
      threads[i].start()
  for i in range(0, len(threads)):
      threads[i].join()
  if res[0] == "{}":
      print("found bug: " + res[0])

print("test begin...")
for i in range(0,50000):
  threads = []
  if i % 1000 == 0:
      print(i)
  for i in range(0,100):
      threads.append(threading.Thread(target= Test))
  for t in threads:
      t.start()
  for t in threads:
      t.join()
print("test Done")

Sample output:

test begin...
0
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}

This behavior can be observed quite readily. We tested it on a number of x86_64 and one ARM machine.

@flypoodles and @overlorde are part of the team, adding them so they get notified about further discussion.

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-129967: Fix race condition in repr(set) #129978
• [3.13] gh-129967: Fix race condition in repr(set) (gh-129978) #130020

[colesbury]

Thanks - this is a bit of a weird one:

• We lock the set before calling str() or repr()
• However, when printing the set we call PySequence_List, which internally tries to lock both the newly created list and the set a second time. This leads to the unlocking + relocking behavior, which allows another thread to clear the set in the meantime. @Yhg1s's recursive critical section optimization doesn't apply here because it's a two-object critical section, not a simple one object critical section.

This can happen in the GIL-enabled build too, at least in Python 3.11 and earlier, where PySequence_List() can trigger arbitrary code by the GC running.

I think we should do two things:

1. We should make set_repr_lock_held() more robust and check the size of the list that results from PySequence_List.
2. We should attempt to avoid the slow relocking behavior in set_repr_lock_held.

[colesbury]

Thanks @luisggpina - this is fixed now in the main and 3.13 branches.