Improve documentation on multiprocessing synchronization-between-processes

[HCharlie]

Documentation

https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes
(A clear and concise description of the issue.)

when running the provided example locally, unlike other examples on this page which can print valid stdout, the code in this section will only print out the error below, which is caused by not providing the Process.join() call at the end.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

An example patch could be

from multiprocessing import Process, Lock

   def f(l, i):
       l.acquire()
       try:
           print('hello world', i)
       finally:
           l.release()

   if __name__ == '__main__':
       lock = Lock()

       processes = []

       for num in range(10):
           p = Process(target=f, args=(lock, num))
           processes.append(p)
           p.start()
   
       for p in processes:
           p.join()

Linked PRs

• GH-116526 patch documentation Synchronization between processes example #116527

[gaogaotiantian]

To be honest this is a pretty bad example to begin with - because you won't notice any difference if you simply remove the lock so it's pretty confusing when users try it out. The stdout is buffered in most environments and won't flush until seeing a newline. We should probably change this to something more obvious - multiple prints maybe (building a triangle?).

The exception mentioned in this issue can be reproduced with spawn, but might be hard to repro with fork as fork is too fast.

[HCharlie]

Hey @gaogaotiantian , thanks for your reply. I think your point makes sense, and it still confuses me when reading it as well. Either with or without the lock, the example doesn't seem to emphasize what difference this lock brings. Tbh, my original intent was to just complete the example to remove some exceptions, now I think maybe a new example is needed.

[HCharlie]

one quick example I could think of is as below, we increment a value 3 times in a process.

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, lock, process_id):
    lock.acquire()
    for _ in range(3):
        counter.value += 1
        print(f"Counter value: {counter.value} incremented by the {process_id} process")
    lock.release()

if __name__ == "__main__":
    # Shared counter value
    counter = multiprocessing.Value('i', 0)
    
    # Creating a Lock
    lock = multiprocessing.Lock()

    # Creating processes
    processes = []
    for j in range(4):  # Creating 4 processes
        p = multiprocessing.Process(target=increment_counter, args=(counter, lock, j))
        processes.append(p)
        p.start()

    # Waiting for all processes to finish
    for p in processes:
        p.join()

    # Output the final value of the counter
    print("Final counter value:", counter.value)

with the lock, the value will consecutively be incremented by the same process, and the example will output something like this

Counter value: 1 incremented by the 1 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 1 process
Counter value: 4 incremented by the 0 process
Counter value: 5 incremented by the 0 process
Counter value: 6 incremented by the 0 process
Counter value: 7 incremented by the 2 process
Counter value: 8 incremented by the 2 process
Counter value: 9 incremented by the 2 process
Counter value: 10 incremented by the 3 process
Counter value: 11 incremented by the 3 process
Counter value: 12 incremented by the 3 process
Final counter value: 12

if we remove the lock-related code,

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, process_id):

    for _ in range(3):
        counter.value += 1
        print(f"Counter value: {counter.value} incremented by the {process_id} process")


if __name__ == "__main__":
    # Shared counter value
    counter = multiprocessing.Value('i', 0)

    # Creating processes
    processes = []
    for j in range(4):  # Creating 4 processes
        p = multiprocessing.Process(target=increment_counter, args=(counter, j))
        processes.append(p)
        p.start()

    # Waiting for all processes to finish
    for p in processes:
        p.join()

    # Output the final value of the counter
    print("Final counter value:", counter.value)

it will randomly print out some duplicates like below, eventually not even reaching 12.

Counter value: 1 incremented by the 3 process
Counter value: 1 incremented by the 2 process
Counter value: 1 incremented by the 0 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 3 process
Counter value: 4 incremented by the 2 process
Counter value: 4 incremented by the 0 process
Counter value: 4 incremented by the 1 process
Counter value: 5 incremented by the 3 process
Counter value: 6 incremented by the 2 process
Counter value: 6 incremented by the 0 process
Counter value: 6 incremented by the 1 process
Final counter value: 6

wdyt?