Attach to specific multiprocessing subprocesses via the CLI?

[garrett361]

Is it possible to attach to a specific multiprocessing subprocess using only the debugpy CLI?

Example:

import multiprocessing as mp

def target(rank):
    a = 2
    b = rank * a
    print(b)
    print(f"Done on {rank=}")

def test():
    procs = [
        mp.Process(target=target, name=f"rank-{rank}", args=(rank,))
        for rank in range(4)
    ]
    for p in procs:
        p.start()
    for p in procs:
        p.join()


if __name__ == "__main__":
    test()

with subProcess = True I can do python -m debugpy --listen 5678 --wait-for-client <path-to-file.py> and then attach to a (seemingly random, but usually rank-zero) process by specifying port = 5678 and debug that one process. But I can't figure out how to either attach to a specifc process or get any of the other ranks to step through their code.

Is it possible to achieve this just through the CLI?

One thing that does work is to explicitly insert the debugpy API calls into the source code:

import multiprocessing as mp

def target(rank):
    import debugpy

    debugpy.listen(5678 + rank)
    debugpy.wait_for_client()
    a = 2
    b = rank * a
    print(b)
    print(f"Done on {rank=}")

def test():
    procs = [
        mp.Process(target=target, name=f"rank-{rank}", args=(rank,))
        for rank in range(4)
    ]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

if __name__ == "__main__":
    test()

Just calling python <path-to-file.py>, I can attach to any port in {5678, 5679, 5680, 5681} and connect to the expected rank. But, it would be great if there were a way to achieve this without needing to change the source code.

[rchiodo]

Transferring to a discussion item as this isn't a bug.

[garrett361]

Ah, sorry, thought I submitted it as a discussion.

[rchiodo]

This launch.json should let you pick any process to debug:

    {
      "name": "Python Debugger: Attach to Process",
      "type": "debugpy",
      "request": "attach",
      "processId": "${command:pickProcess}",
    },

That should let you pick a python process to debug and then inject debugpy into it without you having to add any explicit debugpy code.

[rchiodo]

Although you asked if this was possible through the CLI? I think this might work?

https://github.com/microsoft/debugpy/wiki/Command-Line-Reference#--pid-pid

You'd have to know the pid though.

[garrett361]

Yeah, I was trying the --pid flag, but haven't gotten it to work yet. In this scenario, every process has the same --port and (necessarily) different --pids, is that right?

[rchiodo]

If you use the pid flag, you're only debugging a single process. So you'd do something like this:

• start you app: python foo.py
• have the app output the pids as the subprocesses run
• start debugpy to attach to that single pid python -m debugpy --listen:5678 --pid 12345 (getting this pid from the output)
• Have VS code connect to that port

[rchiodo]

Or if you're using VS code, don't bother with the command line injection, just use the launch.json I showed above. You don't need to run debugpy on your own.

[garrett361]

Thanks, I'm trying to get this to work with nvim-dap-python, so I think my questions need to go over there at this point.

There, python -m debug.adapter is used everywhere, for reasons I don't yet understand, and that module doesn't support the --pid flag