Skip to content

Commit 85931e9

Browse files
henrylhtsangmarkc-614
authored andcommitted
[BE] Save attributes for CppCompileError for pickleing (pytorch#160294)
Differential Revision: [D79977408](https://our.internmc.facebook.com/intern/diff/D79977408/) Context: When testing cutlass backend and used autotune with subproc, sometimes I would see C++ compilation error (expected) followed by ``` Traceback (most recent call last): File "/torch/_inductor/autotune_process.py", line 175, in get result = TuningProcess.recv(self.read_pipe) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/torch/_inductor/autotune_process.py", line 99, in recv return pickle.load(read_pipe) ^^^^^^^^^^^^^^^^^^^^^^ TypeError: CppCompileError.__init__() missing 1 required positional argument: 'output' ``` which is unexpected. After asking claude, it seems > Now I can see the issue. The `CppCompileError` class requires two arguments: `cmd` (a list of strings) and `output` (a string). However, when exceptions are being pickled and unpickled across process boundaries, the pickling process might not be preserving the constructor arguments correctly. > > The problem is likely that when a `CppCompileError` is raised in the subprocess and then pickled/unpickled through the `recv` function, the unpickling process is trying to reconstruct the exception but doesn't have the required constructor arguments. > > The issue is clear now. The `CppCompileError` class doesn't have custom pickle methods (`__reduce__`, `__getstate__`, `__setstate__`), so when it's pickled and unpickled across process boundaries, Python's default pickling mechanism tries to reconstruct it but fails because it doesn't preserve the constructor arguments properly. > > The solution is to add a `__reduce__` method to the `CppCompileError` class to ensure it can be properly pickled and unpickled. Let me implement this fix: Adding these seem to help. fbcode repro: [D79977541](https://www.internalfb.com/diff/D79977541) Pull Request resolved: pytorch#160294 Approved by: https://github.com/masnesral
1 parent 57c275d commit 85931e9

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

torch/_inductor/exc.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,9 @@ def __init__(self, cmd: list[str], output: str) -> None:
9292
if isinstance(output, bytes):
9393
output = output.decode("utf-8")
9494

95+
self.cmd = cmd
96+
self.output = output
97+
9598
super().__init__(
9699
textwrap.dedent(
97100
"""
@@ -108,6 +111,9 @@ def __init__(self, cmd: list[str], output: str) -> None:
108111
.format(cmd=" ".join(cmd), output=output)
109112
)
110113

114+
def __reduce__(self) -> tuple[type, tuple[list[str], str]]:
115+
return (self.__class__, (self.cmd, self.output))
116+
111117

112118
class CUDACompileError(CppCompileError):
113119
pass

0 commit comments

Comments
 (0)