Skip to content

Commit d198cdf

Browse files
Grigory Sizovfacebook-github-bot
Grigory Sizov
authored andcommitted
Race condition in test_chunk (facebookincubator#935)
Summary: Pull Request resolved: facebookincubator#935 `test_chunk` is flaky: ``` input_shape=[{'depth': 0, 'name': None, 'nop': False, 'symbolic_value': 17, 'values': [17]}, {'depth': 0, 'name': None, 'nop': False, 'symbolic_value': 5, 'values': [5]}, {'depth': 0, 'name': None, 'nop': False, 'symbolic_value': 29, 'values': [29]}], chunks=2, dim=0 ... raise ValueError( ValueError: Did not get correct number of outputs expected 5, got 10 ``` Even though the error log contains `chunks=2`, the op sees a different number of chunks. The only explanation I see is a race condition when different tests execute on the same machine, and write to the same test directory, as we've seen before in other AIT tests. This diff makes directory unique for every test case, similar to D46323114 I can't reproduce the issue locally with stress test. Reviewed By: aakhundov Differential Revision: D49571274 fbshipit-source-id: 7daf04bf64d729eee1f3258e2d04958fafe77027
1 parent ab9d8e7 commit d198cdf

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

tests/unittest/ops/test_chunk.py

+6-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,12 @@ def _run_chunk(
4949
Y._attrs["name"] = "output_{}".format(idx)
5050
Y._attrs["is_output"] = True
5151

52-
module = compile_model(Ys, target, "./tmp", "chunk")
52+
# Use unique test name to avoid race condition
53+
shape_hash = hash(str(input_shape))
54+
test_name = f"chunks_{chunks}_{dim}_{input_type}_{shape_hash}"
55+
module = compile_model(
56+
Ys, target, "./tmp", test_name=test_name, dll_name=f"{test_name}.so"
57+
)
5358

5459
for batch_size in input_shape[0]._attrs["values"]:
5560
logging.info(f"Testing {batch_size=}")

0 commit comments

Comments
 (0)