Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_trt_compile test error in pytorch 23.08 #8061

Closed
KumoLiu opened this issue Sep 2, 2024 · 4 comments · Fixed by #8064
Closed

test_trt_compile test error in pytorch 23.08 #8061

KumoLiu opened this issue Sep 2, 2024 · 4 comments · Fixed by #8064
Labels
bug Something isn't working

Comments

@KumoLiu
Copy link
Contributor

KumoLiu commented Sep 2, 2024 

[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] ERROR: test_unet_value_0_fp32 (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
[2024-09-02T09:35:38.944Z]     return func(*(a + p.args), **p.kwargs, **kw)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 91, in test_unet_value
[2024-09-02T09:35:38.944Z]     trt_output = model(input_example)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
[2024-09-02T09:35:38.944Z]     return self._call_impl(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1514, in _call_impl
[2024-09-02T09:35:38.944Z]     return forward_call(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 493, in trt_forward
[2024-09-02T09:35:38.944Z]     return self._trt_compiler.forward(self, argv, kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 350, in forward
[2024-09-02T09:35:38.944Z]     raise e
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 342, in forward
[2024-09-02T09:35:38.944Z]     self._build_and_save(model, build_args)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 483, in _build_and_save
[2024-09-02T09:35:38.944Z]     engine_bytes = self._onnx_to_trt(str(onnx_path))
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 411, in _onnx_to_trt
[2024-09-02T09:35:38.944Z]     return engine_bytes_from_network(network, config=CreateConfig(profiles=profiles, **build_args))
[2024-09-02T09:35:38.944Z] TypeError: CreateConfig.__init__() got an unexpected keyword argument 'bf16'
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] ERROR: test_unet_value_1_fp16 (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
[2024-09-02T09:35:38.944Z]     return func(*(a + p.args), **p.kwargs, **kw)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 91, in test_unet_value
[2024-09-02T09:35:38.944Z]     trt_output = model(input_example)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
[2024-09-02T09:35:38.944Z]     return self._call_impl(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1514, in _call_impl
[2024-09-02T09:35:38.944Z]     return forward_call(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 493, in trt_forward
[2024-09-02T09:35:38.944Z]     return self._trt_compiler.forward(self, argv, kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 350, in forward
[2024-09-02T09:35:38.944Z]     raise e
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 342, in forward
[2024-09-02T09:35:38.944Z]     self._build_and_save(model, build_args)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 483, in _build_and_save
[2024-09-02T09:35:38.944Z]     engine_bytes = self._onnx_to_trt(str(onnx_path))
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 411, in _onnx_to_trt
[2024-09-02T09:35:38.944Z]     return engine_bytes_from_network(network, config=CreateConfig(profiles=profiles, **build_args))
[2024-09-02T09:35:38.944Z] TypeError: CreateConfig.__init__() got an unexpected keyword argument 'bf16'
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] FAIL: test_handler (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 67, in test_handler
[2024-09-02T09:35:38.944Z]     self.assertIsNotNone(net1._trt_compiler.engine)
[2024-09-02T09:35:38.944Z] AssertionError: unexpectedly None
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Ran 15905 tests in 4410.586s
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] FAILED (failures=1, errors=4, skipped=342)
@KumoLiu KumoLiu added the bug Something isn't working label Sep 2, 2024
@KumoLiu
Copy link
Contributor Author

KumoLiu commented Sep 2, 2024

Hi @borisfom, could you please take a look at this error? Our daily CI runs tests on the PyTorch base image 23.08. If test_trt_compile isn’t compatible with this version, could you please add a skip or check for the minimum installed version?

MONAI/tests/utils.py

Line 261 in 6a0e1b0

class SkipIfBeforePyTorchVersion:

MONAI/monai/utils/module.py

Line 342 in 6a0e1b0

version_checker: Callable[..., bool] = min_version,

Thanks.

KumoLiu added a commit to KumoLiu/MONAI that referenced this issue Sep 2, 2024
@KumoLiu
workaround for Project-MONAI#8061
2c6996e 
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu KumoLiu mentioned this issue Sep 2, 2024
Merged
7 tasks
KumoLiu added a commit that referenced this issue Sep 2, 2024
@KumoLiu @ericspod
Ignore warning from nptyping as workaround (#8062)
dbfe418 
workaround for #8061

### Description

Ignore warning from nptyping as workaround

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
@borisfom
Copy link
Contributor

borisfom commented Sep 3, 2024

I have pushed a fix for that issue.

@KumoLiu
Copy link
Contributor Author

KumoLiu commented Sep 3, 2024

Hi @borisfom, may I ask where did you push the fix?

@borisfom borisfom mentioned this issue Sep 3, 2024
Merged
7 tasks
@borisfom
Copy link
Contributor

borisfom commented Sep 3, 2024

Oh, my original PR got merged ...
Here: #8064

@KumoLiu KumoLiu closed this as completed in aea46ff Sep 4, 2024
hjmjohnson pushed a commit to hjmjohnson/MONAI that referenced this issue Sep 8, 2024
@borisfom @pre-commit-ci
@KumoLiu @yiheng-wang-nv @binliunls @hjmjohnson
Trt compiler fixes (Project-MONAI#8064)
89af7b6 
Fixes Project-MONAI#8061.

### Description

Post-merge fixes for trt_compile()

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
Signed-off-by: Yiheng Wang <vennw@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com>
Co-authored-by: Yiheng Wang <vennw@nvidia.com>
Co-authored-by: binliunls <107988372+binliunls@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Trt compiler fixes borisfom/MONAI
2 participants
@borisfom @KumoLiu