Jetson Orin Build

[lippman1125]

How to build an wheel to deploy on Jetson Orin?

[windreamer]

I have made a docker build for Jetson in #3834 . But I do not have a Jetson Orin device to test. Could you please help us test the build?

[nick1115]

@windreamer Hi, I build image as follows:

1. git clone https://github.com/windreamer/lmdeploy.git
2. cd lmdeploy
3. git checkout try_jetson_docker
4. sudo docker build -f docker/Dockerfile.jetson -t lmdeploy:09101732 .

Successfully builded image. But when I try to run a container, it fails

orin@ubuntu:~/workspace/code/lmdeploy$ sudo docker run --rm -ti --runtime nvidia lmdeploy:09101732 bash
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: failed to get image requirements: failed to get version: invalid CUDA version cu12.6: invalid version string: unknown

Run 'docker run --help' for more information

Can you give me some help?

[windreamer]

@nick1115 I think this should be due to incompatibility with the docker (CUDA 12.6) with your device. Can I know which version of l4t is on your Jetson Orin?

[windreamer]

https://github.com/InternLM/lmdeploy/pull/3834/files#diff-ed7c5925328acbcf7edf33a0f9ac70d1c674c8ad9266d124b5e00334b4997e97R2

Here we are using l4t version r36..2.0

[nick1115]

@windreamer I comfirmed the CUDA version is 12.6 on my Orin, but L4T is r36.4.4, and I can not find an appropriate l4t-base for it

[windreamer]

@nick1115 Thank you for your information. It seems I set an Environment Variable CUDA_VERSION. But this Environment Variable is an important one to make nvidia cuda runtime to determine if versions match.

I think I need to change the Docker file to avoid the use of the environment variable.

[windreamer]

I think you can try with windreamer/lmdeploy@try_jetson_docker again. I have renamed the environment variable.

[nick1115]

I think you can try with windreamer/lmdeploy@try_jetson_docker again. I have renamed the environment variable.

@windreamer I rebuild image following your commit, and docker run success, however, lmdeploy occurs error, Traceback as below

orin@ubuntu:~/workspace/code/lmdeploy$ sudo docker run --rm -ti --runtime nvidia lmdeploy:09111527 bash
root@875c395ad35a:/opt/lmdeploy# lmdeploy --help
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 5, in <module>
    from lmdeploy.cli import run
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/__init__.py", line 3, in <module>
    from .api import client, pipeline, serve
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/api.py", line 5, in <module>
    from .archs import autoget_backend_config, get_task
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/archs.py", line 5, in <module>
    from transformers import AutoConfig
  File "/opt/py3/lib/python3.10/site-packages/transformers/__init__.py", line 27, in <module>
    from . import dependency_versions_check
  File "/opt/py3/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/__init__.py", line 24, in <module>
    from .auto_docstring import (
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/auto_docstring.py", line 30, in <module>
    from .generic import ModelOutput
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/generic.py", line 51, in <module>
    import torch  # noqa: F401
  File "/opt/py3/lib/python3.10/site-packages/torch/__init__.py", line 416, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudss.so.0: cannot open shared object file: No such file or directory

[nick1115]

@windreamer I try to workaround as below

• build pytorch 2.8.0 with CUDA on Orin
• docker run lmdeploy image with -v /local_path_to_pytorch:/pytorch
• pip uninstall torch
• pip install /pytorch/torch-2.8.0-cp310-cp310-linux_aarch64.whl
• apt update && apt install openmpi-bin openmpi-common libopenmpi-dev

and then, try lmdeploy --help, it worked

root@d9cac4131599:/opt/lmdeploy# lmdeploy --help
/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
usage: lmdeploy [-h] [-v] {lite,serve,list,check_env,chat} ...

The CLI provides a unified API for converting, compressing and deploying large language models.

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

Commands:
  lmdeploy has following commands:

  {lite,serve,list,check_env,chat}
    lite                Compressing and accelerating LLMs with lmdeploy.lite module
    serve               Serve LLMs with openai API
    list                List the supported model names.
    check_env           Check the environmental information.
    chat

[windreamer]

I think you can try with windreamer/lmdeploy@try_jetson_docker again. I have renamed the environment variable.

@windreamer I rebuild image following your commit, and docker run success, however, lmdeploy occurs error, Traceback as below

orin@ubuntu:~/workspace/code/lmdeploy$ sudo docker run --rm -ti --runtime nvidia lmdeploy:09111527 bash
root@875c395ad35a:/opt/lmdeploy# lmdeploy --help
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 5, in <module>
    from lmdeploy.cli import run
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/__init__.py", line 3, in <module>
    from .api import client, pipeline, serve
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/api.py", line 5, in <module>
    from .archs import autoget_backend_config, get_task
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/archs.py", line 5, in <module>
    from transformers import AutoConfig
  File "/opt/py3/lib/python3.10/site-packages/transformers/__init__.py", line 27, in <module>
    from . import dependency_versions_check
  File "/opt/py3/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/__init__.py", line 24, in <module>
    from .auto_docstring import (
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/auto_docstring.py", line 30, in <module>
    from .generic import ModelOutput
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/generic.py", line 51, in <module>
    import torch  # noqa: F401
  File "/opt/py3/lib/python3.10/site-packages/torch/__init__.py", line 416, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudss.so.0: cannot open shared object file: No such file or directory

Yeah， the new version of PyTorch seems introduced some new dependencies. I will try to fix it later.

[windreamer]

@windreamer I try to workaround as below

• build pytorch 2.8.0 with CUDA on Orin
• docker run lmdeploy image with -v /local_path_to_pytorch:/pytorch
• pip uninstall torch
• pip install /pytorch/torch-2.8.0-cp310-cp310-linux_aarch64.whl
• apt update && apt install openmpi-bin openmpi-common libopenmpi-dev

and then, try lmdeploy --help, it worked

root@d9cac4131599:/opt/lmdeploy# lmdeploy --help
/opt/py3/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
usage: lmdeploy [-h] [-v] {lite,serve,list,check_env,chat} ...

The CLI provides a unified API for converting, compressing and deploying large language models.

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

Commands:
  lmdeploy has following commands:

  {lite,serve,list,check_env,chat}
    lite                Compressing and accelerating LLMs with lmdeploy.lite module
    serve               Serve LLMs with openai API
    list                List the supported model names.
    check_env           Check the environmental information.
    chat

Glad to hear it is working!

[windreamer]

I think you can try with windreamer/lmdeploy@try_jetson_docker again. I have renamed the environment variable.

@windreamer I rebuild image following your commit, and docker run success, however, lmdeploy occurs error, Traceback as below

orin@ubuntu:~/workspace/code/lmdeploy$ sudo docker run --rm -ti --runtime nvidia lmdeploy:09111527 bash
root@875c395ad35a:/opt/lmdeploy# lmdeploy --help
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 5, in <module>
    from lmdeploy.cli import run
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/__init__.py", line 3, in <module>
    from .api import client, pipeline, serve
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/api.py", line 5, in <module>
    from .archs import autoget_backend_config, get_task
  File "/opt/py3/lib/python3.10/site-packages/lmdeploy/archs.py", line 5, in <module>
    from transformers import AutoConfig
  File "/opt/py3/lib/python3.10/site-packages/transformers/__init__.py", line 27, in <module>
    from . import dependency_versions_check
  File "/opt/py3/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 16, in <module>
    from .utils.versions import require_version, require_version_core
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/__init__.py", line 24, in <module>
    from .auto_docstring import (
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/auto_docstring.py", line 30, in <module>
    from .generic import ModelOutput
  File "/opt/py3/lib/python3.10/site-packages/transformers/utils/generic.py", line 51, in <module>
    import torch  # noqa: F401
  File "/opt/py3/lib/python3.10/site-packages/torch/__init__.py", line 416, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudss.so.0: cannot open shared object file: No such file or directory

This issue should be fixed now.

[github-actions]

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.