ModuleNotFoundError when building with --simlink-install

[BrunoB81HK]

When building the example_python package inside my workspace, I get the ModuleNotFoundError. This happens everytime I try to build with --symlink-install. I first encountered this bug with my package and I decided to try the example directly to see if it happened with it too and it does. here is the traceback:

Traceback (most recent call last):
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 33, in <module>
    sys.exit(load_entry_point('generate-parameter-module-example', 'console_scripts', 'test_node')())
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/colcon_ws/build/generate_parameter_module_example/generate_parameter_module_example/minimal_publisher.py", line 33, in <module>
    from generate_parameter_module_example.admittance_parameters import (
ModuleNotFoundError: No module named 'generate_parameter_module_example.admittance_parameters'

I also checked my PYTHONPATH and I think that it may be related to name collisions.

I saw that @pac48 was referencing to some side-effects when using --symlink-install in #122, is this related?

[pac48]

@BrunoB81HK Which version of Ubuntu and ROS 2 are you using? I can build with --symlink-install and run the example on this image no problem.

[BrunoB81HK]

I'm using humble with Ubuntu 22.04. We are also using docker and our container is generated from this one.

[pac48]

Are you building with any extra options? When I run colcon build --symlink-install to build and ros2 run generate_parameter_module_example test_node --ros-args --params-file src/generate_parameter_library/example_python/config/implementation.yaml to run, it works even on the image you mentioned.

[BrunoB81HK]

OK, that is weird. The only other that we use is the following --cmake-args -DCMAKE_BUILD_TYPE=Release. Maybe the error is on my side.

[pac48]

Okay, that is strange. The only other thing I'd suggest is to ensure that rosdep is run: rosdep install --from-paths src --ignore-src -y before building and running.

[BrunoB81HK]

rosdep install --from-paths src --ignore-src -y is indeed called before building and running.

For my custom package, python can find it, but I think its name collide with another package so that parameters.py is not found.
I compiled the differences between a code built with and without --symink-install below.

In my case, using --symink-install cause a duplicated module in /colcon_ws/build that is added in the python path and, since this module is a symlink to the /colcon_ws/src, it doesn't contain the generated parameters.py.
The generated parameters.py is theoretically reachable from /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages, but is hidden behind the one in /home/user/colcon_ws/build/pkg_name.

The most surprising thing to me is that it happens to me but not to you. @pac48, do you have any idea why this happens and how I could remedy this problem? Is it a problem with my setup? Or a problem with my module naming?

Installed with --symlink-install

Executed script (from __file__):

/home/user/colcon_ws/build/pkg_name/pkg_name/script.py

Python path (from sys.path):

Paths
1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name
2️⃣ /home/user/colcon_ws/build/pkg_name
3️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages

Reachable modules (from sys.modules):

Module	Location
pkg_name.script	2️⃣ /home/user/colcon_ws/build/pkg_name
pkg_name	2️⃣ /home/user/colcon_ws/build/pkg_name

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── 2️⃣ pkg_name
│       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
│       ├── resource
│       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
│       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
│       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
│       ├── ...
│       └── share
│           └── pkg_name
│               └── hook
│                   ├── pythonpath_develop.dsv
│                   ├── pythonpath_develop.ps1
│                   └── pythonpath_develop.sh
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 3️⃣ site-packages
│   │   │           ├── pkg-name.egg-link
│   │   │           └── pkg_name
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Installed without --symlink-install

Executed script (from __file__):

/home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages/pkg_name/script.py

Python path (from sys.path):

Paths
1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name
2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
/opt/ros/humble/lib/python3.10/site-packages
/opt/ros/humble/local/lib/python3.10/dist-packages
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages

Reachable modules (from sys.modules):

Module	Location
pkg_name.script	2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages
pkg_name	2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── pkg_name
│       ├── pkg_name
│       │   └── parameters.py
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── build
│       │   └── lib
│       │       └── pkg_name
│       │           ├── __init__.py
│       │           └── script.py
│       └── ...
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 2️⃣ site-packages
│   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
│   │   │           │   └── ...
│   │   │           └── pkg_name
│   │   │               ├── __init__.py
│   │   │               ├── __pycache__
│   │   │               │   └── ...
│   │   │               ├── script.py
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Difference in the directory structure

  colcon_ws
  ├── build
  │   ├── COLCON_IGNORE
  │   └── pkg_name
- │       ├── pkg_name
- │       │   └── parameters.py
+ │       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
  │       ├── pkg_name.egg-info
  │       │   └── ...
- │       ├── build
- │       │   └── lib
- │       │       └── pkg_name
- │       │           ├── __init__.py
- │       │           └── script.py
+ │       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
+ │       ├── resource
+ │       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
+ │       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
+ │       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
+ │       └── share
+ │           └── pkg_name
+ │               └── hook
+ │                   ├── pythonpath_develop.dsv
+ │                   ├── pythonpath_develop.ps1
+ │                   └── pythonpath_develop.sh
  ├── install
  │   ├── COLCON_IGNORE
  │   ├── pkg_name
  │   │   ├── lib
  │   │   │   ├── pkg_name
  │   │   │   │   └── entry_point
  │   │   │   └── python3.10
  │   │   │       └── site-packages
- │   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
- │   │   │           │   └── ...
+ │   │   │           ├── pkg-name.egg-link
  │   │   │           └── pkg_name
- │   │   │               ├── __init__.py
- │   │   │               ├── __pycache__
- │   │   │               │   └── ...
- │   │   │               ├── script.py
  │   │   │               └── parameters.py
  │   │   └── share
  │   │       ├── ...
  │   │       └── colcon-core
  │   │           └── packages
  │   │               └── pkg_name
  │   └── ...
  └── src
      └── pkg_name
          ├── pkg_name
          │   ├── __init__.py
          │   ├── script.py
          │   └── parameters.yaml
          ├── package.xml
          ├── resource
          │   └── pkg_name
          ├── setup.cfg
          └── setup.py

[BrunoB81HK]

@pac48 is there any way you can share me the Dockerfile you used that worked for you?

[pac48]

@BrunoB81HK this is the image that I used. Its called osrf/ros:humble-desktop-full

[BrunoB81HK]

After much investigating, I found out that the probleme lied in my Dockerfile configuration. The build process was not saving any changes made inside the src/ directory. I'll close this issue since it is not a bug. Thanks a lot for your help!

[ahi-bplus]

@BrunoB81HK can you share the changes you made to fix the problem? I have exactly the same problem with --symlink-install

[BrunoB81HK]

@ahi-bplus Are you building the project inside your Dockerfile? If so, the problem for me was pretty simple.

When using --symlink-install, it creates a new module inside the src/ directory. The module is accessible because, if I remember correctly, the whole python module directory (the one who contains the __init__.py file) is symlinked, not just the files. This is not a problem in itself, but becomes one if you mount your src/ into the container. If you do so, the module inside the install/ directory changes to the one mounted in your src/ directory. For that reason, the parameter module is overwritten.

I can give you some ideas to help with your case:

1. Avoid using --symlink-install inside the Dockerfile.
2. Build your project inside the docker container, not the Dockerfile.
3. If used for development, start using devcontainers.
4. Find a way to avoid mounting your src/ directory inside the container.

I hope it helps, feel free to ask more questions if you need! :)

[ahi-bplus]

@BrunoB81HK Thanks for your quick reply. I already use a devcontainer for development. I mount my src files accordingly. As soon as I build my ROS node which implements generate_parameter_library using colcon build --symlink-install --merge-install, I get the problem with ModuleNotFoundError. Without --symlink-install --merge-install it works.

[ahi-bplus]

Okay, I have found the problem. It has nothing to do with --symlink-install or --merge-install. The problem comes from my package structure.

package_name/
├── README.md
├── __init__.py
├── config
│   └── default
│       └── default.yaml
├── models
├── package.xml
├── requirements.txt
├── resource
│   └── package_name
├── setup.cfg
├── setup.py
├── src
│   └── package_name
│       ├── __init__.py
│       ├── node.py
│       ├── parameters.yaml

Unfortunately the package_dir={'': 'src'}, of generate_parameter_library is not considered. If I avoid the subdirecory src/package_name everything works. I think this would be worth an issue?

[BrunoB81HK]

If you think this is a bug, you should create a new issue. Even if it's not, that will give others that have the same problem some help too.