Building ros2 dashing from source error on TinkerOS armv7l due to undefined os

[DentOpt]

Bug report

ros2 is not building as os is not defined on tinkeros.

Required Info:

• Operating System:
  Linux tinkerboard 4.4.132+ Access control #1 SMP Wed Aug 21 19:15:55 CST 2019 armv7l GNU/Linux
• Installation type:
  From source source (https://raw.githubusercontent.com/ros2/ros2/dashing/ros2.repos)
• Version or commit hash:
• DDS implementation:
  • Default
• Client library (if applicable):
  • turtlesim, sros2

Steps to reproduce issue

Standard Build from source process according to
https://index.ros.org/doc/ros2/Installation/Dashing/Linux-Development-Setup/

Expected behavior

compilation succesfull

Actual behavior

sros2 is not compiling as "os" is not defined

... 

Starting >>> turtlesim
Traceback (most recent call last): ongoing] [turtlesim - 3.3s] [ros2msg - 4.2s] [sros2 - 3.6s] [launch_testing_ros - 10.6s]
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/distutils/core.py", line 217, in run_setup
    exec(f.read(), g, l)
  File "<string>", line 17, in <module>
  File "<string>", line 10, in package_files
NameError: name 'os' is not defined
--- output: sros2                                                                                                          
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/usr/lib/python3/dist-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/usr/lib/python3/dist-packages/colcon_ros/package_identification/ros.py", line 130, in getter
    desc.path / 'setup.py', env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 265, in get_setup_information
    setup_py, env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 303, in _get_setup_information
    cwd=os.path.abspath(str(setup_py.parent)), check=True, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1
---
--- stderr: sros2
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/usr/lib/python3/dist-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/usr/lib/python3/dist-packages/colcon_ros/package_identification/ros.py", line 130, in getter
    desc.path / 'setup.py', env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 265, in get_setup_information
    setup_py, env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 303, in _get_setup_information
    cwd=os.path.abspath(str(setup_py.parent)), check=True, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1
---
Failed   <<< sros2	[ Exited with code 1 ]
Aborted  <<< turtlesim
Aborted  <<< launch_testing_ros
Aborted  <<< ros2msg

Summary: 197 packages finished [2min 46s]
  1 package failed: sros2
  3 packages aborted: launch_testing_ros ros2msg turtlesim
  1 package had stderr output: sros2
  70 packages not processed
Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1

[kyrofa]

Traceback (most recent call last): ongoing] [turtlesim - 3.3s] [ros2msg - 4.2s] [sros2 - 3.6s] [launch_testing_ros - 10.6s]
 File "<string>", line 1, in <module>
 File "/usr/lib/python3.5/distutils/core.py", line 217, in run_setup
   exec(f.read(), g, l)
 File "<string>", line 17, in <module>
 File "<string>", line 10, in package_files
NameError: name 'os' is not defined

I believe that function is being called by colcon given the output:

File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 303, in _get_setup_information
   cwd=os.path.abspath(str(setup_py.parent)), check=True, env=env)
 File "/usr/lib/python3.5/subprocess.py", line 398, in run
   output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py' [...]

Which leads me to believe it's choking on this line. The os module is imported right above, though. How confident are you in your python install?

[mikaelarguedas]

it looks like the "NameError: name 'os' is not defined" is outside of the sros2 blocks and is likely not an sros2 related issue.

Could you please run the following and give the console output:

rm -rf build/sros2 install/sros2
colcon build --symlink-install --packages-select sros2 --event-handlers console_direct+

Could you provide more details about your setup ? Was python provided in your base install or did you install it yourself ?

Also maybe of relevance:

• ROS 2 Dashing lost Python 3.5 support Python3.5 support lost in Dashing release ros2#735 and doesnt have an active check (that I know of) ensuring support is still there as all officially supported platforms have more recent versions.
• There is also no active checks for Debian or arm32 for that distro.

This is just to highlight this is uncharted territory in multiple dimensions

[kyrofa]

This is just to highlight this is uncharted territory in multiple dimensions

Agreed.

[DentOpt]

rm -rf build/sros2 install/sros2
colcon build --symlink-install --packages-select sros2 --event-handlers console_direct+

Could you provide more details about your setup ? Was python provided in your base install or did you install it yourself ?

Also maybe of relevance:

• ROS 2 Dashing lost Python 3.5 support ros2/ros2#735 and doesnt have an active check (that I know of) ensuring support is still there as all officially supported platforms have more recent versions.
• There is also no active checks for Debian or arm32 for that distro.

This is just to highlight this is uncharted territory in multiple dimensions

linaro@tinkerboard:~/ros2_dashing$ rm -rf build/sros2 install/sros2
linaro@tinkerboard:~/ros2_dashing$ colcon build --symlink-install --packages-select sros2 --event-handlers console_direct+

Starting >>> sros2   
Traceback (most recent call last):s]
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/distutils/core.py", line 216, in run_setup
    exec(f.read(), g, l)
  File "<string>", line 17, in <module>
  File "<string>", line 10, in package_files
NameError: name 'os' is not defined
Traceback (most recent call last):  
  File "/usr/lib/python3/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/usr/lib/python3/dist-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/usr/lib/python3/dist-packages/colcon_ros/package_identification/ros.py", line 130, in getter
    desc.path / 'setup.py', env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 265, in get_setup_information
    setup_py, env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 303, in _get_setup_information
    cwd=os.path.abspath(str(setup_py.parent)), check=True, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1
--- stderr: sros2
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/usr/lib/python3/dist-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/usr/lib/python3/dist-packages/colcon_ros/package_identification/ros.py", line 130, in getter
    desc.path / 'setup.py', env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 265, in get_setup_information
    setup_py, env=env)
  File "/usr/lib/python3/dist-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 303, in _get_setup_information
    cwd=os.path.abspath(str(setup_py.parent)), check=True, env=env)
  File "/usr/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1
---
Failed   <<< sros2	[ Exited with code 1 ]
                                
Summary: 0 packages finished [6.07s]
  1 package failed: sros2
  1 package had stderr output: sros2
Command '['/usr/bin/python3', '-c', "import sys;from distutils.core import run_setup;dist = run_setup(    'setup.py', script_args=('--dry-run',), stop_after='config');skip_keys = ('cmdclass', 'distclass', 'ext_modules', 'metadata');data = {    key: value for key, value in dist.__dict__.items()     if (        not key.startswith('_') and         not callable(value) and         key not in skip_keys and         key not in dist.display_option_names    )};data['metadata'] = {    k: v for k, v in dist.metadata.__dict__.items()     if k not in ('license_files', 'provides_extras')};sys.stdout.buffer.write(repr(data).encode('utf-8'))"]' returned non-zero exit status 1

Good point, I'll try to update to python 3.8 and keep you updated.

[gdh1995]

Hello, my ROS2 crystal on python3.5 also reported this crash. And I find it seems a bug of Lib/distutils/core.py:216 - it calls exec(f.read(), g, l) and g and l are different dicts, which is not expected when running a "file input".

As said in https://docs.python.org/3/library/functions.html#exec :

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

My python 3.7.5 on Windows has fixed this by using exec(f.read(), g) instead.

I believe this is the root cause and I've run sudo vim core.py to edit it manually. Now I'm running colcon build again. I'll put its results here when it finishes (or crashes again).

extra info

Just now I downloaded source archives of Python 3.5.10, 3.6.0 and 3.6.14, and I found this bug of Lib/disutils/core.py exists on v3.5.10 (2020-09-05) but has been fixed since v3.6.0 (2016-12).

[zymouse]

使用 colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release编译
我也遇到了同样的问题
最后我发现，是自己粗心。不是在【工作空间】下编译导致