Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various improvements to support generating recipes for Noetic #280

Merged
merged 13 commits into from
Dec 12, 2020

Commits on Dec 11, 2020

  1. yocto_recipe: use ROS_PYTHON_VERSION 2 only for melodic

    * noetic should use 3
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    4e54779 View commit details
    Browse the repository at this point in the history
  2. bitbake: sort rosdep_cache entries when dumping yaml file

    * when regenerating the recipes I often see change like this:
    
      suitesparse:
     -- suitesparse-cholmod@meta-ros
      - suitesparse-cxsparse@meta-ros
     +- suitesparse-cholmod@meta-ros
    
    * because the order of the list is not defined
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    f8fa12c View commit details
    Browse the repository at this point in the history
  3. yocto_recipe: add static method to get ROS_PYTHON_VERSION value

    * it will be used for creating context for conditional evaluations in rosdistro as well
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    095152e View commit details
    Browse the repository at this point in the history
  4. bitbake: use own condition_context instead of whole os.environ

    * Use only ROS_OS_OVERRIDE, ROS_DISTRO, ROS_PYTHON_VERSION, ROS_VERSION
      variables with ROS_PYTHON_VERSION, ROS_VERSION derived from distro value
      instead of depending on ros-generate-recipes.sh script from meta-ros
      to export correct value in shell before calling superflore
    
    * this is needed by DependencyWalker evaluate_condition_context
      parameter to correctly resolve the conditionals when evaluating
      dependencies and build_type
    
    * there are 3 variables used in currently generated package.xml files
      not covered by this context:
    
      "ros_version" (lowercase) in lgsvl-msgs (all ROS distributions)
      https://raw.githubusercontent.com/ros2-gbp/lgsvl_msgs-release/release/rolling/lgsvl_msgs/0.0.3-2/package.xml
    
      "BUILD_WITH_LDMRS_SUPPORT" in sick_scan2 (only in foxy)
    
      "IGNITION_VERSION" in ros_ign_image (only in noetic)
      https://raw.githubusercontent.com/ros-gbp/ros_ign-release/release/noetic/ros_ign_image/0.111.0-1/package.xml
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    ab47f70 View commit details
    Browse the repository at this point in the history
  5. bitbake: show list of available packages when pkg from --only isn't f…

    …ound
    
    * the package names aren't the same as generated recipes, so it's easy to make mistake
      and call e.g.
      sh scripts/ros-generate-recipes.sh noetic ros-core
      instead of
      sh scripts/ros-generate-recipes.sh noetic ros_core
    
    * the KeyError doesn't seem to be thrown from regenerate_pkg in
      this case as it raises RuntimeError, but extend the error message
      in both cases (as it's possible that KeyError might be raised
      in some other code-paths)
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    7bffbe9 View commit details
    Browse the repository at this point in the history
  6. bitbake: fix regenerating individual recipes with --only parameter

    * set preserve_existing to False in regenerate_pkg call
      otherwise it will consider old recipe to be up to date
      (even without comparing its version)
    * call add_generated_files before commit, otherwise it
      generates a commit which removes the old version, but
      the new version is left untracked
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    6bbe38f View commit details
    Browse the repository at this point in the history
  7. yocto_recipe.py: Fix setting sensible default for <build_type> when n…

    …ot ROS 1 (ros-infrastructure#267)
    
    * _get_ros_version() expects just a string (as in distro.name) not whole RosDistro class
      This is a bit confusing, because in many places we use the same distro variable
      as this string, rename it to make it more clear in the next commit.
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    cd48b9f View commit details
    Browse the repository at this point in the history
  8. yocto_recipe.py: Override the build_type from catkin_pkg in some cases (

    ros-infrastructure#267)
    
    * if get_build_type() returns 'catkin' and we're generating some ROS 2 distribution
      change it to 'ament_cmake' and show an error message
    
    * there are couple cases where <export><build_type> is missing completely
      and then catkin_pkg get_build_type() will return the default value as
      'catkin' as specified in
      https://www.ros.org/reps/rep-0149.html#build-type-multiple
      "If no <build_type> is provided, catkin is assumed."
      but this assumption doesn't work for ROS 2 distributions and in meta-ros
      we had to override these cases (as of 2020-12-05):
    
      dashing:
        https://raw.github.com/ros2-gbp/async_web_server_cpp-release/release/dashing/async_web_server_cpp/1.0.0-1/package.xml
        https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/dashing/fmilibrary_vendor/0.2.0-1/package.xml
        https://raw.github.com/ros-geographic-info/geographic_info-release/release/dashing/geographic_info/1.0.1-1/package.xml
        https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/dashing/gps_umd/1.0.4-1/package.xml
        https://raw.github.com/ros2-gbp/web_video_server-release/release/dashing/web_video_server/1.0.0-1/package.xml
        https://raw.github.com/KIT-MRT/mrt_cmake_modules-release/release/dashing/mrt_cmake_modules/1.0.8-1/package.xml
    
      eloquent:
        https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/eloquent/fmilibrary_vendor/0.2.0-1/package.xml
        https://raw.github.com/ros-geographic-info/geographic_info-release/release/eloquent/geographic_info/1.0.1-1/package.xml
        https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/eloquent/gps_umd/1.0.2-1/package.xml
    
      foxy:
        https://raw.github.com/boschresearch/fmilibrary_vendor-release/release/foxy/fmilibrary_vendor/0.2.0-1/package.xml
        https://raw.github.com/swri-robotics-gbp/gps_umd-release/release/foxy/gps_umd/1.0.4-1/package.xml
        https://raw.github.com/KIT-MRT/mrt_cmake_modules-release/release/foxy/mrt_cmake_modules/1.0.8-1/package.xml
    
      rolling:
        https://raw.github.com/ros2-gbp/fmilibrary_vendor-release/release/rolling/fmilibrary_vendor/0.2.0-2/package.xml
        https://raw.github.com/ros2-gbp/gps_umd-release/release/rolling/gps_umd/1.0.3-2/package.xml
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    f0d6ad1 View commit details
    Browse the repository at this point in the history
  9. bitbake: rename distro to rosdistro

    * e.g. _get_ros_version() expects just a string (as in distro.name) not whole RosDistro class
      this is a bit confusing, because in many places we use the same distro variable
      as this string, rename it to make it more clear
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    df12276 View commit details
    Browse the repository at this point in the history
  10. bitbake: make sure that the unresolved dependencies are valid OE reci…

    …pe names
    
    * with
      https://raw.githubusercontent.com/ros-gbp/moveit-release/release/noetic/moveit_kinematics/1.1.0-1/package.xml
      <depend condition="$ROS_DISTRO != noetic">orocos_kdl</depend>
      <depend condition="$ROS_DISTRO == noetic">liborocos-kdl-dev</depend>
      we have accidentally got orocos_kdl dependency (because ROS_DISTRO was
      missing in evaluate_condition_context)
    
      which results in dependency on ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl
    
    * but ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl"
      never got generated in generated/superflore-ros-distro.inc simply because
      https://github.com/ros-infrastructure/superflore/blob/649543e232fe1f44fcdf12c90ed5e0fc19bad4b8/superflore/generators/bitbake/yocto_recipe.py#L671
      takes last underscore-separated field as PN, so it generated
      ROS_UNRESOLVED_PLATFORM_PKG_kdl = "UNRESOLVED-kdl"
      instead which in the end caused unexpanded dependency issue like:
      ERROR: Nothing PROVIDES '${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl}' (but meta-ros/meta-ros1-noetic/generated-recipes/robot-calibration/robot-calibration_0.6.4-1.bb, meta-ros/meta-ros1-noetic/generated-recipes/moveit/moveit-kinematics_1.1.0-1.bb DEPENDS on or otherwise requires it)
    
    * similar issue is reproducible e.g. with jsk_pcl_ros:
      https://raw.githubusercontent.com/tork-a/jsk_recognition-release/release/noetic/jsk_pcl_ros/1.2.15-1/package.xml
      which has:
      <exec_depend>ml_classifiers</exec_depend>
      but there is no ml_classifiers package in noetic, it's only in melodic and dashing
      where it also generated:
      ROS_UNRESOLVED_PLATFORM_PKG_classifiers = "UNRESOLVED-classifiers"
      instead of:
      ROS_UNRESOLVED_PLATFORM_PKG_ml-classifiers = "UNRESOLVED-ml-classifiers"
    
      and similarly openni_launch which also doesn't exist in noetic, only in melodic
      ROS_UNRESOLVED_PLATFORM_PKG_launch = "UNRESOLVED-launch"
      ROS_UNRESOLVED_PLATFORM_PKG_openni-launch = "UNRESOLVED-openni-launch"
    
    * we could easily fix that by using whole "ROS_UNRESOLVED_PLATFORM_PKG_" prefix
      as a separator before PN, but there is still a risk of using underscore
      in bitbake variable names, because if some build happens to include
      one of these underscore separated strings (e.g. 'kdl') in OVERRIDES variable,
      then the ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl" assignment will
      be expanded as assignment to variable "ROS_UNRESOLVED_PLATFORM_PKG" with
      "kdl" and "orocos" as an overrides and ${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl}
      will still end undefined - using fewer underscores here is better
    
    * but there is still an option that the whole ROS package name will match with
      something in OVERRIDES (e.g. MACHINE name can be pretty much anything) and then
      these variables will fail the same, to fix it properly we should use uppercase
      for dependency name as well (while possibly changing the prefix to use dash as
      separators for easier readability), e.g.:
      ROS-UNRESOLVED-PLATFORM-PKG_OROCOS-KDL = "UNRESOLVED-OROCOS-KDL"
      but that would require migration of all manually overriden mappings:
      $ git grep ^ROS_UNRESOLVED_PLATFORM_PKG_ | grep '/ros-distro\.inc' | wc -l
      491
      which is easy, but would cause unwelcome diff between newer and older meta-ros
      branches. To mitigate this we plan to upstream all these manual overrides
      from meta-ros to rosdistro rosdep files and then we can re-evaluate this
      change to upper case when we don't use so many overrides
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    82dc39e View commit details
    Browse the repository at this point in the history
  11. bitbake: simplify unresolved dependencies handling

    * use DEPENDENCY instead of PLATFORM_PKG in UNRESOLVED_* variables
      in some cases as documented in previous commit the unresolved
      dependency might be ROS package which just happens to be missing
      in pkg_names, so it's considered as external/system/platform
      package
    
    * parse dependency name by skipping whole
      UNRESOLVED_DEPENDENCY_REF_PREFIX instead of relying on last
      underscore separated string
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    fecb498 View commit details
    Browse the repository at this point in the history
  12. bitbake: fix removing all generated files before re-creating them

    * without split() it was trying to remove all these files as one
      path in quotes and failing as shown after removing --ignore-unmatch
    
      git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
        cmdline: git rm -rf meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt
        stderr: 'fatal: pathspec 'meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt ' did not match any files'
    
      with split() it correctly complains only about superflore-change-summary.txt
      which will be fixed by returning --ignore-unmatch
    
      git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
        cmdline: git rm -rf meta-ros2-eloquent/generated-recipes meta-ros2-eloquent/conf/ros-distro/include/eloquent/generated meta-ros2-eloquent/files/eloquent/generated/newer-platform-components.list meta-ros2-eloquent/files/eloquent/generated/rosdep-resolve.yaml meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt
        stderr: 'fatal: pathspec 'meta-ros2-eloquent/files/eloquent/generated/superflore-change-summary.txt' did not match any files'
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    d367ded View commit details
    Browse the repository at this point in the history
  13. bitbake: change ROS_UNRESOLVED_PLATFORM_PKG_* to ROS_UNRESOLVED_DEP-*

    * this way the lowercase dependency name won't ever be parsed as an
      override
    * use the same prefix in the value (instead of 'UNRESOLVED-')
    
    * there was still a risk of using underscore
      in bitbake variable names, because if some build happens to include
      one of these underscore separated strings (e.g. 'orocos-kdl') in OVERRIDES variable,
      then the ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl = "UNRESOLVED-orocos_kdl" assignment will
      be expanded as assignment to variable "ROS_UNRESOLVED_PLATFORM_PKG" with
      "orocos-kdl" as an override and ${ROS_UNRESOLVED_PLATFORM_PKG_orocos_kdl}
      will still end undefined - using fewer underscores here is better
    
    * there was still an option that the whole ROS package name will match with
      something in OVERRIDES (e.g. MACHINE name can be pretty much anything) and then
      these variables will fail, to fix it properly we should either use uppercase
      for dependency name or change the prefix separator from underscore to dash.
    
    * this will require migration of all manually overriden mappings:
      $ git grep ^ROS_UNRESOLVED_PLATFORM_PKG_ | grep '/ros-distro\.inc' | wc -l
      491
      so I've updated rosdistro/rosdep to include most of them first and
      regenerated all ROS distributions with updated rosdistro.
    
    Signed-off-by: Martin Jansa <martin.jansa@lge.com>
    shr-project committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    bbe1d07 View commit details
    Browse the repository at this point in the history