rosdep: update and add new OpenEmbedded mappings

[shr-project]

No description provided.

[sloretz]

I'm not sure how to review this PR. @tfoote how do you tell what packages exist for openembedded?

[shr-project]

I'm not sure how to review this PR. @tfoote how do you tell what packages exist for openembedded?

You can look at
https://layers.openembedded.org/layerindex/branch/master/recipes/
but as meta-ros is the only user of this mapping and most of this PR was based on the setting I already did there I think you don't need to verify all of them :).

[sloretz]

You can look at
https://layers.openembedded.org/layerindex/branch/master/recipes/
but as meta-ros is the only user of this mapping and most of this PR was based on the setting I already did there I think you don't need to verify all of them :).

@shr-project I made a script to check the rules using that website's API, but there may be a mistake in the logic because it wasn't able to find quite a few rules. Any ideas what's happening here? (I only checked base.yaml)

Python script to check openembedded rules

#!/usr/bin/env python3

import argparse
import requests
import sys
import yaml


class OpenEmbeddedLayerIndex:

    API_LAYERS = "http://layers.openembedded.org/layerindex/api/layers/"
    API_LAYER_BRANCHES = \
        "http://layers.openembedded.org/layerindex/api/layerBranches/"
    API_RECIPES = "http://layers.openembedded.org/layerindex/api/recipes/"

    # API_LAYERS = "/tmp/openembedded/layers.json"
    # API_LAYER_BRANCHES = "/tmp/openembedded/layerBranches.json"
    # API_RECIPES = "/tmp/openembedded/recipes.json"

    def __init__(self):
        self.refresh()

    def refresh(self):
        self._recipes = []
        self._layer_branches = {}
        self._layers = {}

        self._recipes = self._jsonify(self.API_RECIPES)

        for layer_branch in self._jsonify(self.API_LAYER_BRANCHES):
            self._layer_branches[layer_branch['id']] = layer_branch

        for layer in self._jsonify(self.API_LAYERS):
            self._layers[layer['id']] = layer

    def _jsonify(self, url):
        resp = requests.get(url)
        if 200 != resp.status_code:
            raise RuntimeError(f'Failed to fetch {url}')
        return resp.json()
        
        # import json
        # with open(url, 'r') as fin:
        #     return json.loads(fin.read())

    def exists(self, keyname):
        """Given key like autoconf@openembedded returns true if recipe exists in that layer."""
        pn, layer_name = keyname.split('@')
        for recipe in self._recipes:
            if pn == recipe['pn']:
                try:
                    layer_branch = self._layer_branches[recipe['layerbranch']]
                    layer = self._layers[layer_branch['layer']]
                except KeyError:
                    continue
                if layer_name == layer['layer']['name']:
                    return True
        return False


def main():
    parser = argparse.ArgumentParser(description='Check OpenEmbedded Rules.')
    parser.add_argument('rosdep_yaml', metavar='ROSDEP_YAML',
                        help='Path to rosdep yaml database')
    args = parser.parse_args()

    with open(args.rosdep_yaml, 'r') as fin:
        rosdep_rules = yaml.safe_load(fin.read())

    print('Getting OpenEmbedded Layer Index')
    layer_index = OpenEmbeddedLayerIndex()

    print('Checking rosdep rules')
    for key in rosdep_rules:
        rule = rosdep_rules[key]
        if 'openembedded' in rule:
            for recipe in rule['openembedded']:
                if layer_index.exists(recipe):
                    print(f'{key} uses valid recipe: {recipe}')
                else:
                    sys.stderr.write(f'ERROR: {key} uses invalid recipe: {recipe}\n')


if __name__ == '__main__':
    main()

Looking at master these are the errors I see:

ERROR: bullet uses invalid recipe: bullet@meta-ros
ERROR: cppcheck uses invalid recipe: cppcheck@meta-ros
ERROR: graphviz uses invalid recipe: graphviz@meta-ros
ERROR: joystick uses invalid recipe: joystick@meta-ros
ERROR: libblas-dev uses invalid recipe: openblas@meta-ros
ERROR: libglfw3-dev uses invalid recipe: glfw@meta-intel-realsense
ERROR: liblapack-dev uses invalid recipe: lapack@meta-ros
ERROR: linux-headers-generic uses invalid recipe: virtual/kernel@openembedded-core
ERROR: suitesparse uses invalid recipe: suitesparse-cxsparse@meta-ros
ERROR: suitesparse uses invalid recipe: suitesparse-cholmod@meta-ros

And there are much more missing with this PR

ERROR: benchmark uses invalid recipe: google-benchmark@meta-ros2
ERROR: bullet uses invalid recipe: bullet@meta-ros-common
ERROR: collada-dom uses invalid recipe: collada-dom@meta-ros-common
ERROR: cppcheck uses invalid recipe: cppcheck@meta-ros-common
ERROR: festival uses invalid recipe: festival@meta-ros1
ERROR: geographiclib uses invalid recipe: geographiclib@meta-ros-common
ERROR: geographiclib-tools uses invalid recipe: geographiclib@meta-ros-common
ERROR: graphicsmagick uses invalid recipe: graphicsmagick@meta-ros2
ERROR: graphviz uses invalid recipe: graphviz@meta-ros-common
ERROR: joystick uses invalid recipe: joystick@meta-ros-common
ERROR: libblas-dev uses invalid recipe: openblas@meta-ros-common
ERROR: libccd-dev uses invalid recipe: libccd@meta-ros-common
ERROR: libconsole-bridge-dev uses invalid recipe: console-bridge@meta-ros-common
ERROR: libfcl-dev uses invalid recipe: fcl@meta-ros-common
ERROR: libflann uses invalid recipe: libflann@meta-ros-common
ERROR: libfreenect-dev uses invalid recipe: libfreenect@meta-ros-common
ERROR: libglfw3-dev uses invalid recipe: glfw@meta-intel-realsense
ERROR: libogre uses invalid recipe: ogre@meta-ros-common
ERROR: libogre-dev uses invalid recipe: ogre@meta-ros-common
ERROR: liborocos-kdl uses invalid recipe: orocos-kdl@meta-ros1-noetic
ERROR: liborocos-kdl-dev uses invalid recipe: orocos-kdl@meta-ros1-noetic
ERROR: libpcl-all uses invalid recipe: pcl@meta-ros-common
ERROR: libpcl-all-dev uses invalid recipe: pcl@meta-ros-common
ERROR: libqhull uses invalid recipe: qhull@meta-ros-common
ERROR: liburdfdom-dev uses invalid recipe: urdfdom@meta-ros1
ERROR: liburdfdom-headers-dev uses invalid recipe: urdfdom-headers@meta-ros1
ERROR: liburdfdom-tools uses invalid recipe: urdfdom@meta-ros1
ERROR: linux-headers-generic uses invalid recipe: virtual/kernel@openembedded-core
ERROR: log4cxx uses invalid recipe: log4cxx@meta-ros-common
ERROR: suitesparse uses invalid recipe: suitesparse-cxsparse@meta-ros-common
ERROR: suitesparse uses invalid recipe: suitesparse-cholmod@meta-ros-common
ERROR: wx-common uses invalid recipe: wxwidgets@meta-ros-common
ERROR: wxwidgets uses invalid recipe: wxwidgets@meta-ros-common
ERROR: yaml-cpp uses invalid recipe: yaml-cpp@meta-ros-common

[shr-project]

Not sure what the root cause is, but the @layer-name isn't much important, because it's thrown away as soon as it's resolved by rosdep:
https://github.com/ros-infrastructure/superflore/blob/master/superflore/generators/bitbake/yocto_recipe.py#L313

And more importantly the recipe is sometimes different for different OE releases (while all releases are using the same rosdep data) and only bitbake will decide which one to use based on more inputs than what could be practically expressed in rosdep data.

After extending your script to list all layers where it found some recipe before showing the ERROR message, I see e.g.:

WARN: recipe wxwidgets found in different layer openembedded-core instead of meta-ros-common
WARN: recipe wxwidgets found in different layer openembedded-core instead of meta-ros-common
WARN: recipe wxwidgets found in different layer meta-ros instead of meta-ros-common
WARN: recipe wxwidgets found in different layer meta-oe instead of meta-ros-common
WARN: recipe wxwidgets found in different layer meta-oe instead of meta-ros-common
WARN: recipe wxwidgets found in different layer meta-oe instead of meta-ros-common
ERROR: wxwidgets uses invalid recipe: wxwidgets@meta-ros-common

but that doesn't really match with what I see in browser:
https://layers.openembedded.org/layerindex/branch/master/recipes/?q=wxwidgets

not sure why the API call for recipes is missing this one from meta-ros-common:
https://layers.openembedded.org/layerindex/recipe/133800/
but you can see it existed in meta-ros-common since warrior branch (and even before that in 2.7 Thud release as well as shown in https://github.com/ros/meta-ros/blob/thud/meta-ros-common/recipes-extended/wxwidgets/wxwidgets_3.0.4.bb)

Similarly the script shows:
ERROR: suitesparse uses invalid recipe: suitesparse-cxsparse@meta-ros-common
ERROR: suitesparse uses invalid recipe: suitesparse-cholmod@meta-ros-common
but
http://layers.openembedded.org/layerindex/recipe/133795/
http://layers.openembedded.org/layerindex/recipe/133796/
both show meta-ros-common.

Another source of false-positives in the script could be package names which could be mapped to OE packages, not recipes and if they are used only as a runtime dependencies, then it would be OK, but layerindex only resolves the recipe names (1 oe recipe produces number of packages).

[shr-project]

After fixing the script, not to trigger KeyError
-layer = self._layers[layer_branch['layer']]
+layer = self._layers[layer_branch['id']]

I see 4 errors for base.yaml:

ERROR: dnsmasq uses invalid recipe: dnsmasq@openembedded-core
ERROR: liborocos-kdl uses invalid recipe: orocos-kdl@meta-ros1-noetic
ERROR: liborocos-kdl-dev uses invalid recipe: orocos-kdl@meta-ros1-noetic
ERROR: linux-headers-generic uses invalid recipe: virtual/kernel@openembedded-core

The first one has the layer incorrectly, it should be meta-networking:
https://layers.openembedded.org/layerindex/recipe/124690/

The liborocos* are complaining correctly, but only because it's added to meta-ros1-noetic by:
ros/meta-ros@dd044bf
which isn't even merged in meta-ros/master yet, so layerindex had no chance to parse this recipe yet, but on the other hand I need to add this mapping before generating noetic recipes in superflore, so it's correct even when layerindex doesn't see it yet

ERROR: linux-headers-generic uses invalid recipe: virtual/kernel@openembedded-core

is partially valid, as it should be rather linux-libc-headers (like I've added for linux-kernel-headers dependency), but this line wasn't changed by me and layerindex doesn't know how to resolve virtua/kernel (as it's usually set by PREFERRED_PROVIDE in the BSP layer of given MACHINE).

testing python.yaml shows a lot of errors like:
ERROR: python-yaml uses invalid recipe: ${PYTHON_PN}-pyyaml@meta-python
ERROR: python3-pykdl uses invalid recipe: python3-pykdl@meta-ros1-noetic
ERROR: python3-pyqt5.qtwebengine uses invalid recipe: ${PYTHON_PN}-pyqt5@meta-qt5
ERROR: python3-tk uses invalid recipe: python3-tkinter@openembedded-core

but most of them is because it doesn't expand ${PYTHON_PN} before qwery to layerindex, python3-pykdl is the same case as orocos-kdl* above and python3-tkinter is the package name created by python3 recipe in oe-core, so the recipe != package cases mentioned in previous comment.

[hidmic]

I'm a bit wary about this PYTHON_PN envvar in rosdep rules. It breaks the correspondence between rosdep python rules and python version. It presumes that a downstream package can use either interchangeably. If OpenEmbedded does not provide a python2 version of a given package until a given version/distro/release, then it should only define a python- rule for the version/distro/release that does, like every other platform.

May I ask you why the detour @shr-project ?

[shr-project]

It's because meta-ros is trying to support switching from python2 to python3 even for ROS distributions which were released with support for only python2 (like melodic) based on ROS_PYTHON_VERSION variable in OE build and for all other ROS distributions it blacklists all python2 recipes by default (and doesn't even include meta-python2 layer which provides most of these python-* dependencies). So in these cases PYTHON_PN would be set to python3 to get the available provide even when it might need more work in runtime.

OE is different from other distributions from rosdep, because everything is custom (as OE is a tool to build your own custom distribution) - and that's difficult to express in rosdep data.

It wasn't introduced in this PR, but in older one with @herb-kuta-lge 's commit:
5463992