-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rosdep: update and add new OpenEmbedded mappings #27623
Conversation
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
… assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…ignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-12 as of 20201012125146 * include rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…5146 Regenerate with superflore from: ros-infrastructure/superflore#280 and rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution melodic. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro melodic --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-27 as of 20201028183600 * include rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…3600 Regenerate with superflore from: ros-infrastructure/superflore#280 and rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution dashing. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro dashing --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
… assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…20-08-13 as of 20200811171906 * include rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…171906 Regenerate with superflore from: ros-infrastructure/superflore#280 and rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution eloquent. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro eloquent --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…ignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
… as of 20201105055614 * include rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
Regenerate with superflore from: ros-infrastructure/superflore#280 and rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution foxy. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro foxy --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…assignments * with rosdistro/rosdep updated as in: ros/rosdistro#27623 we don't need most of these assignments anymore Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-12 as of 20201012185327 * include rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…5327 Regenerate with superflore from: ros-infrastructure/superflore#280 and rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution rolling. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro rolling --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-12 as of 20201012125146 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…5146 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution melodic. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro melodic --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-27 as of 20201028183600 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…3600 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution dashing. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro dashing --output-repository-path . --upstream-branch HEAD
…20-08-13 as of 20200811171906 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-12 as of 20201012125146 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…5146 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution melodic. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro melodic --output-repository-path . --upstream-branch HEAD Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…-10-27 as of 20201028183600 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…3600 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution dashing. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro dashing --output-repository-path . --upstream-branch HEAD
…20-08-13 as of 20200811171906 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…171906 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution eloquent. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro eloquent --output-repository-path . --upstream-branch HEAD
… as of 20201105055614 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution foxy. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro foxy --output-repository-path . --upstream-branch HEAD
…-10-12 as of 20201012185327 * include more rosdep changes from: ros/rosdistro#27623 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…5327 Regenerate with superflore from: ros-infrastructure/superflore#280 and more rosdistro/rosdep changes from: ros/rosdistro#27623 Recipes generated by superflore for all packages in ROS distribution rolling. This pull request was generated by running the following command: superflore-gen-oe-recipes --dry-run --no-branch --ros-distro rolling --output-repository-path . --upstream-branch HEAD
I'm not sure how to review this PR. @tfoote how do you tell what packages exist for openembedded? |
You can look at |
@shr-project I made a script to check the rules using that website's API, but there may be a mistake in the logic because it wasn't able to find quite a few rules. Any ideas what's happening here? (I only checked Python script to check openembedded rules#!/usr/bin/env python3
import argparse
import requests
import sys
import yaml
class OpenEmbeddedLayerIndex:
API_LAYERS = "http://layers.openembedded.org/layerindex/api/layers/"
API_LAYER_BRANCHES = \
"http://layers.openembedded.org/layerindex/api/layerBranches/"
API_RECIPES = "http://layers.openembedded.org/layerindex/api/recipes/"
# API_LAYERS = "/tmp/openembedded/layers.json"
# API_LAYER_BRANCHES = "/tmp/openembedded/layerBranches.json"
# API_RECIPES = "/tmp/openembedded/recipes.json"
def __init__(self):
self.refresh()
def refresh(self):
self._recipes = []
self._layer_branches = {}
self._layers = {}
self._recipes = self._jsonify(self.API_RECIPES)
for layer_branch in self._jsonify(self.API_LAYER_BRANCHES):
self._layer_branches[layer_branch['id']] = layer_branch
for layer in self._jsonify(self.API_LAYERS):
self._layers[layer['id']] = layer
def _jsonify(self, url):
resp = requests.get(url)
if 200 != resp.status_code:
raise RuntimeError(f'Failed to fetch {url}')
return resp.json()
# import json
# with open(url, 'r') as fin:
# return json.loads(fin.read())
def exists(self, keyname):
"""Given key like autoconf@openembedded returns true if recipe exists in that layer."""
pn, layer_name = keyname.split('@')
for recipe in self._recipes:
if pn == recipe['pn']:
try:
layer_branch = self._layer_branches[recipe['layerbranch']]
layer = self._layers[layer_branch['layer']]
except KeyError:
continue
if layer_name == layer['layer']['name']:
return True
return False
def main():
parser = argparse.ArgumentParser(description='Check OpenEmbedded Rules.')
parser.add_argument('rosdep_yaml', metavar='ROSDEP_YAML',
help='Path to rosdep yaml database')
args = parser.parse_args()
with open(args.rosdep_yaml, 'r') as fin:
rosdep_rules = yaml.safe_load(fin.read())
print('Getting OpenEmbedded Layer Index')
layer_index = OpenEmbeddedLayerIndex()
print('Checking rosdep rules')
for key in rosdep_rules:
rule = rosdep_rules[key]
if 'openembedded' in rule:
for recipe in rule['openembedded']:
if layer_index.exists(recipe):
print(f'{key} uses valid recipe: {recipe}')
else:
sys.stderr.write(f'ERROR: {key} uses invalid recipe: {recipe}\n')
if __name__ == '__main__':
main() Looking at
And there are much more missing with this PR
|
Not sure what the root cause is, but the @layer-name isn't much important, because it's thrown away as soon as it's resolved by rosdep: And more importantly the recipe is sometimes different for different OE releases (while all releases are using the same rosdep data) and only bitbake will decide which one to use based on more inputs than what could be practically expressed in rosdep data. After extending your script to list all layers where it found some recipe before showing the ERROR message, I see e.g.:
but that doesn't really match with what I see in browser: not sure why the API call for recipes is missing this one from meta-ros-common: Similarly the script shows: Another source of false-positives in the script could be package names which could be mapped to OE packages, not recipes and if they are used only as a runtime dependencies, then it would be OK, but layerindex only resolves the recipe names (1 oe recipe produces number of packages). |
After fixing the script, not to trigger KeyError I see 4 errors for base.yaml: ERROR: dnsmasq uses invalid recipe: dnsmasq@openembedded-core The first one has the layer incorrectly, it should be meta-networking: The liborocos* are complaining correctly, but only because it's added to meta-ros1-noetic by: ERROR: linux-headers-generic uses invalid recipe: virtual/kernel@openembedded-core is partially valid, as it should be rather linux-libc-headers (like I've added for linux-kernel-headers dependency), but this line wasn't changed by me and layerindex doesn't know how to resolve virtua/kernel (as it's usually set by PREFERRED_PROVIDE in the BSP layer of given MACHINE). testing python.yaml shows a lot of errors like: but most of them is because it doesn't expand ${PYTHON_PN} before qwery to layerindex, python3-pykdl is the same case as orocos-kdl* above and python3-tkinter is the package name created by python3 recipe in oe-core, so the recipe != package cases mentioned in previous comment. |
1afa106
to
6bd89b8
Compare
* meta-ros uses separate sublayers since layer version 3: https://github.com/ros/meta-ros/wiki/Superflore-OE-Recipe-Generation-Scheme#layer-version-3-ros_superflore_generation_scheme-2 replace the generic @meta-ros with the correct sublayer where the dependency exists, in most cases it will be meta-ros-comon but festival, urdfdom, urdfdom-headers are in meta-ros1 layer and lapack is now in meta-oe instead of meta-ros and wxpython in meta-ros-python2 Signed-off-by: Martin Jansa <martin.jansa@lge.com>
* based on the ROS_UNRESOLVED_PLATFORM_PKG_* overrides we currently use in meta-ros Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…kages Signed-off-by: Martin Jansa <martin.jansa@lge.com>
* the recipes were added to meta-ros-common (for python3) and meta-ros-python2 (for python2) Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…aders-generic OE mapping Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…release * add llvm, llvm-dev, libopenblas-dev introduced in dashing 2020-12-02 release Signed-off-by: Martin Jansa <martin.jansa@lge.com>
…release * add llvm-7, llvm-dev-7, libqt5-websockets-dev, python-click introduced in melodic 2020-11-11 or 2020-12-07 release * there isn't separate recipe for llvm7 and e.g. Dunfell release will build 9.0.1 version by default but that could be changed by bbappend in melodic, so the mapping still needs to be llvm recipe from oe-core Signed-off-by: Martin Jansa <martin.jansa@lge.com>
85667f5
to
4a7a145
Compare
I'm a bit wary about this May I ask you why the detour @shr-project ? |
It's because meta-ros is trying to support switching from python2 to python3 even for ROS distributions which were released with support for only python2 (like melodic) based on ROS_PYTHON_VERSION variable in OE build and for all other ROS distributions it blacklists all python2 recipes by default (and doesn't even include meta-python2 layer which provides most of these python-* dependencies). So in these cases PYTHON_PN would be set to python3 to get the available provide even when it might need more work in runtime. OE is different from other distributions from rosdep, because everything is custom (as OE is a tool to build your own custom distribution) - and that's difficult to express in rosdep data. It wasn't introduced in this PR, but in older one with @herb-kuta-lge 's commit: |
No description provided.