Importing package from other owners or organization

[tomkimsour]

Description

The github action doesn't seem to be able to retrive the repository for my organization or external organization.

Expected Behavior

The action is suppose to pull all the repository necessary for compiling this package and run compilation

Actual Behavior

After the action-ros-ci, it succeeds to retriving the repository the action is started on and fails pulling repo from the organization or public repo from other person/organization

Invoking: bash -c,./install_rosdeps.sh noetic
  /usr/bin/bash -c ./install_rosdeps.sh noetic
  + '[' 1 '!=' 1 ']'
  + DISTRO=noetic
  ++ colcon list --paths-only --packages-up-to manager_pepper perception_pepper navigation_pepper manipulation_pepper dialog_pepper vizbox tablet_pepper ros-naoqi/libqicore ros-naoqi/libqi ros-naoqi/naoqi_bridge_msgs
  Package 'perception_pepper' specified with --packages-up-to was not found
  Package 'navigation_pepper' specified with --packages-up-to was not found
  Package 'manipulation_pepper' specified with --packages-up-to was not found
  Package 'dialog_pepper' specified with --packages-up-to was not found
  Package 'ros-naoqi/libqicore' specified with --packages-up-to was not found

My yaml file :

# This is a basic workflow to help you get started with Actions

name: Build ros pepper manager

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the "main" branch
  push:
    branches: [ "main","devel" ]
  pull_request:
    branches: [ "main" ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
#       Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3
        with:
          ref: main
          token: ${{ secrets.DEVEL_MANAGER }}
      - uses: ros-tooling/setup-ros@v0.3
        with:
          required-ros-distributions: noetic
      - name: Set up git to see all pull requests
        run: git config --global --add remote.origin.fetch '+refs/pull/*:refs/remotes/origin/pull/*'
      - uses: ros-tooling/action-ros-ci@v0.2
        with:
          package-name: |
            manager_pepper
            perception_pepper
            navigation_pepper
            manipulation_pepper
            dialog_pepper
            vizbox
            tablet_pepper
            ros-naoqi/libqicore
            ros-naoqi/libqi
            ros-naoqi/naoqi_bridge_msgs
          import-token: ${{ secrets.DEVEL_MANAGER }}
          target-ros1-distro: noetic
          skip-tests: true
          
      

To Reproduce

1. Create an organization with 2 private repositories that represent a ros package each
2. Set up this yaml file as an action :

# This is a basic workflow to help you get started with Actions

name: Build ros pepper manager

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the "main" branch
  push:
    branches: [ "main","devel" ]
  pull_request:
    branches: [ "main" ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      - uses: ros-tooling/setup-ros@v0.3
        with:
          required-ros-distributions: noetic
      - name: Set up git to see all pull requests
        run: git config --global --add remote.origin.fetch '+refs/pull/*:refs/remotes/origin/pull/*'
      - uses: ros-tooling/action-ros-ci@v0.2
        with:
          package-name: |
            repo1
            repo2
            User/publicRepo
          import-token: ${{ secrets.DEVEL_ }}
          target-ros1-distro: noetic
          skip-tests: true
          
      

4. Run the action by hand

System

• OS: 20.04
• ROS 1 Distro: Noetic

[christophebedard]

I think you're missing something. Are your private repos separate from the repo against which the action is running? If so, I don't see where they are getting imported: you're just giving the package names to action-ros-ci, but you're not telling it how to import the packages (i.e., where to find them).

There are two use-cases for private repositories:

1. The repository that the action is running against is private
2. Other/external privates repositories are needed (this includes potentially use-case 1)

See: https://github.com/ros-tooling/action-ros-ci#Use-with-private-repos

For use-case 1, you simply need to use GITHUB_TOKEN. The repo against which the action is running will be imported by action-ros-ci, so you don't need to use actions/checkout. Example:

  - uses: ros-tooling/action-ros-ci@v0.2
    with:
      package-name: ...
      # If there are no private dependencies, no need to create a PAT or add a secret
      import-token: ${{ secrets.GITHUB_TOKEN }}

For use-case 2, you need to tell action-ros-ci which (external) repos it needs to import. You can do this by creating a .repos file in your main repo which includes those external repos (see this example or the documentation here: https://github.com/dirk-thomas/vcstool/). Then you use actions/checkout to make sure the file is available to action-ros-ci, and you provide its path using the vcs-repo-file-url input. See: https://github.com/ros-tooling/action-ros-ci#build-with-a-custom-repos-or-rosinstall-file. Since the .repos file is in the repo itself, take a look at this example:

action-ros-ci/.github/workflows/test.yml

Line 74 in e0eeca0

vcs-repo-file-url: "${{ github.workspace }}/.github/workflows/ros_tutorials.repos"

You also need to create a personal access token (PAT) with read permission for your private repos (GITHUB_TOKEN only has access to the repo for which the action is running) and add it to your repository as a secret. I think you need to create a PAT at the organization level if the repo is under an organization. You then have something like this:

  - uses: actions/checkout@v3
  # ...
  - uses: ros-tooling/action-ros-ci@v0.2
    with:
      package-name: ...
      vcs-repo-file-url: "${{ github.workspace }}/relative/path/to/.repos"
      # If there are private dependencies (e.g., in a file provided through vcs-repo-file-url), a PAT is required
      import-token: ${{ secrets.REPO_TOKEN }}

The documentation should be updated to give a complete example for this last use-case.

[peredwardsson]

I would rather use a deploy key, which is bound to a repo, over a PAT, which is bound to my account. Can I supply more than one import-token somehow?

Another alternative is to use the actions/checkout once per package to clone that dependency, and place stuff into a src folder. Is that more advisable?

[christophebedard]

I would rather use a deploy key, which is bound to a repo, over a PAT, which is bound to my account. Can I supply more than one import-token somehow?

Unfortunately not :/ at least currently. The provided token is used globally, and only one token is expected:

action-ros-ci/src/action-ros-ci.ts

Lines 436 to 466 in e0eeca0

	if (importToken !== "") {
	// Unset all local extraheader config entries possibly set by actions/checkout,
	// because local settings take precedence and the default token used by
	// actions/checkout might not have the right permissions for any/all repos
	await execShellCommand(
	[
	`/usr/bin/git config --local --unset-all http.https://github.com/.extraheader || true`,
	],
	options
	);
	await execShellCommand(
	[
	String.raw`/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader'` +
	` && git config --local --unset-all 'http.https://github.com/.extraheader' || true`,
	],
	options
	);
	// Use a global insteadof entry because local configs aren't observed by git clone
	await execShellCommand(
	[
	`/usr/bin/git config --global url.https://x-access-token:${importToken}@github.com.insteadof 'https://github.com'`,
	],
	options
	);
	if (core.isDebug()) {
	await execShellCommand(
	[`/usr/bin/git config --list --show-origin || true`],
	options
	);
	}
	}

Not sure if we can add/use multiple tokens using this method.

Another alternative is to use the actions/checkout once per package to clone that dependency, and place stuff into a src folder. Is that more advisable?

That is a great idea, but unfortunately I don't think it's going to work currently. action-ros-ci creates the workspace using a .repos file by itself from scratch (in a specific directory) and deletes the directory if it exists:

action-ros-ci/src/action-ros-ci.ts

Line 407 in e0eeca0

await io.rmRF(rosWorkspaceDir);

Other than using the vcs-repo-file-url option, you can't really add other repos. This would be a great feature to contribute, though! Perhaps we could simply remove that line, though, and let users clone/checkout any other repo into that directory before action-ros-ci runs.

[jclinton830]

Perhaps we could simply remove that line, though, and let users clone/checkout any other repo into that directory before action-ros-ci runs.

@christophebedard Could you give us an example of how this could be achieved via a workflow file?

[christophebedard]

You cannot do it currently, because action-ros-ci deletes the workspace if it exists before it creates it and imports the repo being tested + the repos file(s):

action-ros-ci/src/action-ros-ci.ts

Lines 426 to 431 in 48e2c83

	// Wipe out the workspace directory to ensure the workspace is always
	// identical.
	await io.rmRF(rosWorkspaceDir);
	// Checkout ROS 2 from source and install ROS 2 system dependencies
	await io.mkdirP(rosWorkspaceDir + "/src");

. We could consider not deleting the workspace directory (line 428) so that you can import your own repo into the workspace before action-ros-ci runs. Then you'd just have a - run: git clone ... step before the ros-tooling/action-ros-ci step.

[jclinton830]

So what you are saying is that this way we can provide multiple tokens to clone more than one repo into the workspace before action-ros-ci is invoked?

[christophebedard]

Yeah, you should be able to import any repo using the proper token without action-ros-ci interfering with it. However, like I mentioned above, this isn't currently possible. The workspace directory that action-ros-ci builds is deleted and then re-created every time before building, so you can't clone a repo into it. We would need to change that.