Can we use GitHub deploy keys to get dependencies from multiple private repositories #30
Comments
|
Yes, basically that should work. It’s also described at https://github.com/webfactory/ssh-agent/blob/master/README.md#using-multiple-keys. Can you see in the action output that both keys have been added to the agent? Could it be that your remote server fails already after one wrong key has been presented? |
|
Yes. I can see that both the keys were installed. Here's a part of the output from my github actions: Setup SSH Keys and known_hosts 1s
Run webfactory/ssh-agent@v0.3.0
with:
ssh-private-key: ***
***
env:
GOROOT: /opt/hostedtoolcache/go/1.14.2/x64
Adding GitHub.com keys to /home/runner/.ssh/known_hosts
Starting ssh-agent
Adding private key to agent
Identity added: (stdin) ((stdin))
Identity added: (stdin) ((stdin))
Keys added:
4096 SHA256:nIm8JFZytuVAb/RN1LYFg+ABZrlokJjzpFsKz4FvsO0 (stdin) (RSA)
4096 SHA256:OyXgFF71To+5RlZrSesEMCqae4LqtsiESn0+WjgUYQs (stdin) (RSA)The next step, which includes |
|
Could you please try to take Use |
|
+1
|
|
... using |
|
@felixapitzsch I haven't really had a chance to try the suggestions by @mpdude but I was able to get around the problem by creating GitHub Packages for both my private dependencies and then using those in my |
|
Oh wait! Where did you add which key? If you have one repo running the action, and you need to fetch two private dependencies, then you need to create one SSH key. The private part of it is added as the secret for the repo where the action is run. You just need to add this one key with the action. The public key part is added in both private repos that you depend upon. Set it up as a deployment key in each one. The deployment key basically says "when someone comes along and has an SSH key whose public part matches, they may read (or write) to this repo". Does that make sense? |
|
Also see this improvement for the README: 5ef9e03#diff-04c6e90faac2675aa89e2176d2eec7d8 |
|
I have three private repositories on github, let's call them A, B, and C. A holds an app, while B and C hold libs. A uses B and C as dependencies (in package.json) and has a github action for npm build and sftp deploy on each push to master. Repositories B and C have a deployment key (public key). My first attempt was trying to use the same pubilc key as a deployment key on B and C. However, github refuses to register the same public key for more than one repository and give some "already in use" error. Therefore, I was forced to add two different deployment keys for B and C. Repository A holds the two matching private keys as secrets. Actually adding more than one of these private keys will reproduce the issue described above. Adding only one of the private keys will make cloning one of B/C work and the other one fail with permissons denied (as it should be). If github would just allow reusing the same public key as a deployment key for B and C, everything would be so easy and straight forward ... |
I wasn't aware of that, good to know! Could you please try to add both private keys on your repo A and then try to clone B and C from an action using That should give some SSH verbosity to understand what |
|
Possible explanation: When connecting to the repo where the private key is the second one, GitHub might actually recognize/accept the first key, because it is known. Then, however, the My guess is the SSH handshake happens before GitHub can know about which repo is requested, so they have no chance of rejecting the "wrong" key in the first place. |
|
Not sure if we will be able to come up with a neat solution in this action, though 🤕 |
|
2020-07-02T11:00:27.1232799Z ##[endgroup] |
That would also be my guess. And I don't see any good way to fix this. A colleague suggested to try something like
or
explicitly specifying single individual keyfiles for each repository to clone. However, I want to use npm directly instead of using git clone. This would force me to specify repo A and B in my package.json as dependencies located in the local file system and use npm pre-install hooks to call git clone with the matching keyfiles in order to clone the repositories to the local filesystem and install from there. This looks super hacky to me and will not be very feasible when using the same package.json for local development. Any ideas for a different approach? |
|
Don't worry, you won't have to FWIW, I've contacted GH support and maybe they have someone who can help us here. Are you somewhat familiar with how |
|
I've tried the You can use Now the problem is that when So, the trick is to make up URLs for the individual repos. For example, using something like This leads to another problem, however: You'd not have those made-up URLs in your Clever ideas, anyone? |
|
We could write a shim for That shim would have to figure out which repo Has anybody done this before and knows what invocations such a wrapper would have to expect? (https://github.com/martinemde/git-ssh-wrapper looks a bit like it?) |
|
@felixapitzsch Please try the following:
#!/bin/bash
# The last argument is the command to be executed on the remote end, which is something
# like "git-upload-pack 'webfactory/ssh-agent.git'". We need the repo path only, so we
# loop over this last argument to get the last part of if.
for last in ${!#}; do :; done
# Don't use "exec" to run "ssh" below; then the trap won't work.
key_file=$(mktemp -u)
trap "rm -f $key_file" EXIT
# Try to pick the right key
ssh-add -L | grep $last > $key_file
ssh -i $key_file "$@"
For this, use Can you follow these instructions and report back? |
|
@chetanyakan you could also try this please |
|
GitHub support replied. In short:
Also see https://gist.github.com/vhermecz/4e2ae9468f2ff7532bf3f8155ac95c74 which is linked from the comments there. I only skimmed both briefly, but it seems they are working without |
|
I’m having the same issue except I’m using Swift Package Manager. I’ll just edit the git Config etc. |
|
@Andrewangeta could you please try if the steps described in #30 (comment) work for you? |
|
If the - uses: actions/checkout@v2
- uses: webfactory/ssh-agent@v0.4.0
with:
ssh-private-key: |
${{ secrets.SSH_REPO1 }}
${{ secrets.SSH_REPO2 }}
${{ secrets.SSH_REPO3 }}
- name: Force git to use ssh and create mappings
run: |
echo remove http header created by actions/checkout
git config http.https://github.com/.extraheader ''
echo add insteadof
git config url."git@repo1.github.com:MYORGNAME/repo1".insteadOf "https://github.com/MYORGNAME/repo1"
git config url."git@repo1.github.com:MYORGNAME/repo1".insteadOf "git@github.com:MYORGNAME/repo1"
git config url."git@repo2.github.com:MYORGNAME/repo2".insteadOf "https://github.com/MYORGNAME/repo2"
git config url."git@repo2.github.com:MYORGNAME/repo1".insteadOf "git@github.com:MYORGNAME/repo2"
git config url."git@repo3.github.com:MYORGNAME/repo3".insteadOf "https://github.com/MYORGNAME/repo3"
git config url."git@repo3.github.com:MYORGNAME/repo1".insteadOf "git@github.com:MYORGNAME/repo3"This would allow all of your references to work locally and in a GitHub action, and captures both ssh and https git references. One other note, for capturing SSH traffic in nested git calls (like npm->git or go->git), you can use: env:
GIT_TRACE: 1
GIT_CURL_VERBOSE: 1 |
|
@shaunco Did you try the approach outlined in #30 (comment)? Still hoping that this could pick the right key automatically, without requiring fancy |
|
I did, but was unsuccessful for pulling go packages from private repos ( The route I went down (and PR I provided) seem like a bit of chaos, but it abstracts all the chaos away from anyone using the action. This means they don't have to concern themselves with creating ssh keys in any special way, knowing the path to any extra scripts, or setting any extra env vars. This seems so much more user friendly: - uses: webfactory/ssh-agent@v0.5.0
with:
ssh-private-key: |
${{ secrets.FIRST_KEY }}
${{ secrets.NEXT_KEY }}
${{ secrets.ANOTHER_KEY }}
repo-mappings: |
github.com/OWNERX/REPO1
bitbucket.com/OWNERY/REPO2
github.com/OWNERX/REPO3All you need to know is which key is for which repo. And while it might feel magic, I tried to fully explain how it works in the readme. This also solves the problem of git servers booting you in you try too many invalid ssh keys, as each one only gets a single key. |
|
I made sure that using a wrapper script as described above works, and with only minor tweaks I have written a blog post about it: https://www.webfactory.de/blog/using-multiple-ssh-deploy-keys-with-github Also, I have updated the README file to point to that post and the Gist. @shaunco Regarding Go packages, I have no experience whatsoever with that. But can it be that |
|
On the Go side, it does use https, which is why part of what I added includes redirecting any attempted https request to an ssh request. Without it, all https requests are (Go or otherwise) require a separate personal access token. I pretty strongly disagree that a separate, platform specific, script is easier to maintain than the platform agnostic code I added to this action. We use this (now modified) action in about 20 repos, which would imply 20 copies of that script, and that we'd be left to find and maintain a new option for any Windows build environments. That said, it is your action, so I guess we'll carry on using my fork. |
|
I will also add that this part of your blog post is wrong:
The method that I used is 100% transparent to any apps making use of git. As I documented in the updates to the README.md, git rewrites to the made up domain names happen magically by git. You use When developing locally, you are using a personal SSH token that already has access to everything, so you don't need any of the rewrites or per-host SSH entries. So, again, my changes require NO changes to dependency declarations either locally or in CI/CD scripts on GitHub Actions. |
|
I also have the same problem as mentioned above: trying to fetch multiple private repositories in GitHubActions. Here's what I did in detail:
- uses: webfactory/ssh-agent@v0.4.1
with:
ssh-private-key: |
${{ secrets.B_SSH_PRIVATE_KEY }}
${{ secrets.C_SSH_PRIVATE_KEY }}
- name: Debugging with Git Clone 1
run: git clone git@github.com:some_user/b.git
- name: Debugging with Git Clone 2
run: git clone git@github.com:some_user/c.gitHowever, when I tried to install the private repositories via Elixir's package manager (Mix), I get the following error: Which is weird because I know that the script is there, I ran |
|
Maybe Mix runs from another |
|
I am also waiting for #53 to be merged. I had the exact same issue that @felixapitzsch @chetanyakan and @ryanzidago had. Hope this gets merged in soon |
|
For those having trouble with getting #30 (comment) to work, you can find how I got mine to work at Walawren/github-actions-test with terraform. Things to note:
|
|
I have three private repos, and I want to use their deploy keys as the secret of a forth repo. git clone git@github.com:{user}/{project}.gitHowever it won't work if I run git clone ssh://git@github.com/{user}/{project}.gitCloning a "ssh:"-prefixed url is useful when working with python projects. Pip is going to deprecate "git+git" and one alternative is "git+ssh", as described in pypa/pip#7543 and pypa/pip#7554. And from the pip install failing log, "git+ssh" is running So my question is if there can be a solution to do |
|
@DaHuoKolmostar - using ssh:// (or git_ssh://) works great using the https://github.com/shaunco/ssh-agent/tree/git-repo-mapping branch that is pending as PR #38. We use it for python with a Our action yml has the following before the - uses: shaunco/ssh-agent@git-repo-mapping
with:
ssh-private-key: |
${{ secrets.SSH_MYREPO }}
${{ secrets.SSH_MYREPOB }}
${{ secrets.SSH_MYREPOC }}
repo-mappings: |
github.com/myorg/myrepo
github.com/myorg/myrepob
github.com/myorg/myrepocWe use that branch of my fork for about 60 repos, so it will be there until it is merged back in to this project ... and likely longer than that as a few other people are using it. |
|
Please give #59 a try. |
|
You can try the #59 PR by using Instead of using the wrapper script I suggested above, it uses extra Git and SSH configuration to map particular URLs to matching deployment keys. The good news is that when you created your deployment keys with a key comment denoting the target repository, as it was suggested above and as it was part of @ecarlson94's #53, then you should not need to change anything else. Using Git + SSH extra configuration has the advantage of being more easily portable to Windows as well; also, since it uses mechanisms build into the underlying commands/tools, we avoid situations where it might not (or not easily) be possible to get the wrapper in place. I'd be glad if you could give that branch a try and leave a 👍 here if it works for you. |
I am trying to install 2 or more private GitHub repositories as an NPM dependency to another project.
In my
package.jsonfile, I have dependencies from GitHub in the following format:The following is a part of my github action yml file:
The
SSH_PRIVATE_KEYs are added as deploy keys to this repo.In the next step, I try to run
npm install.Now based on the order of the private keys, only one of these dependencies installs successfully.
I have tried to change the order of these ssh keys, which changes the dependency that cannot be installed.
Is it possible to use deploy keys to access both the private repositories in this case?
Let me know if you need more information.
The text was updated successfully, but these errors were encountered: