Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deprecation/removal of ssh-rsa from PubKeyAcceptedTypes #10074

Closed
lsm5 opened this issue Oct 9, 2020 · 46 comments · Fixed by #11754
Closed

deprecation/removal of ssh-rsa from PubKeyAcceptedTypes #10074

lsm5 opened this issue Oct 9, 2020 · 46 comments · Fixed by #11754

Comments

@lsm5
Copy link

lsm5 commented Oct 9, 2020

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

The podman team recently tried to build Fedora 33 Beta images for use in our CI and we noticed SSH refused to connect and the daemon logs complained about not finding ssh-rsa in PubKeyAcceptedTypes. And this went away after making the relevant change in /etc/crypto-policies/back-ends/opensshserver.config.

Looks like Fedora 33 has removed ssh-rsa by default, and it's quite likely that other distros will as well, if not already.

It would be nice to not depend on ssh-rsa going forward.

@cevich Please correct me if I got anything wrong ^ 😄

Potential References

containers/automation_images#26 (comment)

@cevich
Copy link

cevich commented Oct 9, 2020

Suggestion: Add an ssh_keytype option to allow overriding the temporary key that packer generates. e.g. for the googlecompute builder or globally.

@SwampDragons
Copy link
Contributor

Hi, thanks for reaching out! I agree that it's a good idea to add an option for telling Packer to generate a different key type.

We could probably do this now-ish for Packer with non-rsa type keys, for example ecdsa, and that may actually provide a workaround for users currently bitten by #8609 while we wait for the golang team find the time to look at golang/go#39885.

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

@azr azr self-assigned this Oct 12, 2020
@cevich
Copy link

cevich commented Oct 12, 2020

Great, thank for picking this up, and thanks for the suggested work-around. Unfortunately because of the way our specific setup works, it's more difficult to use a custom key. Allowing the less-secure ssh-rsa key was the simpler solution, and security is a relatively low-concern for our images. I doubt that's acceptable for other people though.

@azr
Copy link
Contributor

azr commented Oct 19, 2020

Hello there, just to say we have binaries that handles this for GCP here.

You can now set the temporary_key_pair_type/temporary_key_pair_bits fields.

I'll leave this issue open until it's fixed for all builders.

@cevich
Copy link

cevich commented Oct 19, 2020

Thanks!

@azr
Copy link
Contributor

azr commented Oct 21, 2020

Hello there, I updated the binaries and the latest ones are here. Seems like to me we are going to keep these. 🙂

@cevich
Copy link

cevich commented Oct 26, 2020

FWIW: I just encountered what appears to be the ssh-rsa problem on Ubuntu 20.10 (recently released).

We're not currently setup to install packer from other than the official site, and it seems circle-ci demands an oauth2 token to access the file. Any chance of having this change included in an official alpha/beta release I can point my scripts at?

@azr
Copy link
Contributor

azr commented Oct 26, 2020

Hey @cevich, #10111 adds that feature for GCP, once it is merged you will be able to use the nightly build the day after it is merged, the PR looks like it's about to be merged so stay tuned ! 🙂

Also the upcoming release should be soon-ish.

@hswong3i
Copy link

hswong3i commented Nov 1, 2020

@defanator
Copy link

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

Well, this doesn't work for me with packer 1.6.5 and Fedora 33 Cloud Base image (ami-04c4b559df358b45a in eu-central-1). I'm using 2048-bit key with SHA256 encryption, and it allows me to log in EC2 instance launched from the above AMI, but when I'm trying to either use ssh_private_key or put the key to my local ssh agent and enable ssh_agent_auth, I'm getting handshake failures from packer logs with the following log entries at the other side (Fedora 33 instance):

Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: error: kex_exchange_identification: Connection closed by remote host
Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: Connection closed by 10.1.2.32 port 43088
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: Connection closed by authenticating user fedora 10.1.2.32 port 43090 [preauth]

I'm currently looking for at least temporary solution (without modifying original Fedora Cloud Base AMIs) - appreciate any input here.

@azr
Copy link
Contributor

azr commented Nov 16, 2020

@defanator That PubkeyAcceptedKeyTypes setting needs to be on in client and server side. If that still doesn't work I'd try with other ssh key types, ( packer can generate RSA, DSA, ECDSA and ED25519 temporary keys types for GCP currently ) . There should be at least one that works !

@defanator
Copy link

@azr I'm using pre-generated (existing) keypair which works fine on the same client (OS X 10.14) when I'm using system ssh client, but it doesn't work with the one bundled into packer.

We are using original AMI images published by OS vendors to build our custom ones, and currently we can't use Fedora 33 Cloud Base image due to this restriction in packer.

@cevich
Copy link

cevich commented Nov 16, 2020

@defanator FWIW, @lsm5 originally discovered the workaround to this problem by switching to an ED25519 key. Just to confirm, are you using the new temporary_key_pair_type option to the googlecompute builder?

@cevich
Copy link

cevich commented Nov 16, 2020

Woops...seems you're using AWS not GCP....my bad. I don't see a corresponding option for the amazon-instance builder (but I'm also no familiar with using it).

@defanator
Copy link

@cevich yes, I'm using EC2 builder, and the funny fact is that there's no way to import non-RSA key to EC2 via keypair API. But this shouldn't be an issue - as I mentioned previously, my SHA256-encoded key works just fine from anywhere except packer built-in ssh client.

@SwampDragons
Copy link
Contributor

@defanator sounds like you're facing #8609. It's an upstream bug and I wrote a patch for the golang crypto library months ago to try to get it fixed and it's waiting for review. You can track it here: golang/go#39885

There's a very messy workaround you can do while we wait: #8609 (comment)

@defanator
Copy link

@SwampDragons appreciate your input here! Yes, it looks more like #8609 but the bad thing is that we can't do the proposed workaround as Fedora 33 official (Cloud Base) images in EC2 are coming with ssh-rsa disabled already. I was thinking of another approach where we would add some bootstrap bits via userdata while launching an instance from AMI and use that for either putting another key for the default user, or even adding another user to let packer log in, but it seems like currently there's no way to specify existing instance ID instead of launching a new one from specified image (AMI). That is, I hope you'll sort this out in #8609 - thanks for your efforts!

@SwampDragons
Copy link
Contributor

Got it. This is starting to affect enough people that we're gonna have to figure something out instead of waiting for Golang to fix it. I'll talk to our security team about options.

@cmollet
Copy link

cmollet commented Jan 13, 2021

Is there any update on the temporary_key_pair_type for non-GCP builders (Digital Ocean for me specifically)? Setting temporary_key_pair_type = "ed25519" in my digitalocean builder doesn't error out but it seems to have no effect. Running packer -debug, pausing at stepCreateSSHKey and inspecting the resulting key with ssh-keygen -l shows it as an RSA key.

This is a bit of a blocker for those of using Fedora 33. The only workaround I have for now is to run packer in debug mode, pause after the DO Droplet is created, use the DO console to reset the root password, launch a web VNC session, manually tweak the sshd config to add ssh-rsa to PubkeyAcceptedKeyTypes and then resume the build. If anyone has a less tedious approach please let me know!

@cevich
Copy link

cevich commented Jan 13, 2021

If anyone has a less tedious approach please let me know!

I dunno about DO, but I imagine it has support for passing in cloud-init metadata? If so, make your edits with boothook or runcmd (docs). For F33, my projects are using a script with something like the following:

    source /etc/os-release
    if [[ "$ID" == "fedora" ]] && [[ "$VERSION_ID" -ge 33 ]]; then
        sed -i -r -e \
            's/^(PubkeyAcceptedKeyTypes )(.*)/\1ssh-rsa,\2/' \
            /etc/crypto-policies/back-ends/opensshserver.config
    fi

@cmollet
Copy link

cmollet commented Jan 14, 2021

Thanks for the suggestion, @cevich ! Directly modifying the config file didn't work, but running update-crypto-policies --set LEGACY in the user data script did! I did that for the initial Packer build and then restored the crypto-policies back to DEFAULT in the provisioner. FWIW, I still think it's a good idea to allow Packer users to specify key types and bit length for every builder type, but this is a good-enough workaround for me for now.

@chrisroberts
Copy link
Member

Hi! We have had this same problem over on the Vagrant side and wanted to share what was learned and how it is being resolved: hashicorp/vagrant#11783 (comment)

I had a skim through the golang ssh libs and it looks like the same underlying issue is causing the problems. For RSA keys the signer needs to be updated to default to sha256 and the public key type value used for the building the user auth request needs to be updated to the signature type.

@lorengordon
Copy link

@danbarr Oh, wow. Thanks for that. Ok. And now I no longer care about ed25519 support. I'll have to disable FIPS in the image, or maybe locally-generate an ecdsa key and use ssh_private_key_file with userdata to setup the public key. Bummed.

@danbarr
Copy link
Contributor

danbarr commented Oct 20, 2021

@lorengordon Indeed; I saw that announcement, got all excited, and then immediately sad again. Needed them to also do ecdsa-sha2-nistp256 or higher.

RSA-2048 keys do still work in general, as long as your SSH client is using sha256 signatures - any recent OpenSSH does, and PuTTY does as of 0.75. The issue is that the Go library used by Packer (and others) does not.

@lorengordon
Copy link

Aha, ok. And my putty version is out-of-date. So at least that explains the other part of my problem, where I couldn't get even my rsa key to work...

@vectorsigma
Copy link

FWIW, now that both Fedora 33+ and AWS support ED25519, in my Packer scripts, I set temporary_key_pair_type to ed25519 and all is well again with no bothersome user_data hacks dealing with crypto policies.

@ProFfeSsoRr
Copy link

@vectorsigma what type of builder you use? amazon-ebs ignore temporary_key_pair_type option and always created rsa.

@crabby345
Copy link

@vectorsigma what type of builder you use? amazon-ebs ignore temporary_key_pair_type option and always created rsa.

I found the same. RSA created every time. I created my own ed25519 pair and then just supplied packer the key to get around it.

@jvperrin
Copy link

I was experiencing this same problem, but using the recently released Packer 1.8.0 that includes a newer version of hashicorp/packer-plugin-amazon (including hashicorp/packer-plugin-amazon#179) and specifying "temporary_key_pair_type": "ed25519" with the amazon-ebs builder has worked great to fix this!

@Miosame
Copy link

Miosame commented Mar 17, 2022

is there any tracked issue for the other builders to implement pair type besides aws?

@puetzk
Copy link
Contributor

puetzk commented May 5, 2022

Add ubuntu 22.04 LTS "Jammy Jellyfish" to the distributions that packer's current ssh client can no longer communicate with (at least not with ssh-rsa keys)

@nywilken
Copy link
Contributor

nywilken commented May 5, 2022

Add ubuntu 22.04 LTS "Jammy Jellyfish" to the distributions that packer's current ssh client can no longer communicate with (at least not with ssh-rsa keys)

Thanks for the ping here @puetzk. Fixes to the Amazon and Azure plugins were released yesterday. You can use any of the respective releases to resolve the authentication issue.

If there is a specific packer-plugin-* that you would like to see that patch applied to please drop a list here so that we can work to prioritize their release with the patch.

@nywilken
Copy link
Contributor

nywilken commented May 5, 2022

is there any tracked issue for the other builders to implement pair type besides aws?

We don't have an master issue for tracking this as the decision to support has been mostly left to the builder plugins to support. The configuration value is already available to all builders they just need to use it when generating their respective ssh keys. That said if you have a list of builders that you would like to see this implemented in please feel free to open an issue with the list so we can work on prioritizing. Cheers!

@nywilken
Copy link
Contributor

nywilken commented May 5, 2022

Hi folks a potential fix for this issue has been merged into Packer main. It is scheduled to be released next week but will be available in the next nightly release. The SSH communication is handled by the Packer Plugin SDK within each of the builders. So even though Packer has the latest SDK the plugin responsible for executing the builder will need to be updated as well in order for the fix to work. We are tracking individual plugin updates in #11761.

To download the latest plugin you can run packer init against a HCL file that is using the required_plugins{...} block with the updated plugin version. Or you can run the packer plugins install command to get the latest version for a given plugin. For example you can run packer plugins install github.com/hashicorp/docker to install the latest version of Docker.

note: The latest nightly has the fix #11712 which should help folks using JSON

@nywilken
Copy link
Contributor

nywilken commented May 5, 2022

Oh one more thing 😄

If you are still running into issues with the proposed fix please feel free to drop a comment on the thread and we will gladly reopen. Thanks again to everyone for providing workarounds and for your patience.

Feder1co5oave added a commit to Feder1co5oave/packer-plugin-hcloud that referenced this issue Jun 10, 2022
Feder1co5oave added a commit to Feder1co5oave/packer-plugin-hcloud that referenced this issue Jun 10, 2022
Feder1co5oave added a commit to Feder1co5oave/packer-plugin-hcloud that referenced this issue Jun 10, 2022
(OpenSSH>=8.2 deprecates `ssh-rsa` pub key algorithm,
see hashicorp/packer#10074 hashicorp/packer#8609),
change docs accordingly
magec pushed a commit to magec/packer-plugin-hcloud that referenced this issue Aug 10, 2022
* set 'ssh_username' (required by SSH communicator) in acceptance test

* * use packer-plugin-sdk to manage temporary ssh keys
* support ecdsa, ed25519 temporary key algos via `temporary_key_pair_type` config
* clean up unused properties

* use Ubuntu 22.04 LTS base image in acceptance test
(OpenSSH>=8.2 deprecates `ssh-rsa` pub key algorithm,
see hashicorp/packer#10074 hashicorp/packer#8609),
change docs accordingly
@m62534
Copy link

m62534 commented Aug 15, 2022

Hello, is there a fix for this for vmware-iso?
Running into issues in Ubuntu 22.04
(Running vmware-iso version 1.0.7)

@tomsa95
Copy link

tomsa95 commented Dec 5, 2022

Hello, is there a fix for this for vmware-iso? Running into issues in Ubuntu 22.04 (Running vmware-iso version 1.0.7)

Hello, the same for me with vmware-iso,
packer version: 1.8.4

@nickvth
Copy link

nickvth commented Jan 11, 2023

A Remote Server Workaround
If you can't change anything on the local computer, or don't want to use a new key, and want to re-enable RSA on the local computer, edit the file /etc/ssh/sshd_config on the remote computer and add this line:

HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa
This will allow the use of unsafe RSA key you already have.

@inflatador
Copy link

I'm still running into this issue using the qemu builder:

SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Would it be possible to address this on all builders? I see @tomsa95 's comment above that it still affects the vmware builder.

@rubenanapu
Copy link

Using Packer 1.8.0 allowed me to use the temporary_key_pair_type key, and I was able to create the AMI (Amazon Cloud).

I was experiencing this same problem, but using the recently released Packer 1.8.0 that includes a newer version of hashicorp/packer-plugin-amazon (including hashicorp/packer-plugin-amazon#179) and specifying "temporary_key_pair_type": "ed25519" with the amazon-ebs builder has worked great to fix this!

I tried before with Packer 1.2.4, 1.3.2, 1.11.1 and none of them allowed me to use the temporary_key_pair_type key, but with Packer 1.8.0 worked.

Thank you very much, @jvperrin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.