Can't connect to new instance created from hardened image

[jcheroske]

Not sure I would call this a bug necessarily. I'm really just posting to see if it rings any bells with the ssh experts around here. Here is the situation:

I want to create a cloud disk image based on Ubuntu Minimal 16 that has this role applied, and then use that image to create additional cloud instances. The following works fine:

• Provision cloud VM with Ubuntu Minimal 16
• Apply this role to newly created VM
• Shutdown VM
• Create new image of the boot disk (Call it hardened-boot-image)

The problem is that if I then create another VM using the newly created hardened-boot-image I can not connect to it via ssh.

Can anyone think of a reason why a boot image that has this role applied to it would fail to allow connections? I can confirm the following:

• Connections to new VMs, created with a custom image that does not have this role applied, work fine
• VMs created with the hardened-boot-image do in fact boot. I just can't connect.

[rndmh3ro]

Hey @jcheroske,
sorry for not answering sooner!

What user are you trying to connect with? Is the user added ro the ssh_allow_users?

[basz]

I might have a similar issue. I provision as root, create a user with sudo capabilities, add public keys, and after hardening (with the ssh_allow_users option) I can't login at all.

It's a freshly provisioned Ubuntu Node at digitalocean

(I logged in first, so I could cat the config...)

zonderbaar@zonderbaar-extern-01:~$ sudo cat /etc/ssh/sshd_config
# #
# Ansible managed
#

# This is the ssh client system-wide configuration file.
# See sshd_config(5) for more information on any settings used. Comments will be added only to clarify why a configuration was chosen.

# Basic configuration
# ===================

# Either disable or only allow root login via certificates.
PermitRootLogin no

# Define which port sshd should listen to. Default to `22`.
Port 22

# Address family should always be limited to the active network configuration.
AddressFamily inet

# Define which addresses sshd should listen to. Default to `0.0.0.0`, ie make sure you put your desired address in here, since otherwise sshd will listen to everyone.
ListenAddress 0.0.0.0

# List HostKeys here.
HostKey /etc/ssh/ssh_host_rsa_key          # Req 20
HostKey /etc/ssh/ssh_host_ecdsa_key          # Req 20
HostKey /etc/ssh/ssh_host_ed25519_key          # Req 20

# Security configuration
# ======================

# Set the protocol version to 2 for security reasons. Disables legacy support.
Protocol 2

# Make sure sshd checks file modes and ownership before accepting logins. This prevents accidental misconfiguration.
StrictModes yes

# Logging, obsoletes QuietMode and FascistLogging
SyslogFacility AUTH
LogLevel VERBOSE

# Cryptography
# ------------

# **Ciphers** -- If your clients don't support CTR (eg older versions), cbc will be added
# CBC: is true if you want to connect with OpenSSL-base libraries
# eg ruby Net::SSH::Transport::CipherFactory requires cbc-versions of the given openssh ciphers to work
# -- see: (http://net-ssh.github.com/net-ssh/classes/Net/SSH/Transport/CipherFactory.html)
#

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

# **Hash algorithms** -- Make sure not to use SHA1 for hashing, unless it is really necessary.
# Weak HMAC is sometimes required if older package versions are used
# eg Ruby's Net::SSH at around 2.2.* doesn't support sha2 for hmac, so this will have to be set true in this case.
#

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256

# Alternative setting, if OpenSSH version is below v5.9
#MACs hmac-ripemd160

# **Key Exchange Algorithms** -- Make sure not to use SHA1 for kex, unless it is really necessary
# Weak kex is sometimes required if older package versions are used
# eg ruby's Net::SSH at around 2.2.* doesn't support sha2 for kex, so this will have to be set true in this case.
# based on: https://bettercrypto.org/static/applied-crypto-hardening.pdf

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

# Authentication
# --------------

# Secure Login directives.
UsePrivilegeSeparation sandbox
LoginGraceTime 30s
MaxAuthTries 2
MaxSessions 10
MaxStartups 10:30:100

# Enable public key authentication
PubkeyAuthentication yes

# Never use host-based authentication. It can be exploited.
IgnoreRhosts yes
IgnoreUserKnownHosts yes
HostbasedAuthentication no

# Enable PAM to enforce system wide rules
UsePAM no

# Force public key auth then ask for pam device input

# Disable password-based authentication, it can allow for potentially easier brute-force attacks.
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

# Only enable Kerberos authentication if it is configured.
KerberosAuthentication no
KerberosOrLocalPasswd no
KerberosTicketCleanup yes
#KerberosGetAFSToken no

# Only enable GSSAPI authentication if it is configured.
GSSAPIAuthentication no
GSSAPICleanupCredentials yes

# In case you don't use PAM (`UsePAM no`), you can alternatively restrict users and groups here. For key-based authentication this is not necessary, since all keys must be explicitely enabled.

AllowUsers zonderbaar





# Network
# -------

# Disable TCP keep alive since it is spoofable. Use ClientAlive messages instead, they use the encrypted channel
TCPKeepAlive no

# Manage `ClientAlive..` signals via interval and maximum count. This will periodically check up to a `..CountMax` number of times within `..Interval` timeframe, and abort the connection once these fail.
ClientAliveInterval 300
ClientAliveCountMax 3

# Disable tunneling
PermitTunnel yes

# Disable forwarding tcp connections.
# no real advantage without denied shell access
AllowTcpForwarding no

# Disable agent forwarding, since local agent could be accessed through forwarded connection.
# no real advantage without denied shell access
AllowAgentForwarding no

# Do not allow remote port forwardings to bind to non-loopback addresses.
GatewayPorts no

# Disable X11 forwarding, since local X11 display could be accessed through forwarded connection.
X11Forwarding no
X11UseLocalhost yes

# User environment configuration
# ==============================

PermitUserEnvironment no

# Misc. configuration
# ===================

Compression no

UseDNS no

PrintMotd no

PrintLastLog no

Banner none

DebianBanner no

# Reject keys that are explicitly blacklisted
RevokedKeys /etc/ssh/revoked_keys

[basz]

In my case, it turned out I created a user (via ansible) without a password. Which meant the account was locked. After giving it a random password I could harden without issues. thanks!

[rndmh3ro]

Problem

By default the user ubuntu on aws-machines has a locked password. This is getting logged on the aws-machine if I try to login with the user ubuntu and after applying the ssh-hardening role:

Aug 31 14:59:13 ip-172-31-38-172 sshd[2119]: Connection from 95.90.134.230 port 53152 on 172.31.38.172 port 22
Aug 31 14:59:13 ip-172-31-38-172 sshd[2119]: User ubuntu not allowed because account is locked
Aug 31 14:59:13 ip-172-31-38-172 sshd[2119]: Connection closed by invalid user ubuntu 95.90.134.230 port 53152 [preauth]

Reason

The default for a newly created ubuntu machine is:

• UsePAM yes in sshd_config
• user ubuntu has a locked password but is not locked (with a ! in the shadow file)

--> User can login

After applying the hardening-role:

• UsePAM no in sshd_config
• user ubuntu has a locked password but is not locked (with a ! in the shadow file)

--> User login is blocked.

The reason is that "the system" interprets a user with a ! password as locked:

# no password in shadow
ubuntu::18139:0:99999:7:::

root@ip-172-31-34-83:/home/ubuntu# passwd -S ubuntu
ubuntu NP 08/31/2019 0 99999 7 -1
# not locked, but with No Password

# exclamation mark in shadow
ubuntu:!:18139:0:99999:7:::

root@ip-172-31-34-83:/home/ubuntu# passwd -S ubuntu
ubuntu L 08/31/2019 0 99999 7 -1
# locked

A user with a * as password is still locked, however the user can still login (but ubuntu has a ! by default):

# star in shadow
ubuntu:*:18139:0:99999:7:::

root@ip-172-31-34-83:/home/ubuntu# passwd -S ubuntu
ubuntu L 08/31/2019 0 99999 7 -1
# locked

However PAM does not interpret a user with a password that contains a star or exclamation mark as locked, only that the user cannot login with a password.
That's why UsePAM yes will let the user login, and UsePAM no won't.

Workarounds

Workaround 1:
Enter UsePAM yes into the sshd_config and restart sshd. Then the user can login again.

root@ip-172-31-38-172:/home/ubuntu# grep ubuntu /etc/shadow
       | --- star = user is *NOT* locked in PAM and has no password
       v
ubuntu:*:18139:0:99999:7:::

Workaround 2:

Replace the exclamation mark with a star in the shadow-file (and do .
sed -i s/ubuntu:!/"ubuntu:*"/g /etc/shadow

see:

With a shadowed password scheme in use, the /etc/passwd file typically shows a character such as '*', or 'x' in the password field for each user instead of the hashed password, and /etc/shadow usually contains the following user information:
[...]
        Empty string – No password, the account has no password (reported by passwd on Solaris with "NP").[7]
        "!" – the account is password locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed.[8]
        "*LK*" or "*" – the account is locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed.[8]
        "!!" – the password has never been set (RedHat)[9]

• https://en.wikipedia.org/wiki/Passwd

           If the password field contains some string that is not a valid result of crypt(3), for
           instance ! or *, the user will not be able to use a unix password to log in (but the
           user may log in the system by other means).

• http://manpages.ubuntu.com/manpages/bionic/en/man5/shadow.5.html

See:

• https://ubuntuforums.org/showthread.php?t=2026413&p=12119675#post12119675
• https://serverfault.com/a/788987

Fix

We will set UsePAM yes by default, as proposed here: #23 (I was slow here ;))

Aknowledgements

After debugging this, I found a nice blog post that describes the problem: https://arlimus.github.io/articles/usepam/. THEN I saw that this blog post is from @arlimus, a dec-sec member! The circle closes.

[rndmh3ro]

Should be fixed in master. I will release a new version as soon as possible. Please try it out! :)