Securing the SSH Server

Important Note Before You Make SSH Changes

It is highly advised you keep a 2nd terminal open to your server before you make and apply SSH configuration changes. This way if you lock yourself out of your 1st terminal session, you still have one session connected so you can fix it.

Thank you to Sonnenbrand for this idea.

SSH Public/Private Keys

Using SSH public/private keys is more secure than using a password. It also makes it easier and faster, to connect to our server because you don’t have to enter a password.

How SSH Public/Private Keys Work

Check the references below for more details but, at a high level, public/private keys work by using a pair of keys to verify identity.

  1. One key, the public key, can only encrypt data, not decrypt it
  2. The other key, the private key, can decrypt the data

For SSH, a public and private key is created on the client. You want to keep both keys secure, especially the private key. Even though the public key is meant to be public, it is wise to make sure neither keys fall in the wrong hands.

When you connect to an SSH server, SSH will look for a public key that matches the client you’re connecting from in the file ~/.ssh/authorized_keys on the server you’re connecting to. Notice the file is in the home folder of the ID you’re trying to connect to. So, after creating the public key, you need to append it to ~/.ssh/authorized_keys. One approach is to copy it to a USB stick and physically transfer it to the server. Another approach is to use use ssh-copy-id to transfer and append the public key.

After the keys have been created and the public key has been appended to ~/.ssh/authorized_keys on the host, SSH uses the public and private keys to verify identity and then establish a secure connection. How identity is verified is a complicated process but Digital Ocean has a very nice write-up of how it works. At a high level, identity is verified by the server encrypting a challenge message with the public key, then sending it to the client. If the client cannot decrypt the challenge message with the private key, the identity can’t be verified and a connection will not be established.

They are considered more secure because you need the private key to establish an SSH connection. If you set PasswordAuthentication no in /etc/ssh/sshd_config, then SSH won’t let you connect without the private key.

You can also set a pass-phrase for the keys which would require you to enter the key pass-phrase when connecting using public/private keys. Keep in mind doing this means you can’t use the key for automation because you’ll have no way to send the passphrase in your scripts. ssh-agent is a program that is shipped in many Linux distros (and usually already running) that will allow you to hold your unencrypted private key in memory for a configurable duration. Simply run ssh-add and it will prompt you for your passphrase. You will not be prompted for your passphrase again until the configurable duration has passed.

We will be using Ed25519 keys which, according to https://linux-audit.com/:

It is using an elliptic curve signature scheme, which offers better security than ECDSA and DSA. At the same time, it also has good performance.

Security Goals

  • Ed25519 public/private SSH keys:
    • private key on your client
    • public key on your server

Notes

  • You’ll need to do this step for every computer and account you’ll be connecting to your server from/as.

Steps - Generate SSH Keys

  1. From the computer you’re going to use to connect to your server, the client, not the server itself, create an Ed25519 key with ssh-keygen:

     ssh-keygen -t ed25519
    
    Generating public/private ed25519 key pair.
    Enter file in which to save the key (/home/user/.ssh/id_ed25519):
    Created directory '/home/user/.ssh'.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/user/.ssh/id_ed25519.
    Your public key has been saved in /home/user/.ssh/id_ed25519.pub.
    The key fingerprint is:
    SHA256:F44D4dr2zoHqgj0i2iVIHQ32uk/Lx4P+raayEAQjlcs [email protected]
    The key's randomart image is:
    +--[ED25519 256]--+
    |xxxx  x          |
    |o.o +. .         |
    | o o oo   .      |
    |. E oo . o .     |
    | o o. o S o      |
    |... .. o o       |
    |.+....+ o        |
    |+.=++o.B..       |
    |+..=**=o=.       |
    +----[SHA256]-----+
    

    Note: If you set a passphrase, you’ll need to enter it every time you connect to your server using this key, unless you’re using ssh-agent.

  2. Now you need to append the public key ~/.ssh/id_ed25519.pub from your client to the ~/.ssh/authorized_keys file on your server. Since we’re presumable still at home on the LAN, we’re probably safe from MIM attacks, so we will use ssh-copy-id to transfer and append the public key:

     ssh-copy-id [email protected]
    
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/user/.ssh/id_ed25519.pub"
    The authenticity of host 'host (192.168.1.96)' can't be established.
    ECDSA key fingerprint is SHA256:QaDQb/X0XyVlogh87sDXE7MR8YIK7ko4wS5hXjRySJE.
    Are you sure you want to continue connecting (yes/no)? yes
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    [email protected]'s password:
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh '[email protected]'"
    and check to make sure that only the key(s) you wanted were added.
    

Now would be a good time to perform any tasks specific to your setup.

Create SSH Group For AllowGroups

To make it easy to control who can SSH to the server. By using a group, we can quickly add/remove accounts to the group to quickly allow or not allow SSH access to the server.

How It Works

We will use the AllowGroups option in SSH’s configuration file /etc/ssh/sshd_config to tell the SSH server to only allow users to SSH in if they are a member of a certain UNIX group. Anyone not in the group will not be able to SSH in.

Goals

Notes

Steps - Create SSH Group For AllowGroups

  1. Create a group:

     sudo groupadd sshusers
    
  2. Add account(s) to the group:

     sudo usermod -a -G sshusers user1
     sudo usermod -a -G sshusers user2
     sudo usermod -a -G sshusers ...
    

    You’ll need to do this for every account on your server that needs SSH access.

Secure /etc/ssh/sshd_config

SSH is a door into your server. This is especially true if you are opening ports on your router so you can SSH to your server from outside your home network. If it is not secured properly, a bad-actor could use it to gain unauthorized access to your system.

How It Works

/etc/ssh/sshd_config is the default configuration file that the SSH server uses. We will use this file to tell what options the SSH server should use.

Goals

  • a secure SSH configuration

Notes

Steps - Secure /etc/ssh/sshd_config

  1. Make a backup of OpenSSH server’s configuration file /etc/ssh/sshd_config and remove comments to make it easier to read:

     sudo cp --archive /etc/ssh/sshd_config /etc/ssh/sshd_config-COPY-$(date +"%Y%m%d%H%M%S")
     sudo sed -i -r -e '/^#|^$/ d' /etc/ssh/sshd_config
    
  2. Edit /etc/ssh/sshd_config then find and edit or add these settings that should be applied regardless of your configuration/setup:

    Note: SSH does not like duplicate contradicting settings. For example, if you have ChallengeResponseAuthentication no and then ChallengeResponseAuthentication yes, SSH will respect the first one and ignore the second. Your /etc/ssh/sshd_config file may already have some of the settings/lines below. To avoid issues you will need to manually go through your /etc/ssh/sshd_config file and address any duplicate contradicting settings.

     ########################################################################################################
     # start settings from https://infosec.mozilla.org/guidelines/openssh#modern-openssh-67 as of 2019-01-01
     ########################################################################################################
    
     # Supported HostKey algorithms by order of preference.
     HostKey /etc/ssh/ssh_host_ed25519_key
     HostKey /etc/ssh/ssh_host_rsa_key
     HostKey /etc/ssh/ssh_host_ecdsa_key
    
     KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
    
     Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
    
     MACs [email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
    
     # LogLevel VERBOSE logs user's key fingerprint on login. Needed to have a clear audit track of which key was using to log in.
     LogLevel VERBOSE
    
     # Use kernel sandbox mechanisms where possible in unprivileged processes
     # Systrace on OpenBSD, Seccomp on Linux, seatbelt on MacOSX/Darwin, rlimit elsewhere.
     # Note: This setting is deprecated in OpenSSH 7.5 (https://www.openssh.com/txt/release-7.5)
     # UsePrivilegeSeparation sandbox
    
     ########################################################################################################
     # end settings from https://infosec.mozilla.org/guidelines/openssh#modern-openssh-67 as of 2019-01-01
     ########################################################################################################
    
     # don't let users set environment variables
     PermitUserEnvironment no
    
     # Log sftp level file access (read/write/etc.) that would not be easily logged otherwise.
     Subsystem sftp  internal-sftp -f AUTHPRIV -l INFO
    
     # only use the newer, more secure protocol
     Protocol 2
    
     # disable X11 forwarding as X11 is very insecure
     # you really shouldn't be running X on a server anyway
     X11Forwarding no
    
     # disable port forwarding
     AllowTcpForwarding no
     AllowStreamLocalForwarding no
     GatewayPorts no
     PermitTunnel no
    
     # don't allow login if the account has an empty password
     PermitEmptyPasswords no
    
     # ignore .rhosts and .shosts
     IgnoreRhosts yes
    
     # verify hostname matches IP
     UseDNS yes
    
     Compression no
     TCPKeepAlive no
     AllowAgentForwarding no
     PermitRootLogin no
    
     # don't allow .rhosts or /etc/hosts.equiv
     HostbasedAuthentication no
    
  3. Then find and edit or add these settings, and set values as per your requirements:

    Setting Valid Values Example Description Notes
    AllowGroups local UNIX group name AllowGroups sshusers group to allow SSH access to  
    ClientAliveCountMax number ClientAliveCountMax 0 maximum number of client alive messages sent without response  
    ClientAliveInterval number of seconds ClientAliveInterval 300 timeout in seconds before a response request  
    ListenAddress space separated list of local addresses <ul><li>ListenAddress 0.0.0.0</li><li>ListenAddress 192.168.1.100</li></ul> local addresses sshd should listen on See Issue #1 for important details.
    LoginGraceTime number of seconds LoginGraceTime 30 time in seconds before login times-out  
    MaxAuthTries number MaxAuthTries 2 maximum allowed attempts to login  
    MaxSessions number MaxSessions 2 maximum number of open sessions  
    MaxStartups number MaxStartups 2 maximum number of login sessions  
    PasswordAuthentication yes or no PasswordAuthentication no if login with a password is allowed  
    Port any open/available port number Port 22 port that sshd should listen on  

    Check man sshd_config for more details what these settings mean.

  4. Make sure there are no duplicate settings that contradict each other. The below command should not have any output.

     awk 'NF && $1!~/^(#|HostKey)/{print $1}' /etc/ssh/sshd_config | sort | uniq -c | grep -v ' 1 '
    
  5. Restart ssh:

     sudo service sshd restart
    
  6. You can check verify the configurations worked with sshd -T and verify the output:

     sudo sshd -T
    
    port 22
    addressfamily any
    listenaddress [::]:22
    listenaddress 0.0.0.0:22
    usepam yes
    logingracetime 30
    x11displayoffset 10
    maxauthtries 2
    maxsessions 2
    clientaliveinterval 300
    clientalivecountmax 0
    streamlocalbindmask 0177
    permitrootlogin no
    ignorerhosts yes
    ignoreuserknownhosts no
    hostbasedauthentication no
    ...
    subsystem sftp internal-sftp -f AUTHPRIV -l INFO
    maxstartups 2:30:2
    permittunnel no
    ipqos lowdelay throughput
    rekeylimit 0 0
    permitopen any
    

Remove Short Diffie-Hellman Keys

Per Mozilla’s OpenSSH guidelines for OpenSSH 6.7+, “all Diffie-Hellman moduli in use should be at least 3072-bit-long”.

The Diffie-Hellman algorithm is used by SSH to establish a secure connection. The larger the moduli (key size) the stronger the encryption.

Goals

  • remove all Diffie-Hellman keys that are less than 3072 bits long

Steps - Remove Short Diffie-Hellman Keys

  1. Make a backup of SSH’s moduli file /etc/ssh/moduli:

     sudo cp --archive /etc/ssh/moduli /etc/ssh/moduli-COPY-$(date +"%Y%m%d%H%M%S")
    
  2. Remove short moduli:

     sudo awk '$5 >= 3071' /etc/ssh/moduli | sudo tee /etc/ssh/moduli.tmp
     sudo mv /etc/ssh/moduli.tmp /etc/ssh/moduli
    

2FA/MFA for SSH

Even though SSH is a pretty good security guard for your doors and windows, it is still a visible door that bad-actors can see and try to brute-force in. Fail2ban will monitor for these brute-force attempts but there is no such thing as being too secure. Requiring two factors adds an extra layer of security.

Using Two Factor Authentication (2FA) / Multi Factor Authentication (MFA) requires anyone entering to have two keys to enter which makes it harder for bad actors. The two keys are:

  1. Their password
  2. A 6 digit token that changes every 30 seconds

Without both keys, they won’t be able to get in.

Why Not

Many folks might find the experience cumbersome or annoying. And, access to your system is dependent on the accompanying authenticator app that generates the code.

How It Works

On Linux, PAM is responsible for authentication. There are four tasks to PAM that you can read about at https://en.wikipedia.org/wiki/Linux_PAM. This section talks about the authentication task.

When you log into a server, be it directly from the console or via SSH, the door you came through will send the request to the authentication task of PAM and PAM will ask for and verify your password. You can customize the rules each doors use. For example, you could have one set of rules when logging in directly from the console and another set of rules for when logging in via SSH.

This section will alter the authentication rules for when logging in via SSH to require both a password and a 6 digit code.

We will use Google’s libpam-google-authenticator PAM module to create and verify a TOTP key. https://fastmail.blog/2016/07/22/how-totp-authenticator-apps-work/ and https://jemurai.com/2018/10/11/how-it-works-totp-based-mfa/ have very good writeups of how TOTP works.

What we will do is tell the server’s SSH PAM configuration to ask the user for their password and then their numeric token. PAM will then verify the user’s password and, if it is correct, then it will route the authentication request to libpam-google-authenticator which will ask for and verify your 6 digit token. If, and only if, everything is good will the authentication succeed and user be allowed to log in.

Goals

  • 2FA/MFA enabled for all SSH connections

Notes

  • Before you do this, you should have an idea of how 2FA/MFA works and you’ll need an authenticator app on your phone to continue.
  • We’ll use google-authenticator-libpam.
  • With the below configuration, a user will only need to enter their 2FA/MFA code if they are logging on with their password but not if they are using SSH public/private keys. Check the documentation on how to change this behavior to suite your requirements.

Steps to Enable 2FA/MFA for SSH

  1. Install it libpam-google-authenticator.

    On Debian based systems:

     sudo apt install libpam-google-authenticator
    
  2. Make sure you’re logged in as the ID you want to enable 2FA/MFA for and execute google-authenticator to create the necessary token data:

     google-authenticator
    
    Do you want authentication tokens to be time-based (y/n) y
    https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/[email protected]%3Fsecret%3DR4ZWX34FQKZROVX7AGLJ64684Y%26issuer%3Dhost
    
    ...
    
    Your new secret key is: R3NVX3FFQKZROVX7AGLJUGGESY
    Your verification code is 751419
    Your emergency scratch codes are:
      12345678
      90123456
      78901234
      56789012
      34567890
    
    Do you want me to update your "/home/user/.google_authenticator" file (y/n) y
    
    Do you want to disallow multiple uses of the same authentication
    token? This restricts you to one login about every 30s, but it increases
    your chances to notice or even prevent man-in-the-middle attacks (y/n) Do you want to disallow multiple uses of the same authentication
    token? This restricts you to one login about every 30s, but it increases
    your chances to notice or even prevent man-in-the-middle attacks (y/n) y
    
    By default, tokens are good for 30 seconds. In order to compensate for
    possible time-skew between the client and the server, we allow an extra
    token before and after the current time. If you experience problems with
    poor time synchronization, you can increase the window from its default
    size of +-1min (window size of 3) to about +-4min (window size of
    17 acceptable tokens).
    Do you want to do so? (y/n) y
    
    If the computer that you are logging into isn't hardened against brute-force
    login attempts, you can enable rate-limiting for the authentication module.
    By default, this limits attackers to no more than 3 login attempts every 30s.
    Do you want to enable rate-limiting (y/n) y
    

    Notice this is not run as root.

    Select default option (y in most cases) for all the questions it asks and remember to save the emergency scratch codes.

  3. Make a backup of PAM’s SSH configuration file /etc/pam.d/sshd:

     sudo cp --archive /etc/pam.d/sshd /etc/pam.d/sshd-COPY-$(date +"%Y%m%d%H%M%S")
    
  4. Now we need to enable it as an authentication method for SSH by adding this line to /etc/pam.d/sshd:

     auth       required     pam_google_authenticator.so nullok
    

    Note: Check here for what nullok means.

     echo -e "\nauth       required     pam_google_authenticator.so nullok         # added by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")" | sudo tee -a /etc/pam.d/sshd
    
  5. Tell SSH to leverage it by adding or editing this line in /etc/ssh/sshd_config:

     ChallengeResponseAuthentication yes
    
     sudo sed -i -r -e "s/^(challengeresponseauthentication .*)$/# \1         # commented by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")/I" /etc/ssh/sshd_config
     echo -e "\nChallengeResponseAuthentication yes         # added by $(whoami) on $(date +"%Y-%m-%d @ %H:%M:%S")" | sudo tee -a /etc/ssh/sshd_config
    
  6. Restart ssh:

     sudo service sshd restart
    

Licenses and Attributions


Speak Your Mind