20240616-u24_vps_hardening.jpeg

<aside> đź”’ Hosting on a VPS is a great option to run a blogging service, but installing services that might expose ports needs to be done with some precaution (or not at all if the service is only to be used by the server itself).

</aside>

Revision: 20240922-0 (init: 20240616)

This post will discuss some of the concepts required to harden your VPS, such as minimizing its attack surface, strengthening access controls, and applying security patches and updates to reduce the risk of unauthorized access.

Securing a VPS (Virtual Private Server) protects data, applications, and the overall system from unauthorized access and potential threats. Without adequate security measures, your VPS could become part of a botnet, be used to host phishing websites, send scam emails, and be used for crypto mining or other illegal activities. Depending on the content of your VPS, this might also cause some data theft for your end users, or your VPS might become encrypted in a ransomware attack.

As such, “hardening” your VPS refers to enhancing your server's security by implementing various measures to reduce its vulnerability to attacks and unauthorized access. It involves configuring settings, applying updates, and utilizing security tools to create a more robust and resilient system.

This setup was done on an OVHcloud VPS (a VLE-4), but the instructions below quickly adapt to Linode or Digital Ocean. We will not cover the initial host creation of a Ubuntu 24.04 server VM on your cloud provider of choice.

Remote Access Setup

Generally, suppose you're primarily using your VPS to host websites, run web applications, or do other server-related tasks. In that case, a server OS is highly recommended for its performance, security, and management advantages.

Depending on the cloud provider, your ubuntu account will get your ssh key added for login or you will get a temporary password to change at initial login.

After obtaining your ubuntu user password and host details, initialize a new ~/.ssh/config.d/vps file (you can include all files from the .config.d directory by adding Include config.d/* at the top of the ~/.ssh/config file) and add access details to it:

Host vps
HostName <VPS-DETAILS>
User ubuntu

If you log in using a password, after your initial ssh vps, change it to a secure one (stored in a password manager if needed) using passwd.

The first step is to make sure our host is up-to-date by running sudo apt update && sudo apt upgrade followed by a reboot using sudo reboot -h now

The reboot will help if any kernel upgrade is performed but will also allow us to confirm the login with the new password is functional.

Creating a new SSH key for use with this VPS is easy; use ssh-keygen -t ed25519 -c "vps" -f ~/.ssh/id_ed25519-vps to create one and armor it with a passphrase if preferred (you will want to make sure to add it to your ssh-add or keychain if you do). Then copy the public key to the remote host’s .ssh/authorized_keys:

rsync -avR ~/./.ssh/id_ed25519-vps.pub vps:./.ssh/authorized_keys

Test to confirm that you are not challenged by a password the next time you log into the vps host.

Changing the default user

Because server installs often create default users, those are the first usernames attempted by bots, as such let’s add a new sudo-capable vpsu user:

sudo adduser vpsu
sudo usermod -aG sudo vpsu
sudo rsync --archive --chown=vpsu:vpsu ~/.ssh /home/vpsu

, then test that you can log in as the vpsu user: ssh vpsu@vps

Hardening ssh

Next, let’s harden the ssh configuration by modifying: sudo nano /etc/ssh/sshd_config.

Add/modify the following options (in order within the file):

Host vps
HostName <VPS-DETAILS>
User vpsu
Port 1327
IdentityFile ~/.ssh/id_ed25519-vps
IdentitiesOnly yes

, then, on your host, try to ssh ubuntu@vps again. You should not only be denied the option to enter a password, but your ubuntu user should also get Permission denied (publickey)

Finally, while still logged in as vps on the VM, on your host, confirm you can ssh vps

Further reading: https://support.us.ovhcloud.com/hc/en-us/articles/115001669550-How-to-Secure-Your-SSH-Connection-in-Ubuntu-18-04

A note on the default list of authorized Ciphers

The default list is considered secure, and modifying it makes you responsible for confirming that your client can support it.

You can find the list of your server ciphers by running on your VPS: sudo sshd -T | grep "\\(ciphers\\|macs\\|kexalgorithms\\)". The ciphers line should only contain chacha and aes answers, which are considered secure.

From your client side, you can find the list of the ctos and stoc ( client to server, server to client) supported ciphers by using ssh -vv vps. The content will be in some of the debug2 lines.

To limit that list to a smaller subset, you can edit the /etc/ssh/sshd_config file and add (or alter if one is already present) ciphers line. For example:

ciphers [email protected],aes256-ctr,[email protected]

, followed by a restart of the ssh daemon: sudo service ssh restart

You can confirm the list matches by re-running sudo sshd -T | grep "\\(ciphers\\|macs\\|kexalgorithms\\)"

Secure DNS

For most of my systems, I use NextDNS or Control-D to block outgoing trackers. I will use either Cloudflare or Quad9’s DNS resolvers for a VPS.

If you have a domain name whose DNS is hosted at Cloudflare and intend to use Cloudflare’s Zero Trust tunnels, cloudflared is the recommended solution. For additional details, see https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/

DNS-over-HTTPS using cloudflared

We will use Cloudflare’s cloudflared's alternate configuration as a ”malware blocking” DNS.

Obtain and install the latest client:

cd /tmp
wget <https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb>
sudo dpkg -i cloudflared-linux-amd64.deb
rm -f cloudflared-linux-amd64.deb

Run a test in a tmux (or have another ssh into the host). In one terminal, start the client:

cloudflared proxy-dns --port 5053 --upstream <https://1.1.1.2/dns-query> --upstream <https://1.0.0.2/dns-query>

Confirm the configuration is working by starting from the other terminal a connection to the DNS resolver on port 5053:

dig @127.0.0.1 -p 5053 cloudflare.com AAAA

You should see the DNS request in the first terminal using the specified https addresses. Kill the cloudflared command and let’s make this configuration a system service; sudo nano /etc/systemd/system/cloudflared-proxy-dns.service and use the following:

[Unit]
Description=DNS over HTTPS (DoH) proxy client
Wants=network-online.target nss-lookup.target
Before=nss-lookup.target

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DynamicUser=yes
ExecStart=/usr/local/bin/cloudflared proxy-dns --upstream <https://1.1.1.2/dns-query> --upstream <https://1.0.0.2/dns-query>

[Install]
WantedBy=multi-user.target

Enable the new service:

sudo systemctl enable --now cloudflared-proxy-dns

, and confirm it is running and using the proper upstream resolvers:

sudo service cloudflared-proxy-dns status

Modify /etc/systemd/resolved.conf having it use DNS=127.0.0.1 as the only active entry in the [Resolve] section, then restart the service:

sudo service systemd-resolved restart

Further reading: https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/dns-over-https-client/

DNS-over-TLS using Quad9

If you do not use Cloudflare tunnels, you can use Quad9 for its threat-blocking features and upgrade our connection from plain DNS to DNS-over-TLS. For details on available services, see https://www.quad9.net/service/service-addresses-and-features

Edit the /etc/systemd/resolved.conf file and enter the following [Resolve] section:

[Resolve]
DNS=9.9.9.9#dns.quad9.net
DNS=2620:fe::fe#dns.quad9.net
DNS=149.112.112.112#dns.quad9.net
DNS=2620:fe::9#dns.quad9.net
DNSOverTLS=yes

Restart the service for the changes to take effect: sudo service systemd-resolved restart

Test that we are using Quad9:

dig +short txt on.quad9.net.

, should return yes.quad9.net.

Preventing access and limiting entry

ufw (Uncomplicated Firewall) acts as a first line of defense by filtering incoming and outgoing traffic based on predefined rules. fail2ban monitors the system logs for suspicious activity and automatically bans IP addresses showing malicious behavior. Both ufw and fail2ban interact with the underlying iptables firewall to manage rules and actions. The combination of ufw and fail2ban helps protect against various threats, including unauthorized access, brute-force attacks, and other automated attacks.

ufw

Set ufw to deny all incoming connections and allow traffic out:

sudo ufw default deny incoming
sudo ufw default allow outgoing

Allow incoming ssh connections using our alternate port:

sudo ufw allow 1327

Show the existing rules to confirm ssh is added by using sudo ufw show added

Enable ufw on the host:

sudo ufw enable

You can check the existing rules and delete some if you want (it is recommended to do it from the higher number first not to have to check the numbered list again):

sudo ufw status numbered
sudo udw delete <RULE_NUMBER>

You can also check all the iptables entry added:

sudo iptables -S

Further reading: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu

ufw-docker

As we intend to run docker commands on this VPS, we want to make sure our firewall rules are not bypassed by Docker. This is currently the case; if we run a service that expose a port, that port is not blocked by ufw as Docker modified alternate rules with iptables.

https://github.com/chaifeng/ufw-docker both details the issue and propose a solution to update the host’s iptables rules to fix this issue.

We will install it and use use it by following the instructions provided on the GitHub repo:

# Download the command
sudo wget -O /usr/local/bin/ufw-docker \\
  <https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker>
sudo chmod +x /usr/local/bin/ufw-docker

# Install the iptables
sudo ufw-docker install

# Restart ufw for those to take effect
sudo systemctl restart ufw

Adding a container to the ufw rules is handled differently. From their example:


Untitled

Untitled