Abstract

Secure Shell Protocol (SSH) is one of the most important communication protocols on the Internet; it is what allows for remote administration of a large majority of Unix and Unix-like operating systems across the open Internet. However, can also be used to create both forward and reverse tunnels to bind ports either to or from one machine to another. This might seem like something silly to do, but it can be extremely useful for adding a security layer to an insecure protocol or adding a layer of security when exposing local services to the Internet. While I have lightly touched on this topic in a previous blog post, this blog post is intended to dive a bit deeper into the topic to hopefully explain how to use the tunnels, and some use-cases that might benefit from them.

Setup

The setup for this project is fairly simple and only requires OpenSSH client and server. Another utility that might be useful for daemonizing the tunnel would be AutoSSH; the utility is best described from their GitHub description:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The original idea and the mechanism were from rstunnel (Reliable SSH Tunnel). With version 1.2 the method changed: autossh now uses ssh to construct a loop of ssh forwardings (one from local to remote, one from remote to local), and then sends test data that it expects to get back. (The idea is thanks to Terrence Martin.)

AutoSSH will monitor the connection and try to re-open it upon failure. This is exactly what you would want for creating a service to make the tunnel persistent, however, we will come back to this utility later.

Creating Basic Tunnels

For setting up and testing these tunnels, we really just need two machines (VMs, containers, or jails work just fine for this). One machine will act as the “remote” and the other will act as the “local”; they will need the OpenSSH server and client respectively. Then for the tunnels to properly work, the following setting needs to be changed on the server in the sshd_config file (usually located at /etc/ssh/sshd_config).

# Change from this line
# GatewayPorts no

# To this line
GatewayPorts yes

and finally restart the sshd service.

# systemD Distros
systemctl restart sshd

# Void Linux
sv restart sshd

# FreeBSD and NetBSD
service sshd restart

# OpenBSD
rcctl restart sshd

Now our system is ready to securely forward ports.

Forward SSH Tunnel

Let’s look at a really basic example of tunneling a VNC connection to a local machine:

ssh -L 3000:localhost:5900 ${USER}@${HOST}

This command will set up an SSH connection and tunnel port 5900 on the remote machine to port 3000 on the local machine. Then, if you want to try to connect to the machine via VNC, you connect to localhost:3000 rather than ${REMOTE_ADDRESS}:5900. Fairly simple, and a much better way to connect to VNC over the opened Internet; however, it is not limited to just VNC. Rather we can tunnel any TCP port on the remote machine to a port on our local machine. From the ssh man page:

-L [bind_address:]port:host:hostport -L [bind_address:]port:remote_socket -L local_socket:host:hostport -L local_socket:remote_socket Specifies that connections to the given TCP port or Unix socket on the local (client) host are to be forwarded to the given host and port, or Unix socket, on the remote side. This works by allocating a socket to listen to either a TCP port on the local side, optionally bound to the specified bind_address, or to a Unix socket. Whenever a connection is made to the local port or socket, the connection is forwarded over the secure channel, and a connection is made to either host port hostport, or the Unix socket remote_socket, from the remote machine.

The documentation is fairly clear about what is actually occurring here, and the limitation of TCP ports makes sense since SSH is a TCP only protocol. While it is possible to do UDP over SSH, it is more complicated and not something that I am going to delve into on this post. The part that was unclear, at least for me, is the order the ports should go in on the command. In short, for the forward SSH tunnels (what the -L flag is used for) it is supposed to be local port, than remote port.

# Generalized example
ssh -L ${LOCAL_PORT}:localhost:${REMOTE_PORT} ${REMOTE_IP}

# Forwarding remote port 8080 to local port 3000
ssh -L 3000:localhost:8080 192.168.122.146

The forward tunnel’s use-case is to secure vulnerable or sensitive remote services. Some examples might be things like VNC, a WordPress admin portal, or any other service that a bit too sensitive or insecure to share over the opened Internet.

Reverse SSH Tunnel

Now let’s look at the reverse tunnels. Again, from the ssh man page:

-R [bind_address:]port:host:hostport -R [bind_address:]port:local_socket -R remote_socket:host:hostport -R remote_socket:local_socket -R [bind_address:]port Specifies that connections to the given TCP port or Unix socket on the remote (server) host are to be forwarded to the local side.

This works by allocating a socket to listen to either a TCP port or to a Unix socket on the remote side. Whenever a connection is made to this port or Unix socket, the connection is forwarded over the secure channel, and a connection is made from the local machine to either an explicit destination specified by host port hostport, or local_socket, or, if no explicit destination was specified, ssh will act as a SOCKS 4/5 proxy and forward connections to the destinations requested by the remote SOCKS client.

Again, the documentation is fairly clear on what is going on, and again the limitation of TCP connections makes sense as SSH is still a TCP protocol. The main difference is the order the ports will be declared in the command, it will go the remote port, than the local port.

# Generalized example
ssh -R ${REMOTE_PORT}:localhost:${LOCAL_PORT}

# Forwarding data from local port 8000 to the remote port 3000
ssh -R 3000:localhost:8000 ${USER}@192.168.122.99

This type of tunnel is going to be best used for something like securely exposing local services to the Internet. It can be thought of as an alternative to a service like a Cloudflare tunnel or a Tailscale Funnel, but are not cloud based services and can be fully self-managed.

Creating Tunnels as Services

While the previous commands are easy enough, there are a few things that can be polished a bit. The first part is after the SSH tunnel is established, we don’t really have a reason to have a shell. Thankfully, the open SSH client already has an option for this, -N. From the man page:

-N Do not execute a remote command. This is useful for just forwarding ports. Refer to the description of SessionType in ssh_config(5) for details.

Perfect, just add that flag to any of the previous commands, and it will only set up the tunnel, rather than starting a shell.

# This command
ssh -R 3000:localhost:8000 ${USER}@192.168.122.99

# Becomes this command
ssh -N -R 3000:localhost:8000 ${USER}@192.168.122.99

# Though order is not important, this command is equivalent
ssh -R 3000:localhost:8000 ${USER}@192.168.122.99 -N

The next issue is being able to set up these tunnels as services and maintaining that connection. SSH doesn’t have a built in way to monitor and maintain the connection. We could certainly use something like the daemon utility in FreeBSD, but if the connection died, it would not be able to automatically recover.

This is where a tool like autossh comes in. From the man page:

autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

The original idea and the mechanism were from rstunnel (Reliable SSH Tunnel). With version 1.2 the method changed: autossh now uses ssh to construct a loop of ssh forwardings (one from local to remote, one from remote to local), and then sends test data that it expects to get back. (The idea is thanks to Terrence Martin.)

The main thing that is different with autoSSH is that we also need to provide a port that is separate for monitoring the connection. This means the stated port will need to be able to receive traffic on whatever firewall is on the remote server. Here is an example of an autoSSH command:

# Monitor on port 6500, then normal SSH commands and flags after
autossh -M 6500 -N -R 3000:localhost:8000 192.168.122.146

Additionally, it has a flag for running in the background, -f; so our previous command can be changed into:

# Monitor on port 6500, then normal SSH commands and flags after
autossh -f -M 6500 -N -R 3000:localhost:8000 192.168.122.146

Now we need to make sure that the service user can login without a password using SSH keys. For this, simply generate the key using ssh-keygen then copy the contents of the generated public key into /home/${SERVICE_USER}/.ssh/authorized_keys; test using SSH to confirm it worked. If you had to enter a password, it will not work as there is not a good way to make a service do that.

Finally, we would just need to create a service. I put the following in /etc/systemd/system/tunneld.service and it worked for my testing:

[Unit]
Description=AutoSSH tunnel service
After=network.target

[Service]
User=${SERVICE_USER}
Group=${SERVICE_GROUP}
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 6500 -i /home/${SERVICE_USER}/.ssh/id_ed25519 -N -R 8000:localhost:8096 ${SERVICE_USER}@192.168.122.146
Restart=always

[Install]
WantedBy=multi-user.target

Obviously replace ${SERVICE_USER} and ${SERVICE_GROUP} with the relevant service user and group for the service. After that, run the following commands to enable and start the service:

# Run as root to enable service
systemctl enable tunneld

# Start service
systemctl start tunneld

Obviously creating services on other init systems are possible, but will be left as an exercise for the reader.