code.jeremyevans.net

Using An SSH Reverse Tunnel for Remote System Maintenance (2022-01-20)

SSH is widely known as the tool to use to remotely connect to other computers and run commands on them. However, what if you need to run commands on a computer that you don't have the ability to currently connect to? Well, that becomes substantially more difficult, and it helps to plan in advance for how you will restore your access to remotely connect. I happened to find myself in such a situation, years ago, and setup such a plan. I mostly forgot about it until recently, when it was needed, and I had to remember what I had setup, and how to use it, in order to restore the access I needed. Read on for the background behind the need, how I setup SSH to handle access restoration, the problems I ran into when I finally had to use it, and how I worked around them.

Background

Like many technically-oriented people, I have other not-so-technically-oriented people in my extended family that I provide technical support for. In this particular case, I have a VPN between my home network and the home network of the other family member. This VPN allows for both remote troubleshooting and for backing up each network to the other for disaster recovery.

I originally set up the VPN between our home networks in late 2005, after the introduction of ipsec.conf in OpenBSD 3.8, which allowed for very easily setting up an IPSEC VPN between two networks. In 2021, I converted the VPN from using IPSEC to using Wireguard, using the instructions in wg(4), but that's not really important to the story at hand.

What is important to the story at hand is that this other home network is behind an DSL modem, and the DSL modem would occasionally have its configuration reset. The default DSL modem configuration did not allow for the VPN to work, because it used network address translation on all devices behind it. The only way to get the VPN to work would be to put the DSL modem in passthrough mode, where a particular device behind the modem (an OpenBSD firewall in this case) would be assigned the public IP address. Once the OpenBSD firewall was assigned the public IP address, things were fine, as I could remotely access the firewall. However, if the DSL modem configuration did get reset, I needed to be able to get access to configure the DSL modem remotely, otherwise the VPN wouldn't work.

For about the first 10 years of this arrangement, if there were problems with the DSL modem configuration, I would just wait until I was on-site and could take care of it. However, when it happened in 2015, I decided to try to setup a way to remotely fix things without having to come on-site, using SSH. It turns out that things were mostly smooth after that, with the DSL modem configuration not resetting for quite a few years. However, earlier this month, the DSL modem configuration was reset, and I wasn't going to be on-site in the near future, so I had to remember the remote access I setup in 2015 and how to use it.

Regaining Access

It took me a while to remember how to use the remote access I had setup, since it had been over 6 years by this point. At first, I only remembered that I had setup remote access to handle this very case, but had completely forgot how to use it. However, remember that one reason I setup this VPN was for backups. I decided to look at the backups and see if there was something there that could help remind me. Thankfully, I found what I was looking for, in a file named setup_tunnel.bat (it was designed to run on a Windows system), which had the following contents:

plink -R 22022:10.1.2.1:22 -N -i reverse_ssh.ppk -P 18989 reverse_ssh@dyndns.mynetwork

First, this is using plink, which is PuTTY's command line equivalent of ssh(1). Let's break down what each of these flags do:

So the main point of this command is that the user in the other network can run a command that will connect to a computer in my network, setting up a port forward so that I can connect through that computer to the firewall on the other network. Once I have that access, I can get access to the DSL modem and modify the configuration.

This command almost worked great. There was only one, minor issue, which was that I had forgotten the password for reverse_ssh.ppk. I had a copy of the reverse_ssh.ppk file, as well as the OpenSSH private key on which it was based, but while I tried many passwords, I was unable to remember the correct one.

However, luckily I had installed also installed PuTTYgen on the other machine, which can be used to generate a new keypair. It had been quite a few years since I had used PuTTYgen, but I was able to Google to find screenshots of the PuTTYgen user interface, and was able to walk the user through generating a new keypair, saving the private key over reverse_ssh.ppk, and sending me the public key via email. I took the public key that they sent me, and modified ~reverse_ssh/.ssh/authorized_keys on the computer they were connecting to to use the new public key.

After that change, I had them try running setup_tunnel.bat again. Unfortunately, the results were still unsuccessful. I had an idea of what the problem might be, so I checked /var/log/authlog on the computer they were connecting to, and found the following lines:

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms
Connection closed by authenticating user reverse_ssh 123.55.110.49 port 52935 [preauth]

This issue comes from a change in OpenSSH 8.8 to tighten security and drop the older, less secure ssh-rsa public key algorithm. The PuTTYgen that generated the public key was from 2015, and was still using this older algorithm. So I opened up sshd_config on the computer they were connecting to, and added:

PubkeyAcceptedAlgorithms +ssh-rsa

As an aside, there was already code to limit what the reverse_ssh user could do in sshd_config:

Match User reverse_ssh
        PubkeyAuthentication yes
        ForceCommand /bin/false

Anyway, after modifying sshd_config and restarting sshd, the setup_tunnel.bat file worked, and the user could successfully remotely connect. Then I had to use the reverse tunnel to setup access to the DSL modem webpage, so I could modify the configuration. The first step was setting up ssh access to the firewall at the remote site. I was able to accomplish that by modifying ~/.ssh/config and adding a entry to connect through the reverse tunnel:

Host via_reverse_tunnel
Hostname localhost
Port 22022
TCPKeepAlive yes

This connects to 22022 on localhost, which the reverse tunnel has forwarded to port 22 on the firewall of the remote network. This worked, successfully restoring my remote access to the firewall at the remote site.

Fixing the Underlying Problem

From there I had to poke around to figure out where the DSL modem configuration website was hosted. Thankfully, one of my first guesses, 192.168.1.254, worked. I quickly added the following to the via_reverse_tunnel entry in ~/.ssh/config:

LocalForward *:8080 192.168.1.254:80

After that change, I disconnected and reconnected the SSH connection. Then I could navigate to http://localhost:8080 on that machine, and get access to the DSL modem configuration. From there it was a simple matter of enabling passthrough mode on the modem, choosing the correct passthrough machine (the OpenBSD firewall), then restarting dhcpleased via doas rcctl restart dhcpleased to pick up the public IP address from the DSL modem. Immediately after that, the Wireguard VPN started working again, fully restoring remote access.

Final Thoughts

My main lesson learned from this experience is to better document my system configurations, and test them more frequently, so that they will work when needed. I've handled the better documentation by writing this post. Time will tell if I follow through on the more frequent testing. Hopefully this post will help some other people see how SSH can be used to solve problems of this nature.