Many people will tell you that running your own email server on the internet is crazy, and that the likely result is the email you send will end up in the recipient's spam folder if it is delivered at all. They aren't wrong, running your own email server on the internet is up there with rolling your own crypto in the list of technical things you should never do. While I'm all about defying established convention, there's a twist to this this story that makes it less crazy than it sounds. Let's go on a journey for the reasons behind the change, alternatives I considered, why I chose to run my own email server, and how I configured it.
Prior to running my own email server, I had been using a service called Tuffmail for my email. There were two reasons for this. First, they were one of the cheapest email service providers, costing $24/year. Second, they allowed an unlimited number of email aliases for accounts. I fancy myself a frugal individual, so the low cost appealed to me. I also make heavy use of email aliases. Almost every online service I interact with gets their own email address in my domain. This allows me to easily see which online services have sold the email address to another party, or had their service hacked and had the email address leaked. Setting up the aliases did require logging into Tuffmail's website and clicking around, but generally took only a couple minutes per alias. I currently have about 200 email aliases, though I would guess at least a third I could safely delete.
In terms of interacting with Tuffmail, I used secure POP3 to download received emails, and secure SMTP on the submission port (587) to send emails. Occassionally I used Tuffmail's web interface if I was worried that a message I expected to receive did not arrive, as that allowed me to check the spam folder (POP3 doesn't support that, it only supports downloading messages from the inbox). It was very rare for Tuffmail to have false positives for spam, as I tended to use conservative spam settings.
I had been using Tuffmail for over 10 years, and was quite happy with their service. Unfortunately, earlier this month, Tuffmail sent an email to all of their customers stating they they were ceasing operation on January 1, 2022. So I knew I had to migrate off Tuffmail, and had to decide where to move instead. I'm not one to procrastinate, so I started looking for alternatives quickly.
Tuffmail helpfully included a few alternatives in their email announcing the ceasing of operations, including Fastmail, Proton Mail, Runbox, Greatmail, MX Route, Zoho, Google, and Microsoft. So I looked into each of these options:
So of these options, Fastmail was the only service that looked like it would meet my needs. If Runbox allowed for hundreds of email aliases, I would definitely have considered it. As I mentioned, I'm frugal, and $60/year just to send and receive email was more than I wanted to spend. At this point, I considered running my own email server in the cloud.
As I mentioned ealier, running your own email server is kind of crazy, but the actual reason it is crazy is that your email deliverability is probably going to suck. Most people use large email providers such as Gmail, and most large email providers treat email from non-large email providers as more likely to be spam. This is unfortunate but not unexpected, as statistically speaking, such email is more likely to be spam.
This led me to a interesting observation. Running your own email server to receive email is probably fine. You only need to use a large email service to send email. There is no requirement that your service for sending email be the same as the service for receiving email. I could easily setup a virtual machine (VM) in the cloud to receive email, and then find a separate service for sending email.
So my next step was to find a service I could use for sending email. Thankfully, I found one quickly. Sendgrid is a well known service for sending email, and it turns out, they have a free plan that allows for sending 100 emails per day. I read well over 100 emails a day, but I probably only send about 100 emails per month, so Sendgrid's free plan sounded like it would work for me.
So the next step was deciding on a cloud service for hosting a VM to receive email. First, I knew I would be using OpenBSD for this VM. OpenBSD well known for its strong security-focus. It also ships with a mail server (smtpd) with a good security record for my configuration (delivery to maildir). It's what I'm most familiar with, since I use it as an email server in my home network. For full disclosure, I'm also an OpenBSD developer, though I mostly stick with maintaining ports related to Ruby.
So I needed to make sure the cloud provider I selected supported OpenBSD. In terms of performance, my needs were very minimal. A OpenBSD VM just to receive email takes almost no resources. You could probably run it on 64MB of RAM if you turned off kernel and library relinking, and it is hard to find a cloud provider that will let you create a VM with less than 512MB of RAM.
I first considered Amazon Web Services (AWS). AWS supports very small VMs with 512MB of RAM. It looks like the best deal for me would have been a t3a.nano VM, which if you reserve for 3 years, ends up being about $17/year. You need to pay separately for storage, but an OpenBSD installation is very light on storage. It's definitely possible to run with 1GB of storage, and if you want to be safe, maybe 2GB. With 2GB, that would bring the total cost to about $19/year. AWS also gives you the equivalent of a full year of free service for new accounts, so it wouldn't cost anything right away.
I was definitely considering AWS, but the main reason I decided against it is they do not officially support OpenBSD. There are tools you can use to build your own OpenBSD AMI file to use with AWS, but I didn't feel like running something unsupported, at least if I could find a reasonable alternative.
I searched for openbsd cloud and the first thing that popped up was Vultr. They officially support OpenBSD in the cloud and make it very easy to use. The price for Vultr's smallest VM was more expensive than the price for the 3-year reserved t3a.nano instance at Amazon. When I first looked at their service, the smallest VM that I saw they supported was $60/year for 1GB of RAM and 25GB of disk. However, I found that they also offered $50 in credit for new signups, which was almost a full year for the smallest server. I decided to go with Vultr. It was the same cost as Fastmail, but I knew that running my own server would make things like alias creation easier, since I could make it scriptable. Plus, if in the future I could benefit from a VM in the cloud, I would already have one available.
Setting up an OpenBSD VM on Vultr was easy. I was lazy and didn't choose to upload my own ISO, I just picked OpenBSD from their available list of operating systems, then picked the most current version (6.9). Installation took a couple minutes, and I was given the root password and the IP address. From there I sshed into the VM as root and started setting things up.
I first changed the root password using passwd. Next, I setup a user named jeremy for myself using adduser, since I didn't want to be running as root. Then I gave that user access to operate as root using doas (similar to sudo, but simpler). I edited the /etc/doas.conf file:
permit persist keepenv :wheel
permit nopass keepenv root
I also added my public key to /home/jeremy/.ssh/authorized_keys. Then I updated the /etc/ssh/sshd_config file to disallow password authentication and the ability for root to login (leaving other lines in the file alone):
PermitRootLogin no
PasswordAuthentication no
Then I started working on configuring the mail server. To ensure the receiving email server supported STARTTLS for encrypted receipt of email, I followed the instructions in starttls(8):
# openssl genrsa -out /etc/ssl/private/vm.jeremyevans.net.key 4096
# openssl req -x509 -new -key /etc/ssl/private/vm.jeremyevans.net.key \
-out /etc/ssl/vm.jeremyevans.net.crt -days 10395
Yes, I actually did setup a self-signed certificate that expires in 30 years. Unlike browsers, servers that send email don't generally care if the SSL certificate for the receiving server is self-signed with a long expiration date, since even that is better than sending the email in plain text, which is what the sending server would do if it didn't accept the SSL certificate. If this changes I can always look into getting a certificate signed by a trusted certificate authority.
I edited /etc/mail/aliases so that mail for root would go to jeremy instead (leaving other lines in this file alone):
root: jeremy
I already had a list of all supported email aliases in a text file on my workstation, with one email per line, similar to:
code@jeremyevans.net
It was easy to convert this to a file that smtpd could use, then upload that to the VM:
$ sed 's/$/ jeremy/' < emails-file > aliases-file
$ scp aliases-file vm:emails
Then I edited the /etc/smtpd.conf file on the VM:
pki vm.jeremyevans.net cert "/etc/ssl/vm.jeremyevans.net.crt"
pki vm.jeremyevans.net key "/etc/ssl/private/vm.jeremyevans.net.key"
table local_aliases file:/etc/mail/aliases
table allowed_addresses file:/home/jeremy/emails
listen on lo0 tls pki vm.jeremyevans.net
listen on vio0 tls pki vm.jeremyevans.net
action from_remote_delivery maildir "/home/jeremy/Maildir/Inbox" user jeremy virtual <allowed_addresses>
action from_local_delivery maildir "/home/jeremy/Maildir/Inbox" alias <local_aliases>
match from any for domain "jeremyevans.net" action from_remote_delivery
match from local action from_local_delivery
This configuration listens on both the loopback address (lo0) and the network interface (vio0), and supports STARTTLS on both (though it isn't needed on the loopback). For emails received on loopback, those use the local aliases file (/etc/mail/aliases). For emails received on the network interface, it uses the list of aliases I uploaded, with all of the aliases pointing to jeremy. In both cases, all email is delivered to a single maildir, with a separate file per email. smtpd will create the maildir for me, with the correct ownership and permissions.
For security, I wanted to make sure that only the SSH and SMTP ports were open. I also wanted to allow ICMP so I could ping the server. So I edited /etc/pf.conf:
if = "vio0"
set skip on lo
block return # block by default
pass in on $if inet proto icmp
pass in on $if inet proto tcp to port {22, 25}
pass out on $if
If I cared more I would probably have setup egress filtering, but I didn't care enough to do that.
Most of the daemons that OpenBSD runs by default are helpful, but in this case, I didn't need to support sound or IPv6, so I disabled the related daemons using rcctl:
# rcctl disable slaacd sndiod
At this point, I remembered it's a good idea to apply the latest security patches using syspatch, so I did that:
# syspatch
# reboot
I then tested with telnet and made sure I could receive an email, and it went through fine, with the email file showing up in /home/jeremy/Maildir/Inbox/new in the VM. Now I needed to get that email file in my local machine, which uses a similar maildir setup. I use mutt for reading the email directly from the maildir. With Tuffmail, I was using secure POP3 to download the email, but I didn't have any experience setting up a POP3 server. However, I determined that wasn't necessary to do that. Since I control both the server and the client, I can just use rsync the files.
Since openrsync would require additional setup, I decided to install rsync on the VM:
# pkg_add rsync
A basic transfer using rsync was fine:
/usr/local/bin/rsync -rt --remove-source-files vm:Maildir/Inbox/new/ ~/Maildir/Inbox/new/
However, this needed to be automated. I had a cron job that ran every 5 minutes to download email from Tuffmail via POP3, and I needed something similar for downloading email from the VM via rsync. However, I don't want the cron job to use my ssh-agent, and therefore decided to setup a separate passwordless SSH key for this using ssh-keygen, and add it to the /home/jeremy/.ssh/authorized_keys file on the VM:
ssh-keygen -t ed25519 -f ~/.ssh/vm-emails_ed25519
cat ~/.ssh/vm-emails_ed25519.pub | ssh vm "sh -c cat >> .ssh/authorized_keys"
Then I updated my ~/.ssh/config file to use this dedicated passwordless key:
Host vm-emails
Hostname vm.jeremyevans.net
IdentitiesOnly yes
IdentityFile ~/.ssh/vm-emails_ed25519
The script to download emails from the VM only changed slightly for the new Host entry:
/usr/local/bin/rsync -rt --remove-source-files vm-emails:Maildir/Inbox/new/ ~/Maildir/Inbox/new/
Then I ran crontab -e to add the cron job:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/jeremy/bin/download-received-emails
This works well, but it would allow the use of the dedicated passwordless SSH key for general server login, which is a bad idea from a security perspective. So I wanted to lock that down. I found a good tutorial on how restrict SSH key usage for rsync, and then added this to the start of the appropriate entry in /home/jeremy/.ssh/authorized_keys in the VM:
command="rsync --server --sender -tre.iLfxCIvu --remove-source-files . Maildir/Inbox/new/"
So with all that setup, every 5 minutes my workstation downloads new emails from the VM using rsync over SSH, which I then can read with mutt.
While this was going on, I was still receiving email via Tuffmail, since I hadn't changed my DNS MX record. After a few more tests, I decided to switch my MX record to point to the VM, and things basically continued to work. I immediately tested manually from Gmail and saw the email go through the VM. Within an hour (the TTL for the MX record), all of the incoming email had switched from Tuffmail to the VM.
It's always important to think about backup, so I setup a backup configuration similar to what I use on my home machines. I created an /etc/backup_list file, with paths of the files and folders I want to backup. I ony needed to add things here that I modified after the VM installed, so the file is small:
etc/adduser.conf
etc/backup_list
etc/doas.conf
etc/group
etc/localtime
etc/mail/aliases
etc/mail/smtpd.conf
etc/master.passwd
etc/passwd
etc/pwd.db
etc/pf.conf
etc/pkg_info
etc/rc.conf.local
etc/ssh
etc/ssl/private/vm.jeremyevans.net.key
etc/ssl/vm.jeremyevans.net.crt
home/jeremy/.ssh
home/jeremy/emails
root/.ssh
var/cron/tabs/root
I have a make_backup_tarball script that I use to create a backup file:
BACKUP_LIST=/etc/backup_list
TAR_FILEPATH=/home/jeremy/vm.tar.gz
pkg_info > /etc/pkg_info
tar zcpf $TAR_FILEPATH -C / -I $BACKUP_LIST
chmod 440 $TAR_FILEPATH
chown jeremy:jeremy $TAR_FILEPATH
Setting up Sendgrid was easy. I signed up for a free account, then walked through the process of verifying my domain, which involved adding 3 DNS CNAME records. After that, I created an Sendgrid API key with only Mail Send permissions.
To switch from using Tuffmail to using Sendgrid to send emails, in /etc/mail/smtpd.conf in my local network, I switched:
table secrets file:/etc/mail/secrets
action relay_mail relay host smtp+tls://tuffmail@smtp.mxes.net:587 auth
to:
table secrets file:/etc/mail/secrets
action relay_mail relay host smtp+tls://sendgrid@smtp.sendgrid.net:587 auth
And updated the /etc/mail/secrets file for the change:
sendgrid apikey:***API_KEY_HERE***
Then I tested sending an email to my Gmail account, and checked that it was correctly send through Sendgrid.
While writing this post, I found that Vultr actually offers a cheaper VM. Instead of $60/year for 1GB of RAM and 25GB of storage, you can spend $42/year for 512MB of RAM and 10GB of storage. There's also a $30/year VM for 512MB of RAM and 10GB of storage, if you are willing to switch to IPv6 only. However, I actually want to receive mail, so I cannot use the IPv6-only option. The reason I didn't see this originally is this VM type wasn't available in the Silicon Valley datacenter I picked. I would have to switch from Silicon Valley to New Jersey to get the cheaper VM.
Again, I'm a frugal guy, so saving $18/year seems like a good idea to me. Vultr doesn't let you downgrade VMs, so I spun up a new VM with the $42/year plan, copied the the backup I had already created, than ran the following commands to restore the configuration onto the cheaper VM:
# tar zxpf vm.tar.gz -C /
# pwd_mkdb /etc/master.passwd
# pkg_add rsync
# syspatch
# reboot
The latency when working on the $42/year VM hosted in New Jersey was noticeably worse than the $60/year VM hosted in Silicon Valley. However, if you are also frugal, I'm sure you'll agree that's a small price to pay to save $18/year.
After checking that the $42/year VM worked, I switched the DNS A record to point to it instead of the $60/year VM. After the DNS A record TTL expired, I checked and made sure there was no received email on the $60/year VM, then I turned off and deleted it. I still have the $50 in Vultr credit, which will now last me about 14 months instead of 10. After the free period expires, this will still be more expensive than the $24/year I was paying previously, but it allows me to write a small script to add an email alias, instead of forcing me to login and mess with a web application. Hopefully prices for VMs will continue to fall, though I expect we will soon be at a point where the cost of the IPv4 address is the majority of the cost of a small VM.
I think this bifurcated setup with a cloud VM receiving email and using Sendgrid's free plan for sending email is the cheapest and simplest option that meets my uncommon needs. It is still more expensive than what I was paying Tuffmail ($42/year instead of $24/year), but I suppose I'll manage. I am not currently doing any spam filtering on received email, which results in more spam getting through. That doesn't bother me much, since I can almost always tell what is spam before I open it in mutt, and I just delete spam before opening it. On the plus side, I no longer have to be worried about email being sent to me not arriving because it was marked as spam. Sendgrid is also a larger email provider than Tuffmail, so I'm guessing my email delivery is better, though that doesn't matter much as I don't send much email.
I'm very glad I posted this, as I got multiple good ideas for how to improve this setup from others.
@mmonerau on Twitter asked about SPF, DKIM, and DMARC. At least two of the DNS CNAME records that Sendgrid has you setup are related to DKIM. I added an SPF record to my domain, which is a DNS TXT record:
v=spf1 include:u22788749.wl250.sendgrid.net -all
The u22788749.wl250.sendgrid.net value for the include comes from the other DNS record that Sendgrid has you add.
I looked into DMARC, but it doesn't seem like I'd get a lot of value from it, so I didn't bother setting it up.
This post spawned some interesting discussion on Lobsters. Fellow OpenBSD committer bentley@ described how he uses a similar setup. However, he uses a Wireguard VPN between his Cloud VM at Vultr and his home email system. Then he has the Cloud VM smtpd configuration relay mail to his home email system over the VPN. Using this approach, you don't need to run a cron job every 5 minutes as I was doing to download new email. It would arrive in mutt on my local machine shortly after it is received by the VM. So I decided to try that approach.
The first step is to setup Wireguard on the VM and on the local machine. I hadn't used Wireguard before, but looking at the OpenBSD documentation for wg, it was quite easy to setup. On the VM, I added an /etc/hostname.wg0 file:
wgport 7112 wgkey $VM_PRIVATE_KEY wgpeer $LOCAL_PUBLIC_KEY wgendpoint $LOCAL_IP 7111 wgaip 10.71.11.2/32 10.71.11.1/24
I added a similar /etc/hostname.wg0 file on my local machine that runs mutt:
wgport 7111 wgkey $LOCAL_PRIVATE_KEY wgpeer $VM_PUBLIC_KEY wgendpoint $VM_IP 7112 wgaip 10.71.11.1/32 10.71.11.2/24
Then I ran sh /etc/netstart wg0 on both to start the Wireguard interface. It didn't work at first, due to the firewall rules on both the local machine and the VM. On the VM, I added the following firewall rules to the end of /etc/pf.conf:
pass in on $if inet proto udp from $LOCAL_IP port 7111 to port 7112
pass on wg0 inet proto icmp
pass out on wg0 inet proto tcp to 10.71.11.2 port 25
The first rule allows the encrypted Wireguard traffic over the network interface. Wireguard runs over UDP, and the 7111 and 7112 come from the wireguard configuration above. The second rule allows ICMP in both directions over the Wireguard interface. The third rule allows the VM to send email over the Wireguard interface. The block return is still above the other firewall rules, so all other traffic over the Wireguard interface is blocked.
I then added similar firewall rules on the local machine:
pass in on $if inet proto udp from $VM_IP port 7112 to ($if) port 7111
pass on wg0 inet proto icmp
pass in on wg0 inet proto tcp from 10.71.11.1 to 10.71.11.2 port 25
Again, these rules are quite restrictive. The encrypted Wireguard traffic is allowed over the network interface. On the Wireguard interface, ICMP is allowed in both directions, and the VM can send email to the local machine. All other traffic is blocked. That mitigates the possible damage the VM can do to the local machine if the VM is compromised.
Next, I had to change the mail server configuration on both the local machine and the VM. On the local machine, I added the following rules to /etc/mail/smtpd.conf:
listen on 10.71.11.2 tls-require pki mail.jeremyevans.local
table emails file:/etc/mail/emails
action from_remote_delivery maildir "/home/jeremy/Maildir/Inbox" user jeremy virtual
match from src 10.71.11.1 for domain "jeremyevans.net" action from_remote_delivery
match from src 10.71.11.1 for domain "vm.jeremyevans.net" action local_delivery
The first line sets up a listening socket on the Wireguard interface to receive email. The second line sets up a table for the emails. This is the same as the ~/emails file on the VM, with a mapping of the allowed email addresses to the jeremy account. The third line sets an action to deliver to a maildir, using the same that the VM uses. The fourth line allows for receiving mail for my domain on the wireguard interface, using that action. The fifth line is for handling local emails on the VM itself. This uses a local_delivery action that is similar to the from_local_delivery action I showed earlier in the VM's smtpd configuration.
I then needed to modify the VM's smtpd configuration. I commented out the existing action and match lines, and added the following lines:
action relay_to_local relay host smtp+tls://10.71.11.2 tls no-verify
match from any for rcpt-to <allowed_addresses> action relay_to_local
match from local action relay_to_local
With the first line, we are setting up a relying action to the smtp server on the Wireguard interface of the local machine. We are requiring TLS, but not verification, since my local machine's smtpd configuration uses a self signed certificate, similar to how the VM is setup. Because this goes over Wireguard, it's already encrypted, and doesn't need TLS for security. However, I think it's a good practice to use TLS anyway.
I need to keep the ability to only accept email for valid email addresses. Since you cannot do that in a relay action, the match rule in the second line changes from accepting for the entire domain to only accepting from a table of addresses. This is similar to the virtual table used in the previous configuration, except it only needs the addresses. On my local machine, I copied up the emails-file directly to the VM, instead of the aliases-file I created using sed.
scp emails-file vm:emails
With the third line, we are relaying all local mail on the VM to the local machine.
After setting that up, I restarted smtpd on both the local machine and the VM using rcctl restart smtpd. Then I tested a local email on the VM, which worked fine. Then I tested an emailing from Gmail to my domain, which also worked. This took significantly more setup, but is less clunky, and results in faster mail delivery.
After successful testing, I commented out the cron job on the local machine that downloaded the emails, since it was not needed any longer. I didn't delete it completely, because it may be useful in the future to temporarily switch back to the configuration that does not relay email, such as if there is an extended outage of the local machine and I need to store the email on the VM until the outage is resolved. I also added etc/hostname.wg0 to the etc/backup_list file on both the local machine and the VM, to make sure the Wireguard configuration is backed up.
bentley@ also mentioned setting up MTA-STS, to force the use of TLS when transferring mail. I looked into this, but it doesn't work with self signed certificates on the MX, and I don't want to be bothered to setup acme-client and run httpd on the VM. I consider the risk added by those, while very small, to be still higher than the risk of someone attempting a MITM attack on my email server.