There are two things that tend to create explosions of ideas in my head, and often result in significant actions. Those are boredom (like being on "vacation" for more than 24h) and a sufficient level of annoyance with something.

There are enough higher quality rants out there, so I'll spare you all the reasons, but I recently decided to start running a homelab again due to the latter.

In this post I'll give a broad overview of the architecture I settled on, and why I made specific choices. I'll try to keep things relatively high level here, and we'll dive into the implementation details in future posts.

My requirements

Let's start from the beginning with what I'm trying to achieve. I want to host multiple services, some of them internet-accessible, from a box on my desk running FreeBSD. Specifically FreeBSD, because I find it has an uncommon trio of properties:

• It's easy to administer.
• It's rock solid stable.
• It has a huge collection of up-to-date packages.

It's surprisingly rare to find all 3 in a system.

I already have the hardware (an old NUC, which I've named bsdcube even though it's not a cube), and my home internet is rock solid, but I don't have a static IP. Even if I did, I'm not sure I want to run some services on a residential IP, or to invite attacks on my home network.

Broad architecture overview

The architecture that I chose keeps bsdcube on my home network behind the NAT firewall. To make the server accessible from the internet, I use a WireGuard tunnel to a remote host that is internet-accessible. PF rules on the accessible server route certain traffic over the tunnel and block the rest.

 +----------+       +---------+                         +---------+
 | Internet |------>| proxbox |<---WireGuard-Tunnel---->| bsdcube |
 +----------+       +---------+                         +---------+

Choosing a host for proxbox

For my internet-accessible box, which I dubbed proxbox, I wanted a reliable host that supported a BSD. I was initially planning on getting a FreeBSD box for the job. I even had a box set up, as described in my post on the subject. (I guess I'll need to write a follow-up now!)

I was somewhat discouraged by the poor support of FreeBSD on most cloud providers. There is ironically a split where a lot of the mega-hosts like AWS have tier 1 support, but there's nothing in the middle. Many of the midrange hosts which had good FreeBSD VM support now require a tedious manual install process. Vultr seemed to be the only host that had at least some level of support, but they were not my first choice.

Then my friend Andrew mentioned OpenBSD Amsterdam. It was love at first sight. I had, somehow (despite being a FreeBSD using since around 2002) never actually used OpenBSD. But I loved their pitch: a community-focused host that donates a large part of every purchase to the OpenBSD foundation.

The onboarding was fast and easy. I filled out the form, and within 2 hours I had an email with connection instructions! I hadn't even been given an invoice yet! Really fantastic service.

proxbox setup using the OpenBSD base system

Now let's talk about the guts of proxbox. Surprisingly, almost everything running on this box is part of the OpenBSD base system!

                            +-------+                                                     
  /--\      +-----HTTP----->| httpd |                                                     
 /    \ ----+               |       |                                                     
 \ PF /                     +-------+               +---------+                      
  \__/  ----+               +--------+  +---blog--->| Varnish |--+   +---------------+
            |               | relayd |--+           |         |  +-->| WG -> bsdcube |
            +-----HTTPS---->|        |--+           +---------+  +-->|               |
                            +--------+  |                        |   +---------------+
                                        +-------matrix-----------+                   

It all starts with PF, the OpenBSD Packet Filter. In addition to rejecting invalid / unwanted traffic, it's responsible for routing the legit packets to the right service.

Serving HTTP(S) with httpd(8) and relayd(8)

An HTTP server sits on the box for handling the ACME challenges (e.g. from Let's Encrypt) to get a TLS cert. OpenBSD has httpd(8), not to be confused with the Apache httpd. It can serve static files and FastCGI, but it's not a reverse proxy.

Normally I'd reach for something like nginx, HAProxy, or Caddy, but OpenBSD also has an excellent (load balancing) reverse proxy in the base system already: relayd(8).

relayd in my setup is responsible for 2 things: routing to the correct backend (e.g. based on SNI), and TLS termination. I'll write a longer post about this soon, including how to configure acme-client(1)to get certs automatically, and configuring relayd to terminate TLS for multiple domains.

Varnish (Vinyl) cache

Since my blog is just a static site, it would be quite reasonable to host it with httpd on proxbox. This would result in faster response times, but I decided to host it on bsdcube to make the setup simpler. The clear separation of concerns, with bsdcube being the final source of truth for all data and services makes it easier for me to maintain. I'll never had to touch proxbox outside of routine system maintenance or adding a new service port.

That said, I'm not going to just blindly forward every single request over to my home network. I've used Varnish Cache (now rebranded to Vinyl) for over a decade in various professional contexts, and it does an excellent job at absorbing traffic spikes, and smoothing over issues when the backend goes flaky. Since I'm tunneling everything over the internet, halfway around the world, this seems like table stakes.

The Middle: WireGuard

Whatever route a legitimate packet takes, if it needs to talk to a backend service, this happens over a WireGuard tunnel.

I could have done this with something like Tailscale or Cloudflare Warp, but I wanted something that wasn't reliant on a "free" external service. Such services are rarely free forever. Rolling your own tunnel with a proven technology is surprisingly easy, and my intentional choice of a somewhat offbeat rest of the stack made the decision a no-brainer. Both FreeBSD and OpenBSD have WireGuard support built-in at the kernel level already.

PF rules on both sides of the tunnel ensure that only certain traffic passes through. I will probably write a post on this later as well as there's quite a lot here.

bsdcube

Let's look at the other side of the tunnel now. bsdcube sits on my home network behind a NAT firewall. It only accepts traffic over the tunnel for specific service ports. And those ports are mapped to jailed services with a limited view of the system. In case you're not familiar with FreeBSD, jails are a FreeBSD-native containerization technology which has been around for over 25 years (that's a long time before Docker popularized the term). While they have many differences with Linux containers, that's a reasonable mental model for understanding the rest of this post.

AppJail

Jails are native to the FreeBSD kernel, and the base system has jail(8) for managing them. But much as in the Linux world, most people rely on higher level tools for management and orchestration.

There are over a dozen such tools for FreeBSD, each with a different take on things. That's both cool and bewildering ;) I ended up settling on AppJail for bsdcube, at least for now, since it has excellent documentation, support for features I'm curious about later (like Linux jails), and a declarative, composable configuration format.

Creating jails with Makejail files

Here's a sample of a Makejail file. As you might guess, it's a set of instructions for building a jail. It looks a bit like a Dockerfile.

# blog/Makejail

# Include directives let you separate out sections of your config across files
# and reuse common sections.
INCLUDE options/options.makejail
INCLUDE options/network.makejail

# Install nginx from the FreeBSD package repository
PKG nginx

# A sysrc directive.
# This is the equivalent of enabling a service on a Linux box with systemd
SYSRC nginx_enable=YES

# Copies nginx.conf from the current directory into the specified path in the jail.
COPY nginx.conf /usr/local/etc/nginx/nginx.conf

# Install Marmite, the static site generator I use for my blog.
# Despite being relatively obscure, there's also an up-to-date package for this!
PKG marmite

INCLUDE buildsite.makejail

SERVICE nginx start

Jail lifecycle management

"Scripted" setups like this can get pretty unwieldy if you're not careful. And for me, one of the worst parts of CLI tools is remembering the half dozen switches to get a particular task done.

To solve this, I structured my jail deployments as a git repository where each subdirectory is a self-contained service. In my setup, there are no dependencies, and I intend to keep it that way, deploying the full application/service in isolated jails, even if I could theoretically have a "common" service like, say, a single Postgres database. As such, it's easy to add a justfile to each directory with deployment recipes so I don't have to remember them.

The usual invocation is something like just freshjail to (re)create a jail. And for jails like my blog, where the application has no need to restart to get an update, I have recipe(s) that perform the necessary updates against the running jail.

Conclusion (or, what's next)

That was a lot... and there's still a lot I didn't cover! Like pointers on the AppJail host setup, PF, handling multiple hosts with SNI via relayd, and a lot more. So stay tuned for the next post, and give a shout on Mastodon if you have any feedback.

In the meantime, I've uploaded the AppJail part of my setup to a public repo on Codeberg. Hopefully they are useful for others!