SSH and User-mode IP WireGuard

But Fly is kind of an odd duck. We run hardware in data centers around the world, connected to the Internet via Anycast and to each other with a WireGuard mesh. We take Docker-type containers from users and transmogrify them into Firecracker micro-VMs. And, when we first got started, we did all this stuff so that our customers could run “edge applications”; generally, relatively small, self-contained bits of code that are especially sensitive to network performance, and that need to be run close to users. In that environment, being able to SSH into an app is not that important.

But that’s not all people use Fly for now. Today, you can easily run your whole app on Fly. We’ve made it easy to run ensembles of services, in cluster configurations, that can talk to each other privately, store data persistently, and talk to their operators over WireGuard. And, if I keep on writing like this, I can probably tag every blog post we’ve written in the last couple months.

Anyways, we didn’t have an SSH feature.

Now, of course, you could just build a container that ran an SSH service, and then SSH into it. Fly supports raw TCP (and UDP) networking; if you told our Anycast network about your weird SSH port in your fly.toml, we’d route SSH connections to you, and that’d work just fine.

But that’s not how people want to build containers, and we’re not asking them to. So, we built an SSH feature. It is wacky. I am here to describe it, in two parts.

Part The First: 6PN and Hallpass

I’ve written a bunch about private networking at Fly. Long story short: it’s like a simpler, IPv6 version of GCP or AWS “Virtual Private Clouds”; we call it “6PN”. When an app instance (a Firecracker micro-VM) is started at Fly, we assign it a special IPv6 prefix; the prefix encodes the app’s ID, the ID of its organization, and an identifier for the Fly hardware it’s running on. We use a tiny bit of eBPF code to statically route those IPv6 packets along our internal WireGuard mesh, and to make sure that customers can’t hop into different organizations.

Further, you can bridge the private IPv6 networks we create with other networks, using WireGuard. Our API will mint new WireGuard configurations, and you can stick them on an EC2 host to proxy RDS Postgres. Or, if you like, use the Windows, Linux, or macOS WireGuard client to connect your development machine to your private network.

You can probably see where this is going.

We wrote a teeny, tiny, trivial SSH server in Go, called Hallpass. It’s practically the “hello world” of Go’s x/crypto/ssh library. (If I was doing it over again, I’d probably just use the Gliderlabs SSH server library, where it would literally be “hello world”). Our Firecracker “init” starts Hallpass on all instances, bound to the instance’s 6PN address.

If you can talk to your organization’s 6PN network (say, over a WireGuard connection), you can log into your instances with Hallpass.

There’s only one interesting thing about how Hallpass works, and that’s authentication. The infrastructure in our production network doesn’t generally have direct access to our API or the databases that back it, nor, of course, do the instances themselves. This makes communicating configuration changes — like “what keys are allowed to log into instances” — a bit of a project.

We sidestepped that work by using SSH client certificates. Rather than propagating keys every time a user wants to log in from a new host, we establish a one-time root certificate for their organization; the public key for that root certificate is hosted in our private DNS, and Hallpass consults the DNS to resolve the certificate every time it gets a login attempt. Our API signs new certificates for users they can use to log in.

You probably have questions.

First: certificates. Decades of X.509 madness have probably left a bad taste in your mouth for “certificates”. I don’t blame you. But you should be using certificates for SSH, because they are great. SSH certificates aren’t X.509; they’re OpenSSH’s own format, and there isn’t much to them. Like all certificates, they have expiration dates, so you can create short-lived keys (which is almost always what you want). And, of course, they allow you to provision a single public key, on a whole bunch of servers, that can authorize an arbitrary number of private keys, without repeatedly updating those servers.

Next: our API, and signing certificates. Welp! We’re pretty careful, but it’s basically as secure as your Fly access token is; it couldn’t, right now, be any more secure than that, because your access token allows you to deploy new versions of your app container. There’s a lot of ceremony involved with WebPKI X.509 CA signing; this isn’t that, and we’re pretty unceremonious.

Finally, the DNS. This, I concede, seems batshit. But it is less batshit than it seems! Every host we run instances on runs a local version of our private DNS server (a small Rust program). eBPF code ensures that you can only talk to that DNS server (technically: you can only make queries of the private DNS API of the server; it’ll recurse for anybody) from the 6PN address of your server. The DNS server can — I know this is weird — reliably discern your organization identity from source IP addresses. So that’s what we do.

All this stuff is happening behind the scenes. What you, as a user, saw was the command flyctl ssh issue -a, which would grab a new certificate from our API and insert it into your local SSH agent, at which point SSH would just sort of work. It’s been neat.

But: things can always be neater.

Part The Second: User-Mode TCP/IP WireGuard

Here’s a problem: not everyone has WireGuard set up. They should; WireGuard is great, and it’s super useful for managing applications running on Fly. But, whatever, some people don’t.

They’d still like to SSH into their apps.

At first glance, not having WireGuard installed seems like a dealbreaker. The way WireGuard works is, you get a new network interface on your machine, either a kernel-mode WireGuard interface (on Linux) or a tunnel device with a userland WireGuard service attached (everywhere else). Without that network interface, you can’t talk over WireGuard.

But if you squint at WireGuard and tilt your head the right way, you can see that’s not technically true. You need operating system privileges to configure a new network interface. But you don’t need any privileges to send packets to 51820/udp. You can run the whole WireGuard protocol as an unprivileged userland process – that’s how wireguard-go works.

That gets you as far as handshaking WireGuard. But you’re still not talking over a WireGuard network, because you can’t just send random strings to the other end of a WireGuard connection; your peer expects TCP/IP packets. The native socket interface on your machine won’t help you establish a TCP connection over a random connected UDP socket.

How hard could it be to put together a tiny user-mode TCP, just for the purposes of doing pure-userland WireGuard networking, so people could SSH into instances on Fly without installing WireGuard?

I made the mistake of musing about this on a Slack channel I share with Jason Donenfeld. I mused about it just before I went to bed. I woke up. Jason had implemented it, using gVisor, and made it part of the WireGuard library.

The trick here is gVisor. We’ve written about it before. For those who aren’t familiar, gVisor is Linux, running in userland, reimplemented in Golang, as a container runc replacement. It is a pants-are-shirts bananas crazy project and, if you run it, I think you should brag about it, because wow is it a lot. And, buried in its guts is a complete TCP/IP implementation, written in Go, whose inputs and outputs are just []byte buffers.

Some tweets were twote, and, a couple hours later, I got a friendly email from Ben Burkert. Ben had done some gVisor networking work elsewhere, was interested in what we were working on, and wanted to know if we wanted to collaborate. Sounded good to us! And, long story short, we now have an implementation of certificate-based SSH, running over gVisor user-mode TCP/IP, running over userland wireguard-go, built into flyctl.

To use it, you just use flyctl to ssh:

flyctl ssh shell personal dogmatic-potato-342.internal

To give you some perspective on how bananas this is: dogmatic-potato-342.internal is an internal DNS name, resolving only over private DNS on 6PN networks. It works here because, in ssh shell mode, flyctl is using gVisor’s user-mode TCP/IP stack. But gVisor isn’t providing the DNS lookup code! That’s just the Go standard library, which has been hornswoggled into using our bunko TCP/IP interface.

Flyctl, by the way, is open source (it has to be; people have to run it on their dev machines) so you can just go read the code. The good code, under pkg, Ben wrote. The nightmare code, elsewhere, is me. User-mode IP WireGuard is astonishingly straightforward in Go. Like, if you’ve done low-level TCP/IP work before: you probably won’t believe how simple it is: the objects gVisor’s TCP stack code hand back plug directly into the standard library’s network code.

Here, let’s take a look:

tunDev, gNet, err := netstack.CreateNetTUN(localIPs, []net.IP{dnsIP}, mtu)
if err != nil {
    return nil, err
}

// ...

wgDev := device.NewDevice(tunDev, device.NewLogger(cfg.LogLevel, "(fly-ssh) "))

CreateNetTUN is part of wireguard-go; it packages up gVisor and gives us (1) a sythetic tun device we can use to read and write raw packets to drive WireGuard, and (2) a net.Dialer that wraps gVisor that we can drop into Go code to use that WireGuard network.

And that’s… it? I mean, here’s us using it for DNS:

resolv: &net.Resolver{
    PreferGo: true,
    Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
        return gNet.DialContext(ctx, network, net.JoinHostPort(dnsIP.String(), "53"))
    },
},

It’s just normal Go networking code. Bananas.

So Obviously Everybody Should Be Doing This

For a couple hundred lines of code (not counting the entire user-mode Linux you’ll be pulling in from gVisor, HEY! Dependencies! What are you gonna do!) you can bring up a new, cryptographically authenticated network, any time you want to, in practically any program.

Granted, it’s substantially slower than kernel TCP/IP. But how often does that really matter? And, in particular, how often does it matter for the random tasks that you’d ordinarily build a weird, clanking TLS tunnel for? When it starts to matter, you can just swap in real WireGuard.

At any rate, it solves a huge problem for us. It’s not just SSH; we also host Postgres databases, and it’ll be super handy to be able to bring up a psql shell anywhere, regardless of whether you can install macOS WireGuard right that second, with a simple command.

Also Ben Burkert is fantastic and you should all bid his consulting rates up, because he got this project done in what seemed like a few hours.

I would write more gracefully about him, and about Jason, and about gVisor, and about this whole project, except that Tailscale tweeted this morning about a gVisor TCP/IP feature they’re going to roll out, and I won’t be beaten to the punch. Not this time, Tailscale! Not this time!