Private Networking

Fly Apps are connected by a mesh of WireGuard tunnels using IPv6.

Applications within the same organization are assigned special addresses (6PN addresses) tied to the organization. Those applications can talk to each other because of their 6PN addresses, but applications from other organizations can’t. The Fly.io platform won’t forward packets between different 6PN networks.

Private networking is always available to apps by default; you don’t have to do anything special to get it.

You can connect apps running outside of Fly.io to your 6PN network using WireGuard. You can even connect your dev laptop to your 6PN network. To do that, you’ll use flyctl, the Fly.io CLI, to generate a WireGuard configuration that has a 6PN address.

Fly.io .internal DNS

A Fly Machine is configured to resolve domain names with a custom DNS server from the Fly Platform. This DNS server can resolve arbitrary DNS queries, so you can look up google.com with it. But it’s also aware of 6PN addresses, and will let you look up 6PN addresses for other apps in your organization. Those addresses live under the custom top-level domain .internal. You might want to use .internal domains to connect your app to databases, API servers, or other apps in your 6PN network.

Underneath .internal there are second-level domains for every app in your Fly organization. For example, if your app is in an organization with another app called my-app-name, then there will be a AAAA record at my-app-name.internal. The AAAA record will contain all the 6PN addresses of the started Fly Machines that make up the my-app-name Fly App. Note that different libraries and tools will use multi-address AAAA records differently; most will only use the first address that is returned, but others might round-robin between entries for every request – if you’d like to know more, consult the documentation for the library or tool you are using for DNS lookup.

Each <appname>.internal domain has further subdomains which can be used to return a more precise subset of the started Machines in that app. For example, you can add a region name qualifier to return the 6PN addresses of an app’s Machines in a specific region: iad.my-app-name.internal. Querying this domain returns the 6PN addresses of my-app-name Machines in the iad region.

Some .internal domains do not contain an AAAA record, but instead contain a TXT record with Machine, app, or region information. For example, if you request the TXT records using regions.my-app-name.internal, then you’ll get back a comma-separated list of regions that my-app-name is deployed in. And you can discover all the apps in the organization by requesting the TXT records associated with _apps.internal. This will return a comma-separated list of the app names.

The following table is a complete list of the available .internal domains:

Examples of retrieving this information are in the fly-examples/privatenet repository.

Listening for connections on 6PN addresses

When deploying a Fly Machine, we alias the 6PN address of the Machine to fly-local-6pn in the Machine’s /etc/hosts file.

For a service to be accessible via its 6PN address, it needs to bind to/listen on fly-local-6pn. For example, if you have a service running on port 8080, then you need to bind it to fly-local-6pn:8080 for it to be accessible to other Machines over the 6PN network.

Learn more about connecting to app services.

Connect a Fly Machine to the Fly.io DNS

The custom Fly.io DNS server is always available at the IPv6 address fdaa::3. If you want to get fancy, you can install dig on the Machine and query the DNS directly. For example:

root@f066b83b:/# dig +short aaaa my-app-name.internal @fdaa::3

fdaa:0:18:a7b:7d:f066:b83b:2

When deploying a Fly Machine, we overwrite /etc/resolv.conf to point the DNS server to fdaa::3. Since this is the default configuration we set up for Machines on Fly.io, you probably don’t need to do anything special to make this work.

In uncommon circumstances, such as having an unusual file system layout, or using a networking stack that needs the nameserver explicitly indicated, you may need to point your application to the Fly.io DNS yourself. It’s easy: from any Fly Machine, the nameserver is always at fdaa::3.

6PN addresses in detail

In most cases, connecting to other Machines via the .internal DNS is the most convenient and accessible way to connect your Fly Apps and Machines.

Most of the time, the .internal DNS is all you’ll need for routing. If you need more complicated routing, you might be able to take advantage of the structure of 6PN addresses in your app’s design. Rather than a single address, each Fly Machine is assigned a /112 6PN subnet, which is structured as follows:

The machine identifier portion of the 6PN address is not related to the 14 character Machine ID; the two are independent. A Fly Machine’s current 6PN address can be found in the environment variable FLY_PRIVATE_IP. As noted above, a Machine’s 6PN address is not static, so do not assume that a Fly Machine’s Machine ID can be permanently mapped to a particular 6PN address. 6PN addresses will change when an app is moved into a new Fly Org, or when a Fly Machine is migrated onto a new host server. However, an 6PN address change can only happen on a reboot, so supplying a procedure to check for a change in 6PN address on Machine startup is sufficient to handle this event.

Flycast - Private Fly Proxy services

A Flycast address is an app-wide IPv6 address that the Fly Proxy can route to privately. Use a Flycast address to direct private network traffic through the Fly Proxy to take advantage of features like geographically aware load balancing and autostart/autostop based on traffic.

Use Flycast to do the following entirely within your organization’s private network:

• Autostart and/or autostop Machines based on network requests.
• Use Fly Proxy’s geographically aware load balancing for private services.
• Connect to a service from another app that can’t use DNS.
• Connect from third-party software, like a database, that doesn’t support round-robin DNS entries.
• Access specific ports or services in your app from other Fly.io organizations.
• Use advanced proxy features like TLS termination or PROXY protocol support.

The general flow for setting up Flycast is:

1. Allocate a private IPv6 address for your app on one of your Fly.io organization networks.
2. Make sure your app binds to 0.0.0.0:port. Binding to fly-local-6pn:<port> is insufficient for Flycast.
3. Expose services in your app’s fly.toml [services] or [http_service] block; do not use force_https as Flycast is HTTP-only.
4. Deploy your app.
5. Access the services on the private IP from the target organization network.

Assign a Flycast address

By default, the Flycast IP address is allocated on an app’s parent organization network.

 fly ips allocate-v6 --private

VERSION IP                  TYPE    REGION  CREATED AT
v6      fdaa:0:22b7:0:1::3  private global  just now

You can also use Flycast to expose an app in one Fly organization to another Fly organization by using the --org option when you allocate the Flycast address:

 fly ips allocate-v6 --private --org my-other-org

VERSION IP                  TYPE    REGION  CREATED AT
v6      fdaa:0:22b7:0:1::3  private global  just now

Flycast and Fly.io DNS

Flycast addresses can also be found by using the Fly.io DNS. If an app has a Flycast address allocated to it, there will be an AAAA record at <appname>.flycast.

Private Network VPN

You can use the WireGuard VPN to connect to the 6PN private network. WireGuard is a flexible and secure way to plug into each one of your Fly.io organizations and connect to any app within that organization.

Set up a private network VPN

To set up your VPN, you’ll use flyctl to generate a tunnel configuration file with private keys already embedded. Then you can load that file into your local WireGuard application to create a tunnel. Activate the tunnel and you’ll be using the internal Fly.io DNS service which resolves .internal addresses - and passes on other requests to Google’s DNS for resolution.

1. Install your WireGuard App

Visit the WireGuard site for installation options. Install the software that is appropriate for your system. Windows and macOS have apps available to install. Linux systems have packages, typically named wireguard and wireguard-tools, you should install both.

2. Create your tunnel configuration

To create your tunnel, run:

fly wireguard create

Select the organization you want the WireGuard tunnel to work with:

? Select organization:  [Use arrows to move, type to filter]
> My Org (personal)
  Test Org (test-org)

The fly wireguard create command configures the WireGuard service and generates a tunnel configuration file, complete with private keys which cannot be recovered. This configuration file will be used in the next step. First, save the configuration file:

!!!! WARNING: Output includes private key. Private keys cannot be recovered !!!!
!!!! after creating the peer; if you lose the key, you’ll need to remove    !!!!
!!!! and re-add the peering connection.                                     !!!!
? Filename to store WireGuard configuration in, or 'stdout':  mypeer.conf
Wrote WireGuard configuration to 'mypeer.conf'; load in your WireGuard client

We suggest you name your saved configuration with the same name as the peer you have created. Add the extension .conf to ensure it will be recognized by the various WireGuard apps as a configuration file for a tunnel. Note that the name (excluding the .conf extension) shouldn’t exceed 15 characters since this is the maximum length for an interface name on Linux.

3. Import your tunnel

Windows

Run the WireGuard app. Click Import tunnel(s) from file. Select your configuration file. The WireGuard app will display the details of your tunnel. Click Activate to bring the tunnel online.

macOS

Run the WireGuard app. Click Import tunnel(s) from file. Select your configuration file and click Import. You might be prompted by the OS that WireGuard would like to add VPN configurations; click Allow. The WireGuard app will display the details of your tunnel. Click Activate to bring the tunnel online.

Ubuntu Linux

If you don’t have wg-quick installed, then run the command below. For Ubuntu 18.04 to 22.04, openresolv is also required.

sudo apt install wireguard-tools openresolv

Copy the configuration file to /etc/wireguard; you’ll need root/sudo permissions:

sudo cp basic.conf /etc/wireguard

Run wg-quick to bring up the connection by name (i.e. less the .conf extension). For example:

wg-quick up mypeer

[#] ip link add mypeer type wireguard
[#] wg setconf mypeer /dev/fd/63
[#] ip -6 address add fdaa:0:4:a7b:ab6:0:a:102/120 dev mypeer
[#] ip link set mtu 1420 up dev mypeer
[#] resolvconf -a tun.mypeer -m 0 -x
[#] ip -6 route add fdaa:0:4::/48 dev mypeer

Connect to the Fly.io DNS over WireGuard

The DNS server address is different on WireGuard connections than on Machines. That’s because you can run multiple WireGuard connections; your dev laptop could be WireGuard-connected to multiple organizations, but a Machine can’t be.

Your DNS server address for a WireGuard connection is part of the WireGuard tunnel configuration that flyctl generates. Your platform WireGuard tools might read and automatically configure DNS from that configuration, or they might not. Here’s an example of a WireGuard configuration file generated by the fly wireguard create command:

[Interface]
PrivateKey = [redacted]
Address = fdaa:0:18:a7b:d6b:0:a:2/120
DNS = fdaa:0:18::3

You guessed it; it’s the DNS line. Your DNS server address will also start with fdaa:, but the next two parts are unique to your organization’s network. All 6PN addresses are prefixed by the organization’s network ID; that’s the part of the address that locks it to your organization.

All our WireGuard DNS addresses follow this pattern: take the organization prefix, and tack ::3 onto the end. In this example, the WireGuard peer address is:

fdaa:0:18:a7b:d6b:0:a:2

The 6PN prefix is the first 3 :-separated parts, so the DNS server address for this example is:

fdaa:0:18::3

To use dig to probe DNS on a WireGuard connection, supply the DNS server address to it. For example:

root@f066b83b:/# dig +short aaaa my-app-name.internal @fdaa:0:18::3

fdaa:0:18:a7b:7d:f066:b83b:2

Test the VPN tunnel

If you have the dig tool installed, a TXT query to _apps.internal will show all the apps in the organization you are connected to:

dig +noall +answer _apps.internal txt

_apps.internal.     5   IN  TXT "my-app-name,my-app-name-0,my-app-name-1"

You can also query for peer names and addresses:

dig +short txt _peer.internal @fdaa:0:18::3

"my-peer"

dig +short aaaa my-peer._peer.internal @fdaa:0:18::3

fdaa:0:18:a7b:7d:f066:b83b:102

Query for the 6PN addresses of all started Machines in an app:

dig +short aaaa my-app-name.internal

Manage WireGuard on Fly.io

List the tunnels

To list all the tunnels set up for an organization, run fly wireguard list. You can provide an organization on the command line or select an org when prompted.

Remove a tunnel

To remove a tunnel, run fly wireguard remove. You can specify the organization and tunnel name on the command line or be prompted for both.

Troubleshoot a WireGuard VPN connection

Having trouble connecting to a Fly.io hosted app? When you can’t connect to something, it’s helpful to establish a baseline of what is, or what is not, working.

Am I connected to Fly.io VPN?

When connected locally, you can run a dig command to list all the apps your connection has access to.

dig _apps.internal TXT +short

my-app,my-app-db

If results are returned, you have a VPN connection to an org at Fly.io and the results list all the app names.

If no results are returned, either you do not have a WireGuard VPN connection open or there are NO apps running in the org.

Am I connected to the right Fly.io org?

WireGuard connections are created to a specific org. Each org’s network is isolated from other orgs. You may have a VPN connection to Fly.io, but it may be to a different org than where your app is located. By default, you are working with your personal organization. If the app is in a business focused org or a shared org, you need to ensure your VPN connection is the org where the app lives.

You can also try to ping your app’s Machine.

Mac version:

ping6 my-app.internal

Linux version:

ping -6 my-app.internal

If the ping succeeds, you have a VPN connection to the same org that your application’s Machine is running in.

If the ping fails, check which org the application is deployed in.

fly status

App
  Name     = my-app
  Owner    = my-biz
  Hostname = my-app.fly.dev
  Image    = my-app:deployment-0123456789

Machines
PROCESS ID              VERSION REGION  STATE   ROLE    CHECKS  LAST UPDATED
app     90706e10f12094  10      ord     started                 2024-04-16T20:20:59Z

The Owner is the organization. In this example, it is my-biz.

Then check that your WireGuard connection is to the same organization. To be certain, you can remove the current connection and re-create it explicitly specifying the org.