Keeping Global Deployments in Sync

We’re a fan of NATS here at Fly.io. NATS enables your server instances to communicate with each other with very little hassle - think PubSub, although there are a few different ways to push data around with NATS.

The general setup is to run 1 or more NATS servers. Then any number of clients (typically your code, perhaps running in many VMs around the world) can connect to a server instance and send/receive data.

This allows all of your servers to efficiently communicate. This really helps when you want server instances in one part of the world to know about things going on in instances somewhere else.

Playing Locally

Part of any good global deployment is the ability to play with it locally. This gives you a feel for what’s going on. Let’s try it out.

If you have Docker installed, you can run a NATS server and play with it pretty easily. Here’s a quick command to spin it up:

docker run -it \
    -p 4222:4222 -p 8222:8222 -p 6222:6222 \
    --name nats-server \
    nats:latest

You can install it via brew as well.

Once it’s up and running, one of the first things you can do is head to http://localhost:8222/ in your browser and poke around NATS metrics. That’s vaguely interesting to look at, but let’s do something fun.

Using NATS

Since we’re presumably in Laravel-land over here, let’s find a PHP library for NATS.

# https://github.com/basis-company/nats.php
composer require basis-company/nats

Once the library is installed, we can test out NATS PubSub. PubSub isn’t the ONLY way NATS can push data around, but it is the most straight-forward.

To test it out, we’ll create 2 commands - one to subscribe to a topic, and one to publish to that topic.

First we’ll create a command to subscribe and listen for messages:

php artisan make:command NatsSub --command "nats-sub"

That generates file app/Console/Commands/NatsSub.php which can contain the following to listen to topic some-topic:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class NatsSub extends Command
{
    protected $signature = 'nats-sub';

    protected $description = 'Command description';

    public function handle()
    {
        $client = new \Basis\Nats\Client;

        $client->subscribe('some-topic', function ($message) {
            printf("Data: %s\r\n", $message);
        });

        while(true) {
            $client->process();
        }

        return Command::SUCCESS;
    }
}

Then we can create a command to publish a message to that topic:

php artisan make:command NatsPub --command "nats-pub"

That generates file app/Console/Commands/NatsPub.php which can contain:

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;

class NatsPub extends Command
{
    protected $signature = 'nats-pub';

    protected $description = 'Command description';

    public function handle()
    {
        $client = new \Basis\Nats\Client;

        $client->publish('some-topic', "we published a thing");

        return Command::SUCCESS;
    }
}

To test this out, I subscribed to this topic 3 times, and then published to it once. We’ll see that each subscriber gets the message.

This means that any VM running our application can subscribe to the same topic, and each one will receive any message published to that topic.

So, we have a fun way to communicate amongst our application nodes!

NATS on Fly.io

As you’ve likely heard, Fly.io is good at running your applications all over the world. Often, it’s useful for globally distributed applications to communicate with each other (to help sync data or events).

Let’s see how to run NATS on Fly.io!

There is a handy GitHub repository that makes deploying NATS easy. It runs both an instance of a nats-server and makes some metrics/health checks available for Prometheus to read.

Here’s how to run a NATS cluster in 3 regions:

# Clone the repo
git clone https://github.com/fly-apps/nats-cluster.git
cd nats-cluster

# Create a fly app.
# I chose dfw as the first region
# I named my app "fid-nat"
fly launch --no-deploy

# Init a deploy
fly deploy

# Add 2 regions
# e.g. `fly regions add lhr nrt`
fly regions add <regions>

# Scale up amongst the 3 regions
## Using "--max-per-region=1" ensures each region 
## gets one VM, otherwise 2 may go into a single region
fly scale count 3 --max-per-region=1

The fly.toml file may contain some health checks in the [services] section. If so, delete the entire [services] section as we won’t need them.

Let’s test this cluster out. Assuming you have setup a Fly VPN connection (or an instance you can SSH into), we can run something like the following.

I first tested by installing nats, a CLI tool for making requests against a NATS server. I ran the following on my Mac (while connected to my Fly private network via VPN):

# Install nats
brew tap nats-io/nats-tools
brew install nats-io/nats-tools/nats

# Connect to our NATS servers
# This creates file ~/.config/nats/context/fid-nat.demo.json
nats context add fid-nat.demo --server fid-nat.internal:4222 --description "Fideloper's Cluster" --select

# In one window
nats sub fid-nat.demo

# In another window
nats pub fid-nat.demo "Hello World"

That worked! At least, it did for me. I assume it will for you also!

Let’s see how to use this in a Laravel application also deployed to Fly.io.

Using Our Cluster

We saw our application run locally. If we push it up to Fly.io (as a separate application, not the fid-nat app), we can do the same thing but just tweak the host that we connect to. The default in the PHP library is localhost. We’ll instead connect to our application’s nearest NATS instance via hostname fid-nat.internal. More on Fly internal hostnames here.

Our code can be updated to connect to the .internal hostname pretty easily:

$config = new \Basis\Nats\Configuration([
    'host' => 'fid-nat.internal',
]);
$client = new \Basis\Nats\Client($config);

If you deploy that code (or run it locally while connected to Fly.io via VPN) it still works!

What Else Might We Do?

The main benefit of NATS comes from its ability to “cluster” itself. If you spin up a few instances, the nats-server instances will become aware of each other and ensure each can be used to pass data to any connected clients.

In addition to PubSub, there’s also a way to stream data (JetStream), complete a “request-response” style round-trip, keep track of data via key-value store, and more. Check out the README in the composer package we used above to see how those are used.

Whether using a multi-region deployment or not, the possibilities for how NATS can be useful for you are endless-ish!