Multi-Region Laravel with PlanetScale

We’re going to see how to serve Laravel globally.

Fly.io is great at serving your application from servers in multiple regions. PlanetScale is great at putting your data close to your application servers!

We’re going to use PlanetScale Portals to replicate data from a main database across the world to read-replicas. Having the data close to the application servers speeds up our applications, as they don’t need to reach for a database that might be on the other side of the world.

The Setup

We’ll have 3 application instances on Fly, each in a different region. Similarly, PlanetScale will have 3 database instances. One is the primary database, and 2 will be read-replicas. Each instance will be in a different region.

PlanetScale has regions available close to Fly.io’s. When our app serves a request, we want to connect to the nearest PlanetScale database.

Our mapping of Fly.io region to PlanetScale region will look like this:

1. DFW → us-east-1 (Virgina), the primary database
2. FRW → eu-central (also Frankfurt), a read-replica
3. SIN → ap-southeast (also Singapore), a read-replica

Fly.io has some regions closer to PlanetScale’s Virginia region than Dallas, but that won’t affect our results too much.

Scaling Laravel Globally

First, let’s see the easy part. What does it look like to scale a Laravel application out to multiple regions on Fly.io?

Let’s say you already have an application running in DFW (Dallas, Texas). You can add additional regions to your application like so:

# Head to your app, where fly.toml lives
cd ~/Fly/my-app

# Add additional regions Frankfurt and Singapore
fly regions add fra sin

Now our application is allowed to run in 3 regions. However, it’s still only running in DFW!

To get the app running in all 3 regions, we need to scale our application up. Fly will take care of splitting your app instances amongst the available regions:

# Head to your app, where fly.toml lives
cd ~/Fly/my-app

fly scale count 3

Good news, everyone!

You’re running your Laravel application in multiple regions! Customers are certain to flock to your site, likescribe to your content, and purchase your widgets.

But wait! It’s slow!?

It turns out that making your application code closer to your users may also require moving your data along with it. Shooting TCP packets around the globe is slow! It would be much better if our database was just a short hop away from the application code querying it.

How do we do this? The short answer is: Read Replicas.

Read Replicas

Yes, read replicas! An eventually-consistent copy of your data, propagated from a primary database instance to read-only replica instances spread around the world.

You can set this up yourself but…lets not. Save some of your life-hours and choose a quick swipe of the credit card. PlanetScale can do the work for you (without being super expensive).

Within PlanetScale, you just click some buttons. Here are the buttons to click. Just create a database and add some regions, we’ll cover the rest here.

For the Laravel side of things, read on!

Laravel Configuration

You may recall that Laravel has configuration to help us split read/write connections. Here’s what it looks like:

// Found in file config/database.php

'mysql' => [
    'read' => [
        'host' => [
            '192.168.1.1',
            '196.168.1.2',
        ],
    ],
    'write' => [
        'host' => [
            '196.168.1.3',
        ],
    ],
    'sticky' => true,
    // more stuff
];

With this, Laravel will send any “write” queries to the write connection, and any read queries to one of the hosts in the read query array.

The Problem

Unfortunately this isn’t the whole story. This is a great feature, but has a few implementation details.

1. Laravel is going to choose a read host at random. However, we want a specific database host to be chosen per Fly region.
2. Write queries will always go to the database in DFW, even if the application serving a request is in Singapore. Database connections going across the globe are going to be 200-year-old-tortoise slow 🐢.

We know ahead of time what Fly.io regions and what PlanetScale regions we operate within (we chose them!). We’ll use that knowledge to make Laravel smart about how it talks to the database.

Choosing the Database Connection

Traffic will be routed to a specific region depending on where in the world you are. In our case, that’s either DFW, FRA or SIN. We want to choose our database connection based on the region that is serving a request.

Let’s say we have some environment variables like this (in your fly.tomlfile):

[env]
DB_HOST_dfw = "db_closest_to_dallas.psdb.cloud"
DB_HOST_fra = "db_closest_to_frankfurt.psdb.cloud"
DB_HOST_sin = "db_closest_to_singapore.psdb.cloud"

Something like this allows us to hack some configuration on the read connection, like so:

'read' => [
    'host' => [
        // Get hostname from env var: DB_HOST_xxx (e.v. DB_HOST_dfw)
        // But then we couldn't use the config cache :/
        env('DB_HOST_'.env('FLY_REGION', 'dfw'), 'default_host.psdb.cloud'),
    ],
],

However, this will break if we try to use Laravel’s config cache (which you want to do, as it speeds up Laravel a bunch).

To avoid that problem, we can turn to some code. Inside of your AppServiceProvider (or service provider of your choice), we can add some logic to override the read host set in the configuration.

public function boot()
{
    // "match" is fancy PHP 8+ stuff, we could use a switch{} instead
    $dbHostname = match (env('FLY_REGION')) {
        'mad' => env('DB_HOST_fra'), // Frankfurt
        'sin' => env('DB_HOST_sin'), // Singapore
        'dfw' => env('DB_HOST_dfw'), // Dallas
        // Default to primary/write host:
        default => config('database.connections.mysql.write.host'),
    };

    // Only give Laravel one "read" host to choose from
    // Based on the closest database server
    config()->set('database.connections.mysql.read.host', [$dbHostname]);
}

This will set our read host to the appropriate hostname on the fly!

Wait, what’s happening here?

A few things!

First, the write host is always going to be an array of length 1, containing our primary database hostname. This is where all write operations must go (the others are read-only replicas). We default to sending queries to this database if we don’t match another region.

The read hosts can be set to an array of 1 or more hosts. However, Laravel will just choose one of those randomly. We need to make our application smarter than that. To make it smarter, we do the dumbest thing possible - only give it one option to choose from!

There’s another issue!

PlanetScale gives us a username, hostname, and password that’s unique for each database instance.

This means our application needs to know the connection details for all 3 database instances.

So our application can use environment variables / secrets like this:

# DB DEFAULTS
DB_DATABASE=laravel-multi-region
MYSQL_ATTR_SSL_CA=/etc/ssl/certs/ca-certificates.crt

# DFW
DB_HOST_dfw=xxx.us-east-2.psdb.cloud
DB_USERNAME_dfw=xxxyyyzzz
DB_PASSWORD_dfw=pscale_pw_some_long_password1

# FRA
DB_HOST_fra=eu-central.connect.psdb.cloud
DB_USERNAME_fra=aaabbbccc
DB_PASSWORD_fra=pscale_pw_some_long_password2

# SIN
DB_HOST_sin=ap-southeast.connect.psdb.cloud
DB_USERNAME_sin=dddeeefff
DB_PASSWORD_sin=pscale_pw_some_long_password3

The passwords should be set as secrets in your app.

fly secrets set DB_USERNAME_dfw="pscale_pw_some_long_password1" \
    DB_USERNAME_fra="pscale_pw_some_long_password2" \
    DB_USERNAME_sin="pscale_pw_some_long_password3"

Assuming our env vars / secrets are set, our config/database.php file can be updated to this:

# mysql connection info
'mysql' => [
    'read' => [
        'host' => [
            env('DB_HOST_dfw')
        ],
        'username' => env('DB_USERNAME_dfw'),
        'password' => env('DB_PASSWORD_dfw', ''),
    ],
    'write' => [
        'host' => [
            env('DB_HOST_dfw')
        ],
        'username' => env('DB_USERNAME_dfw'),
        'password' => env('DB_PASSWORD_dfw', ''),
    ],
    'sticky' => true,
    // other stuff
],

Luckily, Laravel lets us set a unique username and password per read or write connection.

To ensure Laravel uses the correct connection information per region (and to connect to the correct read-replica when serving from Frankfurt or Singapore), we can update our AppServiceProvider with the following:

  public function boot()
  {
      /**
       * Get hostname, username, password for a given region
       */

      $dbHostname = match (env('FLY_REGION')) {
          'fra' => env('DB_HOST_fra'), // Frankfurt
          'sin' => env('DB_HOST_sin'), // Singapore
          'dfw' => env('DB_HOST_dfw'), // Dallas
          default => config('database.connections.mysql.write.host'),
      };

      $dbUsername = match (env('FLY_REGION')) {
          'fra' => env('DB_USERNAME_fra'), // Frankfurt
          'sin' => env('DB_USERNAME_sin'), // Singapore
          'dfw' => env('DB_USERNAME_dfw'), // Dallas
          default => config('database.connections.mysql.write.username'),
      };

      $dbPassword = match (env('FLY_REGION')) {
          'fra' => env('DB_PASSWORD_fra'), // Frankfurt
          'sin' => env('DB_PASSWORD_sin'), // Singapore
          'dfw' => env('DB_PASSWORD_dfw'), // Dallas
          default => config('database.connections.mysql.write.password'),
      };

      config()->set('database.connections.mysql.read.host', [$dbHostname]);
      config()->set('database.connections.mysql.read.username', $dbUsername);
      config()->set('database.connections.mysql.read.password', $dbPassword);
  }

Now our application will send read queries to a database pretty close to Fly’s application servers!

Test it out

To test this out, I created an app with a route that measures times for one read query and and one write query.

To help us test timings, we can use the (semi-secret) header fly-prefer-region, which tells Fly’s proxy which region to route a request to.

# The "fly-prefer-region" header tells FLy which region to route
# an HTTP request through

## Dallas
curl -i -H "fly-prefer-region: dfw" https://global-af.fly.dev/dump

REGION: dfw
CONNECTION: x0f5nogmbz0w.us-east-2.psdb.cloud via TCP/IP
READ QUERY TIME: 77.87
WRITE QUERY TIME: 445.74


## Frankfurt
curl -i -H "fly-prefer-region: fra" https://global-af.fly.dev/dump

REGION: fra
CONNECTION: eu-central.connect.psdb.cloud via TCP/IP
READ QUERY TIME: 12.37ms 
WRITE QUERY TIME: 1068.77ms


## Singapore
curl -i -H "fly-prefer-region: sin" https://global-af.fly.dev/dum

REGION: sin
CONNECTION: ap-southeast.connect.psdb.cloud via TCP/IP
READ QUERY TIME: 11.96ms
WRITE QUERY TIME: 2892.84ms # nearly 3 seconds!

We can see that our write query time gets higher the further away from DFW we are.

That’s Not Ideal

We have Laravel splitting read/write connections. The sticky feature is pretty cool too. But what’s happening with write queries?

Well, a lot of our requests are going to make connections to a database that’s far away. This is exasperated in an app that does a lot of database writes, and still further trouble with the sticky feature.

While our HTTP requests are going to be routed to an app server pretty close to a user, the database connection may still be galavanting across the globe.

So that sucks.

Here’s a secret though - It turns out it’s usually much faster to route an HTTP request to region closest to the primary database than it is to have a database connection from far away.

This is because (among other reasons) the typical app has an asymmetry. A single HTTP request can easily result in a crap-ton (that’s a technical term) of queries.

These queries are perfect, don’t @ me.

If we route this HTTP request to Singapore but then force our app to connect to a database in Dallas, we’re gonna have a bad time.

It would be faster overall if we made the single, smaller HTTP request traverse the globe instead of the heavily-used database connection.

The Fly-Replay Header

Enter, stage left, the Fly Replay header. Any sufficiently advanced technology is indistinguishable from magic Rust, and we have some fairy Rust in the form of Fly’s proxy layer.

🪄 If your code returns an HTTP request with a fly-replay header, Fly’s proxy will read the value of that header and replay the HTTP request to your app in another region of your choice (or even another app in your org)!!

This gives your application the opportunity to say “this request probably is going to write to the database, how about you go to our primary region instead?”.

More often than not, this is orders of magnitude faster.

Fly-Replay in Laravel

One way we can make use of fly-replay is through an Http middleware. Let’s create a new one:

php artisan make:middleware ReplayWriteRequest

We can edit our new middleware at app/Http/Middleware/ReplayWriteRequest.php:

public function handle(Request $request, Closure $next)
{
    // Perhaps make available as a config:
    // config('fly.regions.primary');
    $primaryRegion = 'dfw';

    // Any route that has this middleware applied
    // will inform Fly to replay the request against dfw
    // if the current region is not already dfw
    if (env('FLY_REGION') && env('FLY_REGION') != $primaryRegion) {
        return response('', 200, [
            'fly-replay' => $primaryRegion,
        ]);
    }

    // Else, continue as normal
    return $next($request);
}

Note that we use env('FLY_REGION') and not $request→headers→get('fly-region'). This got me at first. As per the docs, the header is the edge location accepting your request, while env var is the region the application instance resides within.

This middleware isn’t meant to be global, but instead applied at your discretion.

You can register/assign this middleware a name in app/Http/Kernel.php:

protected $routeMiddleware = [
    // Other stock middleware omitted
    'fly-replay' => \App\Http\Middleware\ReplayWriteRequest::class,
];

That’s now named fly-replay. We can assign that to any route that we think will make write requests that should not traverse across the globe.

Then we can apply this middleware to whatever route we want:

// File routes/web.php

// This route will tell Fly to replay the request against
// the primary region (if it's not currently the primary region)
Route::delete('/user/{id}', \App\Http\Controllers\DeleteUserController::class)
    ->middleware(['auth', 'fly-replay']);

Using this method evens out our write-query time, since they are all between Fly’s DFW region and PlanetScale’s Virginia region.

curl -H "fly-prefer-region: dfw" https://global-af.fly.dev/insert

REGION: dfw
CONNECTION: x0f5nogmbz0w.us-east-2.psdb.cloud via TCP/IP
READ QUERY TIME: 67.43ms
WRITE QUERY TIME: 384.07ms

curl -H "fly-prefer-region: fra" https://global-af.fly.dev/insert

REGION: dfw
CONNECTION: x0f5nogmbz0w.us-east-2.psdb.cloud via TCP/IP
READ QUERY TIME: 70.17ms
WRITE QUERY TIME: 380.57ms

curl -H "fly-prefer-region: sin" https://global-af.fly.dev/insert

REGION: dfw
CONNECTION: x0f5nogmbz0w.us-east-2.psdb.cloud via TCP/IP
READ QUERY TIME: 67.81ms
WRITE QUERY TIME: 428.83ms

Requests being routed to Singapore, then being told to replay the request in DFW will take a bit of extra time. However, it’s still much faster (and consistent) than making long-distance database connections!

What if I miss a route?

Note that I never said to undo the configuration where we split read/write connections!

If our middleware isn’t applied to a route that writes to the databases, it should still work. Laravel will send the write queries to the write connection in the primary reason. It’s just a lot slower.

What hath we wrought?

Running apps in multiple regions is traditionally really hard. The fact that I was able to set this all up in an afternoon is actually blowing my mind. Is it perfect? Not really! We can’t escape the speed of light. But PlanetScale and Fly.io helps us solve the thorniest issue - keeping latency low when your database is far away from your users.