Concurrent Tasks on Machines

Previously I wrote about spinning up Fly Machines to run tasks “on-demand”. Machines are great because they stop when a task finishes - no need to waste (paid) CPU cycles.

However, I hand-waved over the need for concurrency. To run multiple tasks at the same time, you need multiple Machine VMs! Let’s explore some fun ways to manage that.

Tasks to Run

We saw a contrived example previously - the code retrieved some value from GitHub’s API.

This time we’ll create some queue workers, but with a bit of flair. Instead of running 24/7, we’ll have the queue workers churn through available jobs and then shut off.

To do that, we’ll run php artisan queue:work --stop-when-empty. That’ll process jobs as long as there are jobs to be found, and then exit.

Since the machine doesn’t need to run 24/7, we won’t be charged for unused CPU time!

However, it begs the question: How to start the stopped workers?

Starting Machines

If our Machines are gonna stop, we need something to turn them on again.

This means we need some event to hook into. Here are some options:

1. Periodically turn on the workers via scheduled machines
2. Have our code turn them on when needed

Let’s explore some fun things we can do, especially that 2nd option.

Scheduled Machines

Machines can be scheduled. At the moment, the shortest interval currently available is every hour. That’ll change, so if it’s the future, I may have just lied to you.

In any case, we can say “every hour, turn on these Machines, and churn through all available jobs in the queue”.

Here’s how to do that (note the --schedule flag):

# Run this for as many machines as you want
fly m run -a on-demand \
    --env "APP_ENV=production" \
    --env "LOG_CHANNEL=stderr" \
    --env "LOG_LEVEL=info" \
    --schedule hourly \
    . \
    "php" "artisan" "queue:work" "--stop-when-empty"

We want multiple machines, so we’d run the above command for as many machines as we want. Every hour, each machine will run queue:work, and process jobs until the given queue is empty.

This is nice, but not perfect. What if we need a queue job to send an email ASAP? Do we really need to wait up to an hour to send that email?

It turns out that we can stagger the timing for each machine! The interval calculated (hourly, daily, etc) starts when a Machine is created. If we create an hourly machine at 3:15pm, the next run will be ~4:15pm.

If you want to take the time to stagger machine creation over an hour or so, you can! This will let you run a machine every 15 minutes, if you stagger the creation of 4 machines every 15 minutes (get some coffee)! You can create more machines to fill up that hour as you’d like.

That of course is a little wonky. Let’s see what other hijinks we can get into.

Start Machines via API

Scheduled Machines let you run tasks without thinking about when to start the VMs yourself (you don’t need to build it into your application logic). That’s nice, but the timing could be an issue.

The other thing you can do is build the logic into your application. We can programmatically start a given machine with the Machines API. This might be useful if you fire a job into a queue, and then decide to start a machine to ensure one is around to process that job.

In Laravel, that would look something like this:

$token = "foo";
$app = "on-demand";
$machine = "xyz133";
$url = "https://api.machines.dev/v1/apps/${app}/machines/${machine}/start"

$result = Http::asJson()
    ->withToken($token)
    ->post($url); 

The trick here is managing multiple machines! If you have multiple, you’ll want to decide which to start. Perhaps you roll through them one by one and select the “next” machine on each call. Perhaps you just start every machine each time! It’s up to you, but it’s not a simple problem. You have to think carefully about when this is done so you don’t accidentally end up having jobs sit in queue longer than makes sense.

One thing you can do is use Laravel’s Scheduler to start machines up to every minute (since it’s based on CRON). This is especially nice if you calculate the number of jobs in a queue before deciding to turn on a worker.

That might look like this:

$token = "xyz";
$machines = ["aaabbbccc", "xxxyyyzzz", ...]
$app = "on-demand";
$url = "https://api.machines.dev/v1/apps/%s/machines/%s/start"

if (Queue::size('some-queue') > 0) {
    foreach($machines as $machine) {
        Http::asJson()
            ->withToken($token)
            ->post(sprintf($url, $app, $machine)); 
    }

}

Start Machine via HTTP

One really nice feature of Machines is the ability to wake on network access. This means a machine can be created, but not running. Once a network request is made to the machine, it will be started!

If we have 2+ Machines listening for HTTP requests within an app, the Fly Proxy will actually load balance requests amongst the Machines! So, another tricky way to start a Machine is to send an HTTP request to it.

To do this little bit of fanciness, we need to create our machines via API to set some specific options. Here’s what that looks like:

# Be sure image "registry.fly.io/on-demand:latest" exists
# and has been pushed, as described here:
# https://fly.io/laravel-bytes/on-demand-compute/#create-a-machine-vm


FLY_API_TOKEN="$(fly auth token)"
FLY_APP_NAME="on-demand"

curl -X POST \
    -H "Authorization: Bearer ${FLY_API_TOKEN}" \
    -H "Content-Type: application/json" \
    "https://api.machines.dev/v1/apps/${FLY_APP_NAME}/machines" \
    -d '{
  "config": {
    "image": "registry.fly.io/on-demand:latest",
    "processes": [
      {
        "name": "start-worker",
        "cmd": ["php", "artisan", "queue:work", "--stop-when-empty"],
        "env": {
          "APP_ENV": "production",
          "LOG_CHANNEL": "stderr",
          "LOG_LEVEL": "info"
        }
      },
      {
        "name": "web-requests",
        "cmd": ["php", "-S", "[::0]:8080", "-t", "start"],
        "env": {
          "APP_ENV": "production",
          "LOG_CHANNEL": "stderr",
          "LOG_LEVEL": "info"
        }
      }
    ],
    "services": [
      {
        "ports": [
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          },
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          }          
        ],
        "concurrency": {
          "type": "requests",
          "soft_limit": 1
        },
        "protocol": "tcp",
        "internal_port": 8080
      }
    ]
  }
}'

There’s a few things going on here!

First, we define two processes to run in the VM! The first is the queue worker (artisan queue:work...). The second is an HTTP listener that just serves our application using php -S [::0]:8080 -t start. This is PHP’s built-in web server. It’s using a directory start as its web root. That directory just contains an empty index.html file:

# From your project's root directory:
mkdir start
touch start/index.html

This lets us make web requests into the Machine without having to make our Laravel application function (by setting needed environment variables, etc). That being said, you’ll need those env vars/secrets for the queue worker to work, so you likely could use php artisan serve --port 8080 if you wanted. In any case - to wake up the machine, all we need is something that listens for HTTP requests.

Note: If either of the two processes exits with status 0 (success), then the Machine will stop. This means we don’t have to worry about the HTTP listener staying on forever and keeping the Machine awake.

Second, we add a service definition. This lets the proxy know that we’re expecting HTTP requests. We set concurrency to a single request at a time (via soft_limit), telling the proxy to spread each request as evenly as possible amongst Machines listening for web requests.

Waking it Up

To wake the machine up, we need to send an HTTP request to it. The magic of waking up a machine only works if we send requests through the Fly Proxy. This means that if we stay within our private network and do something like curl machine-id.vm.<app-name>.internal:8080, it won’t wake the machine up. This sends the request directly to the VM without going through the Fly Proxy.

To use the Fly Proxy, we need to allocate it an IP address. We can allocate a private-only IPv6 address like so:

# Use your own app name, mine is "on-demand"
fly ips allocate-v6 --private -a on-demand

Now, to wake up your machines, you can make curl requests to that IP address:

# Assume the IPv6 I got was fdaa:0:6ba9:0:1::2
$response = Http::get("http://[fdaa:0:6ba9:0:1::2]")
$response->status(); // 200

We send requests to port 80, as the services portion of our Machine configuration listens on port 80, and forwards those requests to port 8080 in the machine (where php -S is listening).

Since the IP address is allocated to the application (not the individual Machine VM), this will load balance amongst the Machines, turning them on if they are off. You can keep making requests to this for as many Machines as you want enabled! This may require a bit of concurrency:

# If we have 3 machines we want on
# See https://laravel.com/docs/9.x/http-client#concurrent-requests
$responses = Http::pool(fn (Pool $pool) => [
    $pool->get("http://[fdaa:0:6ba9:0:1::2]"),
    $pool->get("http://[fdaa:0:6ba9:0:1::2]"),
    $pool->get("http://[fdaa:0:6ba9:0:1::2]"),
]);

Note that when allocating a private IP address, Fly.io doesn’t have a hostname you can use that routes requests through the Fly Proxy. If you want a hostname, you’ll have to set one up yourself on a domain you own. You can set a DNS record on your domain and point it to the private IP address.

If you need to send requests to your application over the public internet, you can allocate an IP that is NOT private (omit the --private flag), and send requests to <your-app>.fly.dev (or use your own domain).

What We Did

This is a fun and tricky way to let our application just send some HTTP requests to Fly whenever we want our workers running. Fly will handle deciding which Machine VM to start.

Sending a request will start a stopped Machine. Since we defined 2 processes, both the queue worker and the web server will be started. Our jobs will be processed, and the queue worker will eventually exit. The Machine will then stop.