Ad Hoc Tasks, GitHub Actions Style

We’re going to create a setup where we can instruct Fly.io to spin up a VM and run some code. The code will read an instructions.yaml file and follow its instructions. The neat part: We can create the YAML file on-the-fly when we spin up the VM.

This lets us create one standard VM that can do just about any work we need. I structured the YAML a bit like GitHub Actions.

In fact, I built a similar thing before:

I re-created github actions on @flydotio machines in this little proof of concept

👨‍🍳 Make a yaml, sprinkle in some go, make an API request to Fly, and away we go. pic.twitter.com/HjfF6k6ohX

— Chris Fidao (@fideloper) August 11, 2023

It uses Golang, but here we’ll use Laravel 🐐.

Here’s a repository with the code discussed.

The Code

The code is pretty straight forward (partially because I ask you to draw the rest of the owl). Within a Laravel app, we’ll create a console command that does the work we want.

Here’s what I did to spin up a new project and add a console command to it:

composer create-project laravel/laravel ad-hoc-yaml
cd ad-hoc-yaml

composer require symfony/yaml

php artisan make:command --command="ad-hoc" AdHocComputeCommand

We need to parse some YAML, so I also included the symfony/yaml package.

And the command itself is quite simple 🦉:

<?php

namespace App\Console\Commands;

use Symfony\Component\Yaml\Yaml;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Log;

class AdHocComputeCommand extends Command
{
    protected $signature = 'ad-hoc';

    protected $description = 'Blindly follow some random YAML';

    public function handle()
    {
        try {
            $instructions = Yaml::parseFile("/opt/instructions.yaml");
        } catch(\Exception $e) {
            Log::error($e);
            $this->error($e->getMessage());
            return SELF::FAILURE;
        }


        foreach($instructions['steps'] as $step) {
            // draw the rest of the <bleep>ing owl
        }

        return SELF::SUCCESS;
    }
}

For a slightly-more-fleshed out version of this code, see this command class here.

Package it Up

Fly uses Docker images to create (real, non-container) VMs. Let’s package our app into a Docker image that we can deploy to Fly.io.

We’ll borrow the base image Fly.io uses to run Laravel apps (fideloper/fly-laravel:${PHP_VERSION}). We’ll just assume PHP 8.2:

FROM fideloper/fly-laravel:8.2

COPY . /var/www/html

RUN composer install --optimize-autoloader --no-dev

CMD ["php", "/var/www/html/artisan", "ad-hoc"]

Once that’s created, we can build that Docker image and push it up to Fly.io. One thing to know here: You can only push images to Fly.io’s registry for a specific app. The image name to use must correspond to an existing app, e.g. docker push registry.fly.io/some-app:latest. You can use any tags (e.g. latest or 1.0) as well as push multiple tags.

So, to push an image up to Fly.io, we’ll first create an app (an app is just a thing that houses VMs) and authenticate Docker against the Fly Registry. Then, when we spin up a new VM, we’ll use that image that’s already in the registry.

This is different from using fly deploy... which builds an image during deployment, and is meant more for hosting your web application. Here we’re more using Fly.io for specific tasks rather than hosting a whole application.

The following shows creating an app, building the Docker image, and pushing it up to the Fly Registry:

APP_NAME="my-adhoc-puter"

# Create an app
fly apps create $APP_NAME

# Build the docker image
docker build \
    -t registry.fly.io/$APP_NAME \
    .

# Authenticate with the Fly Registry
fly auth docker

# Push the docker image to the Fly Registry
# so we can use it when creating a new VM
docker push registry.fly.io/$APP_NAME

We make 2 assumptions here:

1. You have Docker locally
2. You’re on an Intel-based CPU

Pro tip: If you’re on a ARM based machine (M1/M2 Macs), you can actually VPN into your Fly.io private network and use your Docker builder (all accounts have a Docker builder, used for deploys) via DOCKER_HOST=<ipv6 of builder machine> docker build ....

Run It

To run our machine, all we need is a YAML file and to make an API call to Fly.io.

As mentioned before, Machines (VMs) spun up via API call let you create files on-the-fly! What you do is provide the file name and the base64'ed file contents. The file will be created on the Machine VM before it runs your stuff.

Here’s what the code to make such an API request would look like within PHP / Laravel:

# Some YAML, similar to GitHub Actions
$rawYaml = '
name: "Test Run"

steps:
  - name: "Print JSON Payload"
    uses: hookflow/print-payload
    with:
      path: /opt/payload.json

  - name: "Print current directory"
    run: "ls -lah $(pwd)"

  - run: "echo foo"

  - uses: hookflow/s3
    with:
      src: "/opt/payload.json"
      bucket: "some-bucket"
      key: "payload.json"
      dry_run: true
    env:
      AWS_ACCESS_KEY_ID: "abckey"
      AWS_SECRET_ACCESS_KEY: "xyzkey"
      AWS_REGION: "us-east-2"
';

$encodedYaml = base64_encode($rawYaml);

# Some random JSON payload that our YAML
# above references
$somePayload = '
{
    "data": {
        "event": "foo-happened",
        "customer": "cs_1234",
        "amount": 1234.56,
        "note": "we in it now!"
    },
    "pages": 1,
    "links": {"next_page": "https://next-page.com/foo", "prev_page": "https://next-page.com/actually-previous-page"}
}
';

$encodedPayload = base64_encode($somePayload);

# Create the payload for our API call te Fly Machines API
$appName = 'my-adhoc-puter';
$requestPayload = json_decode(sprintf('{
    "region": "bos",
    "config": {
        "image": "registry.fly.io/%s:latest",
        "guest": {"cpus": 2, "memory_mb": 2048,"cpu_kind": "shared"},
        "auto_destroy": true,
        "processes": [
            {"cmd": ["php", "/var/www/html/artisan", "ad-hoc"]}
        ],
        "files": [
            {
                "guest_path": "/opt/payload.json",
                "raw_value": "%s"
            },
            {
                "guest_path": "/opt/instructions.yaml",
                "raw_value": "%s"
            }
        ]
    }
}
', $appName, $encodedPayload, $encodedYaml));

// todo 🦉: create config/fly.php 
// and set token to ENV('FLY_TOKEN');
$flyAuthToken = config('fly.token');

use Illuminate\Support\Facades\Http;

Http::asJson()
    ->acceptJson()
    ->withToken($flyAuthToken)
    ->post(
        "https://api.machines.dev/v1/apps/${appName}/machines", 
        $requestPayload
    );

I created an artisan command that does that work here.

In your case, you might want to trigger this in your own code whenever you want some work to be done.

After you run some tasks, you should see the Machine VM spin up and do its work! Two things to make this more fun:

1. Liberally use Log::info() in your code so Fly’s logs can capture what’s going on (helpful for debugging)
2. Set your Logger to use the stderr logger so Fly’s logging mechanism can get the log output

Assuming that’s setup, you can then run fly logs -a <app-name> to see the log output as the Machine VM boots up, runs your code, and then stops.