Running Laravel Workers

We’re going to talk about running Laravel as a worker on Fly.io. This isn’t really anything special (we aren’t out to make things hard), but it’s useful to see how it essentially becomes a lesson in Docker.

The “usual” use case on Fly.io is running a web app, and then perhaps adding on some other services as needed. However, Fly.io is also great for just running background (or temporary) work loads, like queue workers!

So, here’s a few ways to run Laravel’s queue workers. Along the way we’ll pick up some tricks about Docker and Fly Machines.

Workers as Another Process

The “standard” way to run a worker is as we document it. It assumes you’re running your web application on Fly.io, and just want to add some queue workers.

To quickly do so, we can define a new process in our fly.toml file, and name it something useful, such as worker.

[processes]
  app = ""
  worker = "php artisan queue:listen"

The app process is the standard web app (you always have an app process), while our new worker process will run the command given (artisan queue:listen).

We also need to adjust [http_service] section to ensure the HTTP health checks only apply to the app process:

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
+ processes = ["app"]

Note that defining additional processes will spin up a VM per process. In fact, for reliability, you’ll get 2 VMs! One is on standby, so you can pretend it’s just one VM for the most part.

How does the [processes] section work? It has to do with the Docker setup used to generate the Fly.io VM for Laravel apps.

Our Laravel Container

Docker uses ENTRYPOINT and CMD by concatenating them together to form the command used to run a Docker image (and therefore a Fly.io VM). For example, if we define an ENTRYPOINT as script /entrypoint.sh, and then a CMD as my-app, we’d end up running our VM with command /entrypoint.sh my-app.

In the standard Laravel container image we provide via fly launch, we define an ENTRYPOINT (but no CMD).

The ENTRYPOINT script is just some bash (/entrypoint.sh) that looks like this:

#!/usr/bin/env bash

if [[ $# -gt 0 ]]; then
    exec "$@"
else
    exec supervisord -c /etc/supervisor/supervisord.conf
fi

If we pass it nothing (CMD is empty) it runs supervisord. If we pass it a different command (define a different CMD), it runs that command instead. Our default process app = "" runs Supervisor (which turns on php-fpm/nginx). Our process worker sets artisan queue:listen as the CMD and therefore runs that instead!

Workers Alongside Your App

What if we wanted a worker running in the same VM as our app, we can do that! This is a bit more traditional - like Laravel Forge’s setup - where everything is crammed into a single VM.

In this scenario, we can update Supervisor so it runs a worker as well as nginx/php-fpm. It takes a bit more work than defining a process, and also competes for resources in your Fly VM.

To do it, you can edit the files in the standard Docker setup you get via fly launch.

The base Docker image used is fideloper/fly-laravel. Inside the base image is Supervisor config found within /etc/supervisord.

The configuration for nginx/php-fpm (or octane, if you use it) will be found in /etc/supervisord/conf.d/(nginx.conf|fpm.conf). What you can do is add another file to the conf.d directory, with something similar to this:

[program:worker]
priority=5
autostart=true
autorestart=true
stdout_events_enabled=true
stderr_events_enabled=true
command=php /var/www/html/artisan queue:listen
user=www-data
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

If you create this file in your codebase at .fly/worker.conf, and then update your Dockerfile, you can get this file to the correct place.

Note that I’m assuming you have a .fly directory, generated by the fly launch command. Here’s the updated Dockerfile that will incorporate the new worker.conf file:

# Somewhere after this line...
COPY . /var/www/html


# Move the worker conf into conf.d to live with the
# other configurations
RUN mv .fly/worker.conf /etc/supervisor/conf.d/worker.conf


# Do the above before the line doing Nodejs stuff
FROM node:${NODE_VERSION} ...<and so on>

If you deploy your app with this setup, the worker.conf config will run artisan queue:listen in addition to the other standard configurations (php-fpm/nginx or octane).

We do NOT define an extra process in the [processes] section.

Just a Worker

If we wanted JUST a worker to be run, and never a web server, we could do a few things! One is to erase the other Supervisor configurations and replace it with our worker.conf from above. We’d have to adjust the Dockerfile like above, erase all other supervisor configurations, and finally remove the fly.toml‘s [http_service] section.

However I think the following is a bit less work - we can instead update our ENTRYPOINT script to never run Supervisor and always run a worker by default.

Here’s what it looks like - edit file .fly/entrypoint.sh and adjust it to run your worker instead of Supervisor:

#!/usr/bin/env sh

# Run user scripts, if they exist
for f in /var/www/html/.fly/scripts/*.sh; do
    # Bail out this loop if any script exits with non-zero status code
    bash "$f" || break
done
chown -R www-data:www-data /var/www/html

if [ $# -gt 0 ]; then
    # If we passed a command, run it as root
    exec "$@"
else
    # Don't run supervisord by default
    # exec supervisord -c /etc/supervisor/supervisord.conf
    exec php /var/www/html/artisan queue:listen
fi

That’ll never run Supervisor and will only run whatever flavor of the queue:listen command you’d like. You can incorporate environment variables defined in your fly.toml as well:

# If `QUEUE` is an env var in your fly.toml file
exec php /var/www/html/artisan queue:listen --queue $QUEUE

Any time you deploy this app, it’ll automatically run a worker now instead of the web app. Yet again, we do NOT define an extra process in the [processes] section.

Automated Workers

You can run a Fly Machine directly (without using fly launch…fly deploy) via CLI or API calls. This is handy for automating workloads!

I outline how to do that in this article, where we run a custom artisan command. The only real difference for our use case here would be the command we define (artisan queue:listen... instead of artisan get-release).

The main trick is ensuring your app has a Docker image that Fly can use, either hosted publicly (Docker Hub, for example), or built and pushed up to the Fly registry before attempting to create a machine from that image.

Using the CLI, you can build the image on the fly - but through the API, you need to push the Docker image up to a registry ahead of time.

For CLI, it might look like this:

fly apps create --name my-worker-app

# Assuming the app's Dockerfile and code
# base are in the current directory...
fly m run -a my-worker-app \
    --env "APP_ENV=production" \
    --env "LOG_CHANNEL=stderr" \
    --env "LOG_LEVEL=info" \
    --env "LOG_STDERR_FORMATTER=Monolog\\Formatter\\JsonFormatter" \
    . \
    "php" "artisan" "queue:listen"

Using the API requires that you build and push your Docker image to the Fly registry so the image is available. That looks like this:

# Get your access token from ~/.fly/config.yml
# Or via `fly auth token`
export FLY_API_TOKEN="$(fly auth token)"

curl -X POST \
    -H "Authorization: Bearer ${FLY_API_TOKEN}" \
    -H "Content-Type: application/json" \
    "https://api.machines.dev/v1/apps" \
    -d '{
      "app_name": "my-worker-app",
      "org_slug": "personal"
}'

# Build the image locally. The image name must match
# app name we used when creating the machine app
docker build -t registry.fly.io/my-worker-app:latest

# Authenticate against Fly's registry
fly auth docker

# Push our newly tagged image
docker push registry.fly.io/my-worker-app:latest

# Create a machine using that Docker image
curl -X POST \
    -H "Authorization: Bearer ${FLY_API_TOKEN}" \
    -H "Content-Type: application/json" \
    "https://api.machines.dev/v1/apps/my-worker-app/machines" \
    -d '{
  "config": {
    "image": "registry.fly.io/my-worker-app:latest",
    "processes": [
      {
        "name": "do-some-work",
        "cmd": ["php", "artisan", "queue:listen"],
        "env": {
          "APP_ENV": "production",
          "LOG_CHANNEL": "stderr",
          "LOG_LEVEL": "info",
          "LOG_STDERR_FORMATTER": "Monolog\\Formatter\\JsonFormatter"
        }
      }
    ]
  }
}'

That’s more verbose and requires some extra work, but gives you an avenue to spin workers up dynamically (via code) if that’s what you need!