Handling Signals in Laravel

Laravel queues are stopped gracefully. What does this mean?

During a deployment, you likely restart your queue workers using something like artisan queue:restart or supervisorctl restart <worker-name>.

Laravel has graciously noticed that we don’t like it when an in-process job is suddenly killed. What Laravel does is check to see if a queue worker should stop. If it should, the worker waits until the currently running job is finished, and then exits.

It does this using signals. (We all know that if a page is using 1990’s default HTML formatting, it’s legit).

Signals?

Yes, signals. The type of Linuxy thing Julia Evans is great at teaching.

Signals are events that a process can listen for and choose to respond to. Hitting ctrl+c in your terminal, for example sends a SIGINT (interrupt) to the currently running process. This usually results in the process stopping.

You can also use the kill command to send a signal (any signal). Sending SIGINT via kill looks like this:

# These are equivalent
kill -2 <process-id>`
kill -s INT <process-id>

Sending SIGKILL (kill -9 <process-id> or kill -s KILL <process-id>) is special - it will kill a process immediately. The process has no choice in the matter.

Let’s see how Laravel accomplishes graceful queue restarts, and how we can use that idea in our own code.

Laravel Queues

Laravel’s Queue library implement signals in order to stop workers gracefully. When a SIGTERM or SIGQUIT signal is received, the worker waits until the currently processing job finishes before actually stopping.

Therefore a job isn’t interrupted in middle of processing - it has the chance to finish.

This is done with a simple boolean variable. The Worker is basically just a while() {} loop. Every iteration it checks this variable, and stops if $shouldQuit == true.

We can see that Laravel listens for both SIGTERM and SIGQUIT signals here. Listening for these signals is setup just before starting the aforementioned while() loop.

Not too magical!

SIGINT

The terminate (SIGTERM) and quit (SIGQUIT) are both obviously named - they want a process to end (the difference is that SIGQUIT generates a core dump).

What about SIGINT (interrupt)?

This is the signal sent via ctrl+c. It’s typically only used in an interactive terminal - when we’re at our keyboard (for local development or for those 1-off tasks in production you really should automate).

You’ll notice that SIGINT isn’t listened for in Laravel’s queue worker! Instead, that’s handled by PHP, and it just quits whatever happens to be running. It is, therefore, not a way to gracefully exit a process!

Let’s see that quick. I created a job named LongJob that just sleeps for 10 seconds:

<?php

namespace App\Jobs;

use ...

class LongJob implements ShouldQueue
{
    use ...

    public function handle(): void
    {
        Log::info("starting LongJob ".$this->job->getJobId());
        Sleep::for(10)->seconds();
        Log::info("finished LongJob ".$this->job->getJobId());
    }
}

I had a terminal open, running php artisan queue:work. Then I dispatched this job, and quickly hit ctrl+c. The logs showed that the job started, but never finished!

[2023-06-28 14:19:09] local.INFO: starting LongJob 1 

If instead I send it signal SIGTERM, it will finish the job and then exit:

# Start a worker
php artisan queue:work

# Find the process ID
ps aux | grep queue:work

# Dispatch a job, and then 
# kill the  worker with SIGTERM
# Process ID 69679 in my case
kill -s TERM 69679

We’ll see the job finishes before the process exits! However the sleep didn’t sleep for 10 seconds. More on that below!

[2023-06-28 14:19:09] local.INFO: starting LongJob 2  
[2023-06-28 14:19:11] local.INFO: finished LongJob 2  

Graceful Restarts during Deployment

An in-production queue worker is typically monitored by a process monitor such as Supervisor. This restarts any process that stop in an unexpected way.

Laravel’s queue worker takes advantage of this by stopping the worker in a bunch of cases (error conditions, when artisan queue:restart is run, etc) as it can assume the queue worker will restart when needed.

Supervisor and friends usually stop a process by sending a SIGTERM signal, then waiting for the process to gracefully exit itself (which Laravel will do itself as describe above).

Typically, Supervisor (or whatever) will give the process a certain number of seconds to exit. If that time elapses, then SIGKILL is sent, and the process is forcefully stopped (since processes can’t ignore SIGKILL).

In Supervisor, the timeout is set by the stopwaitsecs options. The Laravel docs have this set to one hour (in seconds) in their example. You can lower this if your jobs don’t run a long time and wouldn’t ever need an hour to complete.

Using Signals in Our Code

Let’s see how to implement signals ourselves!

Jobs already complete fully when a signal comes in to stop them. From the perspective of us developers, we’re more likely to want to capture signals from within our artisan commands.

The first thing we’ll see is how a signal will stop a process (a running command) immediately.

Note that I’ll use “process” and “command” interchangeably here. Running a command like php artisan whatever spins up a PHP process. That process happens to run artisan, which boots the framework, runs our command, yadda yadda yadda. The point is: both words work here!

I created command LongCommand and had it sleep for 10 seconds (just like LongJob).

public function handle()
{
    $this->info('starting long command: '.now()->format('H:i:s'));

    Sleep::for(10)->seconds();

    $this->info('finished long command: '.now()->format('H:i:s'));
}

If I run this via artisan longtime and use ctrl+c, it stops immediately:

php artisan longtime   

starting long command 14:27:59
^C%   

There was no output showing it finishing the command!

Trap

We can “trap” - listen for - a signal. This lets us run code before the command exits.

Trap lets you capture a signal, and do something in response, but it will then exit immediately.

public function handle()
{
    $this->trap(
      [SIGINT, SIGTERM], 
      fn($s) => $this->info('signal received: ' . $s)
    );

    $this->info('starting long command: '.now()->format('H:i:s'));

    Sleep::for(30)->seconds();

    $this->info('finished long command: '.now()->format('H:i:s'));
}

We trap SIGINT and SIGTERM and just echo out some information. This gives us a hook to run some cleanup code before exiting!

php artisan longtime   

starting long command: 14:29:50
^Csigint received   

The signal SIGINT was “trapped” but it still stopped the process! We get similar behavior for SIGTERM:

php artisan longtime   

starting long command: 14:33:39
sigint received 

So we can respond to a signal, but we can’t actually stop the process from exiting once the callback is run.

It would be useful if we could ignore the signal until we’re ready to exit! Luckily we can.

Implementing SignalableCommandInterface

If our command implements Symfony’s SignalableCommandInterface, we can get the command to finish running before it exits.

On the surface, it looks like this works just like the $this->trap() method. However, if we return false in our handler, the command is able to finish its work.

Here’s what that looks like:

# Some stuff omitted
use Symfony\Component\Console\Command\SignalableCommandInterface;

class LongCommandTwo extends Command implements SignalableCommandInterface
{
    public function handle()
    {
        $this->info('starting long2 cmd: '.now()->format('H:i:s'));

        Sleep::for(30)->seconds();

        $this->info('finished long2 cmd: '.now()->format('H:i:s'));
    }

    public function getSubscribedSignals(): array
    {
        return [SIGINT, SIGTERM];
    }

    public function handleSignal(int $signal)
    {
        $this->info('signal received: ' . $signal);
        return false;
    }
}

Since the signal handler returns false, our command is able to finish. How that works is just an implementation detail of Symfony’s handling of SignalableCommandInterface - it tells the code to not run exit($statusCode);.

After implementing that, we can see our “finished…” line is run:

php artisan long2

starting long2 cmd: 15:14:01
^Csignal received: 2
finished long2 cmd: 15:14:02

You might notice that we didn’t actually sleep for 30 seconds! This is specific to using sleep() for testing. Using signals actually shortcuts currently running sleep() calls, so if your code relies on that, it could be an issue!

Why do this?

A useful pattern for this is to do some cleanup work before exiting:

# Some stuff omitted
use Symfony\Component\Console\Command\SignalableCommandInterface;

class LongCommandTwo extends Command implements SignalableCommandInterface
{
    protected $shouldExit = false;

    public function handle()
    {
        $this->info('starting long2 cmd: '.now()->format('H:i:s'));

        while(! $this->shouldExit) {
            $this->info("We're doing stuff");
            Sleep::for(1)->seconds();
        }

        // Pretend we're working hard on
        // cleaning everything up
        // Oh, also, this sleep actually happens
        // since it was started after the signal was received
        Sleep::for(10)->seconds();

        $this->info('finished long2 cmd: '.now()->format('H:i:s'));
    }

    public function getSubscribedSignals(): array
    {
        return [SIGINT, SIGTERM];
    }

    public function handleSignal(int $signal,)
    {
        $this->shouldExit = true;
        $this->info('Cleaning up: signal received: '.$signal);
        return false;
    }
}

You can now run some cleanup code either in the handleSignal() method or after the while() loop. This is great for deleting temporary files, ensuring data integrity when a command is turned off, closing networking connections, and all sorts of other stuff!