Reading time • 7 min Share this post on Twitter Share this post on Hacker News Share this post on Reddit

Health Checks

sick birdie with thermometer in mouth
Image by Annie Ruygt

Rails 7.1 is adding discoverable health checks, which fly launch will automatically configure and monitor. This page will tell you what you need to know.

No matter how well you plan, you will always need to be prepared to deal with unforeseen and unforeseeable events. As Fly.io will route requests to the nearest healthy server, deploying your application across multiple regions not only means that you will be serving requests close to your users when things are running smoothly, it also means that you will be able to continue processing requests when there are isolated or even regional outages.

Let’s start with a demo. Run the following commands in a terminal window.

You can play with this right now.

It’ll take less than 10 minutes to get your Rails application running globally.

Try Fly for free → 
rails new health_demo --main --minimal
cd health_demo
echo 'Rails.application.routes.draw {root "rails/welcome#index"}' \
  >> config/routes.rb
fly launch

At this point you will be asked a series of questions. Feel free to accept the default by pressing enter in response to each question. You are now ready to deploy:

fly deploy

This command won’t complete until the health checks pass and your application is ready to process requests. You can visit your application by using the fly open command.

Now let’s take a look at the results of the latest health checks:

fly checks list

You will see two health checks were run. One is a TCP check which verifies that your application is listening for requests. The other is a HTTP check which verifies that your application is capable of producing responses.

An example of where your application might be listening for responses but not actually ready to process a request is when you have left the puma defaults at 5 threads and you currently are experiencing a spike in traffic. In cases like these you might want to review your config/puma.rb, or deploy more servers.

Another example is where you have a CDN or an nginx server handing static requests that pass or proxy the remainder to your Rails application, and your CDN or nginx server is healthy and your Rails server is, um, not.

If you like, you can add a nginx server to your deployment by running the following commands:

bin/rails generate dockerfile --nginx --force
fly deploy

An example of a successful Rails 7.1 response:

200 OK Output: <html><body style="background-color: green"></body></html>[✓]

So far we’ve only deployed our application to one region. Now lets make it interesting and scale up to two regions. I’m in the US, so I’ll deploy my second server in Europe:

fly regions add cdg
fly scale count 2

Feel free to pick another region. For demo purposes, you might want to avoid the Paid Plan Only regions.

If you rerun the fly checks list command you will see four checks now. If you run this command quickly enough you might catch it before the second server has fully started.

Also, try adding --json to the fly checks list command. This output could be processed by scripts.

Before you try this on your own project: please check the Rails FAQ for important information on how to get Fly.io to prepare your databases once per deploy and not once per server.

Behind the scenes

So far everything has been taken care of for you. But perhaps your application isn’t yet on Rails 7.1 (which is understandable as it hasn’t been released). Or perhaps you want to tweak how often health checks are run. Either way, we have you covered.

All you need to do to get a health check is to add a route, and tell us to call it.

The most simplest route will do. For example, you can add the following to config/routes.rb:

get "/up", to: proc { [200, {}, ["ok"]] },
  as: :rails_health_check

This won’t be as pretty as the Rails 7.1 with its fancy green background, but in every way that is important it will get the job done.

If you don’t like /up as the endpoint name, feel free to change it. This goes for the one that Rails 7.1 provides too.

If this route is present before you run fly launch, your application will be automatically configured to call this endpoint.

If you have already run fly launch, it is not too late to tell us to call your health check. All you need to do is add the following to your fly.toml file:

  [[services.http_checks]]
    interval = 10000
    grace_period = "5s"
    method = "get"
    path = "/up"
    protocol = "http"
    restart_limit = 0
    timeout = 2000
    tls_skip_verify = false
    [services.http_checks.headers]

More information on what each value means can be found in our reference documentation. Adjust to your tastes, and then run fly deploy.

Going beyond “/up”

Your Rails application undoubtedly does more than display a splash screen. The more moving parts you have, the more that can go wrong. You can run out of memory, or run out of disk space. You might be using a database running on another machine or hosted by a third party. You may be using redis. Or perhaps Amazon, Azure, or Google Cloud Services.

Any one of these could go down at any time. Often such outages are local or regional. If you monitor your dependencies, Fly.io will route requests around the failures until the outage is cleared.

There is nothing magical about the implementation of the “/up” route. It can be routed to any controller action in your application. All that application needs to do is produce a successful response when things are good. And return a response like the following when things are not-so-good:

render plain: "BUSY", status: :service_unavailable

Don’t worry too much about the text of the response or the status code used. :service_unavailable is good for outages that you are prepared and explicitly check for. But if your application gets to the point where it is incapable of producing a coherent response, 500 :internal_server_error works too.

Best of all, you don’t have to start from scratch writing these controllers. easymon and rails-healhcheck both will not only provide examples which check for common dependencies, they also provide means for you to add your own checks.

In many cases, adding a check is becomes a matter of adding a one liner to to a configuration file and deploying.

Neither of these gems produce routes that are currently autodiscoverable, but we have reached out to both (easymon, rails-healthcheck). In the meanwhile, you can scroll back on this page to see what you need to add to your fly.toml to tell fly.io to check these endpoints.

Recap

At this point, you’ve:

  • run a demo where an application has been deployed to two regions, and health checks are provisioned for you automatically.
  • seen how you can add health checks to an existing application and tailor parameters like how often the checks are to be called.
  • found two gems that make it easy to extend these checks to handle third party failures.

With this knowledge you are prepared to run your application in as many regions as it takes to make you comfortable that your application can survive an outage.