BEAM Clustering made easy

One of the reasons Fly.io is so great for Elixir and Phoenix is that we have servers in nearly every region on the globe, and they are all connected via built in WireGuard networking. As an Elixir developer, this is an incredible opportunity because it means we get to use Erlang Distribution and clustering with minimal setup, globally.

Distribution?

The history of the BEAM is basically to handle as many concurrent phone calls as possible, and route them to the correct place. So being able to connect to other nodes and send messages transparently from node to node is a necessity.

Without any third party tools you can simply call Node.connect/1 with a host name and ip for example:

Node.connect(:"name@127.0.0.1")

And assuming nothing fails you’ve connected the two BEAM. Send messages, start processes on either machine, the world is your oyster!

Now the sharp readers might be asking…

But if we run on a globally distributed cloud, how do you know which machines to connect to?

Fly.io Private Networking

Fly apps are connected by a mesh network of WireGuard tunnels using IPV6. And in order to set up those tunnels, Fly needs to know, which machines to connect to… which is provided by a Fly .internal addresses. If you do a AAAA DNS query against app_name.internal you will get back a list of nodes to connect to. For example, if we only had one app:

:inet_res.lookup(~c"my_app.internal", :in, :aaaa)
|> Enum.map(&to_string(:inet.ntoa(&1)))

["fdaa:1:36c9:a7b:198:c4b1:73c6:1"]

Adding a second region is as simple as cloning an existing Fly Machine to another region.

fly machine clone MACHINE_ID --region NEW_REGION

Once setup the above DNS lookup will return two IPV6.

To scale up multiple machines with in a region it’s as simple as

fly scale count 2

Making it Easy

The Phoenix Team has released a new project called dns_cluster which aims to simplify working with clustering in the case where you use DNS for discovery. I highly recommend you read the code to see what it does, it is almost unnecessary to make into a library for a paltry 191 lines of code.

Simply follow the README and add one line to your application.ex and another to runtime.exs and you are off to the races! You can verify that you are connected to another node by calling, Node.list/0 and you’re off.

Wrap Up

There are many ways to connect a bunch of nodes and the go-to library for more complex setup’s is the fantastic libcluster from Paul Schoenfelder (Bitwalker). Using libcluster you can do zookeeper or multicast UDP or K8s, and he even has a DNS strategy as well.

We’ve just started scratching the surface of what’s possible:

• Dig into the OTP modules like global or erpc
• Try out mnesia the distributed DBMS that ships with OTP
• Fly.RPC makes it really easy to execute functions on specific nodes
• Try out Phoenix PubSub!