Making Phoenix LiveView Sing!

Let’s be real. You’re probably never going to create a 3D game in Phoenix LiveView. It’s not the best tool for that job, at least yet. However, we can still create interesting, compelling, and responsive games in LiveView!

What’s a fundamental piece of games? Sound. Yes, the simple “nom-nom” sound effects that accompanied the early Pac-Man game were an important part of the experience.

What? You don’t care about creating a web-based LiveView game? Not everyone does or should! But adding sound effects to a web application can communicate additional information. In fact, sound effects can convey meaning on their own. Have you noticed that most games even use sound effects in their settings pages? Why? Because it’s part of creating an immersive experience. It connects the user to the interface in an additional dimension.

Before we dig in, let’s see a demo of what we’re talking about.

Notice how when the count incrementing button can’t go higher, the sound emitted from a click changes, communicating more about the state and the effect of the user’s action.

Additionally, the button triggering a delayed sound demonstrates how the server can push an event from the LiveView to the browser, triggering a sound event based on the server’s state.

Now, let’s make some noise!

How do we add sound?

An obvious place to start is with the HTMLAudioElement which “provides access to the properties of <audio> elements, as well as methods to manipulate them.”

This works fine for embedding an audio file like a podcast. The browser renders play/pause controls along with playback speed settings. Nice!

But this isn’t what we want for our simple sound effects.

Turns out there’s another Web Audio API that’s better suited for our purposes. From the W3C spec’s introduction:

The introduction of the audio element in HTML5 is very important, allowing for basic streaming audio playback. But, it is not powerful enough to handle more complex audio applications. For sophisticated web-based games or interactive applications, another solution is required.

While this gives us more granular control, it’s at the cost of added complexity. We are responsible for setting up and managing an AudioContext that other sounds are associated with. That ends up being a bit of a headache.

But we only want to play a simple sound effect! Isn’t there an easier way?

Mobile audio munging

Our usage of audio sound effects is more aligned with web based games. Mozilla has some great resources regarding Audio for Web Games. One of the first things they call out is the challenges with mobile devices. Here’s what they say:

By far the most difficult platforms to provide web audio support for are mobile platforms. Unfortunately these are also the platforms that people often use to play games. There are a couple of differences between desktop and mobile browsers that may have caused browser vendors to make choices that can make web audio difficult for game developers to work with.

Gah! It’s just getting worse! How can we solve this?

JS libraries to the rescue!

Fortunately for us, there is an easier way! There are a number of JS audio libraries available that help address all this complexity.

Which is the “right” library depends on the use case. If we were building a DJ-style mixer, we’d want a different sort of audio control and manipulation. The library we’ll use for our needs is Howler.js. The project describes itself this way:

howler.js is an audio library for the modern web. It defaults to Web Audio API and falls back to HTML5 Audio. This makes working with audio in JavaScript easy and reliable across all platforms.

Phew! Sounds like it’ll do the trick!

We need sounds to play!

Before we go further, we need some sound effects to play. There are many sources of sound effects online, or we can even create them ourselves. For our simple needs, we’ll turn to the sound effects section of OpenGameArt.org.

After choosing some sounds, trimming them down tighter, and exporting them as MP3 files, we’re ready to move on to the next step.

Hook-ing up our audio

Phoenix LiveView supports JavaScript interoperability to let us connect the JS library (Howler.js), to our LiveView page.

This next part links to the demo project source code for the details.

• Add the audio files to the directory priv/static/audio.
• Add audio to the list of static paths
• Add assign_sounds to the LiveView’s mount callback

The assign_sounds function looks like this:

defp assign_sounds(socket) do
  json =
    Jason.encode!(%{
      click: ~p"/audio/button-click.mp3",
      donk: ~p"/audio/button-no-click.mp3",
      win: ~p"/audio/Jingle_Win_00.mp3"
    })

  assign(socket, :sounds, json)
end

We create a JSON object attached to :sounds that links the name of a sound to the audio file.

In our LiveView’s index.html.heex template, we link the sounds to the DOM element with the following markup:

<div
  id="settings"
  phx-hook="AudioMp3"
  data-sounds={@sounds}
>
  ...

Here, we assign the AudioMp3 hook to the page using the phx-hook attribute. Then our JSON mapped sound effects object is embedded in the data-sounds attribute. This allows the AudioMp3 JS hook to read in the mapping of sound names to actual files.

Now’s the perfect time to add and register a hook named AudioMp3! Let’s ensure we have that file added now. It reads in the JSON sounds data written to the html data attribute and creates a new Howl object for each audio file entry, pre-loading the audio file in the process.

The other important function of the hook is to let us trigger events either from JavaScript in the browser, or from a LiveView event pushed from the server. Sound effects are triggered by the name we give it, like “click” or “donk”.

To finish out the support for our hook, we’ll configure the app.js file to load and support the AudioMp3 hook.

With all of that in place, we can finally link a button’s click to a sound effect!

Tie a sound to a click event

We’re ready to attach our first sound effect! The first place we should attach our audio effect is to a button click.

Why?

The server cannot trigger a delayed sound event on the page until after the user has interacted with the site. As you can imagine, Ads would be super obnoxious if these rules didn’t exist. In fact, it was past abuses that defined how the features work today.

Here’s an explanation from Mozilla:

Many browsers will ignore any requests made by your game to automatically play audio; instead playback for audio needs to be started by a user-initiated event, such as a click or tap. ~Mozilla Audio for Web Games - Autoplay docs

So, before we start playing audio, we need to tie it to a user-initiated event. Buttons are the perfect solution.

How to trigger a sound effect?

If we had a button on our LiveView that triggered an event on the server named “start-game”, it might look like this:

<.button phx-click="start-game">
  Start Game!
</.button>

The phx-click event "start-game" will fire on the server. Using LiveView’s JS feature, we can add a locally played sound effect and still keep our server-pushed event. It looks like this:

<.button phx-click={
    JS.dispatch("js:play-sound", detail: %{name: "click"})
    |> JS.push("start-game")
  }
>
  Start Game!
</.button>

Nice! We add in a call to JS.dispatch/2 passing the sound event to fire with the details specifying the name of the sound we want to play. The JS.push/3 event then sends along the click event to the server like normal.

Slick!

This will always play the “click” sound effect. What if we want to change the sound based on the some internal state?

Using server logic based on the state of the LiveView, we can change sound effect that’s triggered like this:

defp assign_start_game_sound(socket) do
  if can_start_game?(socket) do
    assign(socket, :start_game_sound, "click")
  else
    assign(socket, :start_game_sound, "donk")
  end
end

Then in our html, we use the sound’s name from our assigns:

<.button phx-click={
    JS.dispatch("js:play-sound", detail: %{name: @start_game_sound})
    |> JS.push("start-game")
  }
>
  Start Game!
</.button>

This makes it easy for our page to change the sound effect played when a button is clicked based on the state of the LiveView (or game). Immediately our page starts communicating state and giving user feedback in a new audible dimension!

Can the server trigger a sound?

Once the user has started interacting with the page and we’ve played sounds connected to those actions, then we’re able to push sounds from the server that aren’t closely tied to a user’s action.

What kinds of sounds? When would we do that?

Here are a few reasons we might want to push a sound effect from the server:

• It’s the user’s turn (another player finished)
• The user’s report was prepared as is now available
• The user won or lost the game based on delayed internal server state

How do we play a sound effect from the server? Like this:

{:noreply, socket |> push_event("play-sound", %{name: "win"})}

At some point, the server determines a sound should be played. That might happen in a handle_info/2 event for instance. The server just needs to push the "play-sound" event, which triggers the "phx:play-sound" event in the AudioMp3 hook, which plays the named sound.

Excellent!

We’ve covered all that’s needed to add sound effects to a LiveView page. With this, we can trigger sound effects when a button is clicked or even have the sound triggered by the server.

Automatically stopping audio

The AudioMp3 hook implements the destroyed() callback, which is called when the attached DOM node is removed. This means when the user navigates away from the page playing sound effects, all sounds are stopped and the audio files are unloaded, cleaning up the resources automatically.

Discussion

Audio adds a new dimension for giving feedback to the user based on the application’s state.

Once we’ve gotten over the hump of supporting audio in our LiveView, it’s quite simple to extend and use it as desired.

Here are a few tips and things to consider as we close:

• Don’t be afraid of using sounds - Sounds offer a new dimension for giving user feedback. Explore how it might serve you and your application!
• Don’t be annoying - Don’t add sound effects that are obnoxious or don’t communicate anything of value.
• WAV files are large, compress them to MP3 files.
• Consider where audio can help communicate state or make an experience more immersive.

Happy hacking!