Adding Dialyzer without the Pain

This post is about how to start using Dialyzer on an established Elixir project without losing your mind in the process. If you’ve been in the Elixir community for any length of time, you’ve probably heard about Dialyzer. It’s tool that does static code analysis to help spot bugs in our application before it goes to production. Dialyzer can also be finicky and the reported problems can be difficult to interpret. This all results in some mixed feelings about Dialyzer in the community.

Here, we assume you want the benefits of type checks but you don’t want to stop the project to clean up the 100s of legacy issues.

We’ll start with a little background, you can jump ahead and get started, if you prefer.

Why Check Types?

Erlang and Elixir are dynamically-typed, so why should we care about types? The nice part is that you aren’t strictly required to care about types at all. If you don’t want to write type definitions and check them with external tools, you don’t have to. However, many developers find some form of type specification useful as a form of documentation and a tool for preventing bugs. Luckily for us, Erlang and Elixir programmers are spoiled for choice when it comes to tooling that can help us through the use of types without needing to run our code first. The rest of this post will be an analysis of those tools as well as an example of how to implement a common Erlang/Elixir type-checking tool: Dialyzer.

The First Poll

A year ago, there was a poll posted to the Elixir Forum asking whether or not people used Dialyzer in their projects. Of the 207 votes (at the time of this post), about 72% of them used Dialyzer for at least some of their projects. The poll replies were where the real meat of the subject was; some highlights:

I find dialyzer to work well for projects, which are deployed. I tend to not like it for libraries, which don’t really use their code, because that usually means dialyzer doesn’t find half of the issues. - LostKobrakai

I have to admit that I do have quite a love/hate relationship with the Dialyzer. It does really catch potential errors, but the error messages can be really difficult to understand, especially when largish structs are involved. - xpg

No. Too slow, too cryptic. Hoping for something like Gradualizer to be usable. In the meantime, I invest time in end to end tests. - stefanchrobot

Dialyzer as it exists today is a net negative value add for me. I only use it if forced. In principle it checks out, but in practice it always costs more than it saves. - chrismccord (Creator of Phoenix)

Adding it into any project of reasonable size is slow, sometimes unsustainably so. - Qqwy (TypeCheck Core Team)

Admittedly, I’ve cherry-picked portions of responses that are critical of Dialyzer, but only to point out that Dialyzer is a controversial tool among Elixir developers.

The Second Poll

In order to get more opinions, I created a followup poll asking what tool (if any) people preferred to typecheck their Elixir/Erlang code. Poll options included:

• Dialyzer/Dialyxir
• eqWAlizer
• TypeCheck
• Gradualizer
• Gradient (built using Gradualizer)
• Other

In hindsight, I should have also added Gleam as an option.

As you can probably tell, we are fortunate to be spoiled for choice when it comes to type-checking in the Erlang/Elixir ecosystem, though of the 107 votes (at the time of this post), about 65% listed Dialyzer/Dialyzir as a typechecking tool they used with Elixir/Erlang. This seems pretty consistent with the previous poll. There weren’t as many replies as the original poll, but it still gave a good impression of the Elixir/Erlang type-checking landscape.

The Inspiration

Six months before making the followup poll, I watched a talk titled “Slaying the Type Hydra, or How We Went from 12,000 Dialyzer Errors to None”. In it, Jesper Eskilson describes how Klarna added Dialyzer into a preexisting system and how (at a high level) they integrated it into their CI and review process in order to steadily reduce their Dialyzer error count to zero. It was not an effortless/painless journey, but it was encouraging. I set out to try the same thing within my own company.

Adding Dialyzer to an Existing Project

As an exercise, let’s try to add Dialyzer (more specifically, Dialyxir) to some existing Elixir code. Remember that even though we may run mix dialyzer, we’re actually using Dialyxir as a convenience for Elixir projects. We’ll test out this process using a few large, well-known projects to see what we can learn: (ordered by GitHub stars at the time of this post)

• Phoenix
• Plausible Analytics
  • Plausible already uses Dialyzer, good on them!
• Supabase Realtime
  • Supabase already uses Dialyzer, good on them!
• Firezone
  • Firezone already uses Dialyzer, good on them!
• Livebook
• Changelog
• Membrane
  • Membrane already uses Dialyzer, good on them!

Three seems like a good exercise, no? We’ll add Dialyzer To Phoenix, Livebook, and Changelog.

Add The Dialyxir Dependency

This is the easy part:

# mix.exs
defp deps do
  [
    {:dialyxir, "~> 1.3", only: [:dev], runtime: false},
  ]
end

Followed by: mix do deps.get, deps.compile

Run Dialyzer

At this point, you’d normally run mix dialyzer, which would both generate your Dialyzer PLT files (more on those later) and then typecheck your project with them. We’re going to split up this step slightly for learning purposes.

Generating PLTs

Running mix dialyzer --plt only generates the required PLT files to do typechecks, so let’s try that. Note that the times listed below are only for the time taken to generate the project PLTs, not the core Erlang/Elixir PLTs, as those are cached. This is not a performance benchmark by any means. Faster PLT generation does not equal better performance or higher-quality code. It’s dependent on the application, its dependencies, how many extra_applications it has configured, etc.

Time taken: (all times are done on the same machine, a Macbook Pro with an M2 Pro)

• Phoenix: done in 0m18.76s
• Livebook: done in 0m29.42s
• Changelog: done in 0m38.99s

Now we can run mix dialyzer to perform the actual typecheck and see how many errors we find!

Phoenix:

Total errors: 173, Skipped: 0, Unnecessary Skips: 0
done in 0m0.84s
# ...a bunch of errors printed here...
done (warnings were emitted)
Halting VM with exit status 2

Livebook

Total errors: 27, Skipped: 0, Unnecessary Skips: 0
done in 0m3.37s
# ...a bunch of errors printed here...
done (warnings were emitted)
Halting VM with exit status 2

Changelog:

Total errors: 74, Skipped: 0, Unnecessary Skips: 0
done in 0m3.2s
# ...a bunch of errors printed here...
done (warnings were emitted)
Halting VM with exit status 2

Not bad, honestly! The first few work projects I ran this on had upwards of 400 errors. Regardless, these numbers can sometimes seem daunting. Who wants to fix 173 errors all at once? In my experience, no one. How can we make these errors easier to deal with? By ignoring them!

Ignoring Existing Errors

In the previous step, you may have noticed the “skips” and “unnecessary skips” given when running Dialyzer. This is actually a special feature of dialyxir that allows us to selectively ignore certain errors using a special flag to mix dialyxir: --format ignore_file. See the dialyxir docs on the subject for more details, but essentially all this does is generate one line for each error that already exists in a project. You can put these lines in a .dialyzer_ignore.exs file to make subsequent mix dialyzer runs ignore that particular error.

So, let’s try it out for each of our projects using mix dialyzer --format ignore_file | wc -l to count how many lines are generated.

• Phoenix: 56
• Livebook: 15
• Changelog: 62

You’ll notice that the number of ignore lines doesn’t match the number of total errors from the previous step. This is because --format ignore_file will generate one line per error per entire file, not one line per specific error.

In the next release of dialyxir, a new --format ignore_file_strict will be added that improves on the original --format ignore_file pattern by allowing you to ignore specific errors in each file, even if the same error occurs multiple times in a file.

So, let’s copy each project’s ignore lines into a new file with a single list: .dialyzer_ignore.exs (you can configure the name of this file, but the default is fine for us)

[
  # line 1...
  # line 2...
  # line 3...
  # ...
]

Rerunning mix dialyzer gives us:

Phoenix:

Total errors: 173, Skipped: 173, Unnecessary Skips: 0
done in 0m0.88s
done (passed successfully)

Livebook:

Total errors: 27, Skipped: 27, Unnecessary Skips: 0
done in 0m3.48s
done (passed successfully)

Changelog:

Total errors: 74, Skipped: 74, Unnecessary Skips: 0
done in 0m3.19s
done (passed successfully)

Hooray, the task passed successfully! Let’s commit our changes, push, and make a PR! We’ll talk more about configuring our CI to check for Dialyzer errors, but for now let’s assume that’s set up for us.

Preventing New Errors

With our new type-checking in place, let’s say someone makes a PR with an incorrect typespec on a new function in Phoenix:

# lib/phoenix.ex

# Here our `@spec` says the function's return type is an integer:
@spec my_cool_func(integer()) :: integer()
def my_cool_func(an_integer) do
  new_integer = an_integer + 1

  # But the function is actually returning a string!
  Integer.to_string(new_integer)
end

Let’s see what happens when we run mix dialyzer again:

Total errors: 174, Skipped: 173, Unnecessary Skips: 0
done in 0m0.87s
lib/phoenix.ex:61:invalid_contract
The @spec for the function does not match the success typing of the function.

Function:
Phoenix.my_cool_func/1

Success typing:
@spec my_cool_func(integer()) :: binary()

_________________________________________________________________________
done (warnings were emitted)
Halting VM with exit status 2

We still ignore the other 173 errors, but now we’ve caught the new error!

Fixing Errors

Ignoring all our current errors is a great way to start using Dialyzer/Dialyxir, but eventually we’ll want to fix them. Let’s say we decide to ignore the above error by adding {"lib/phoenix.ex", :invalid_contract}, to the .dialyzer_ignore.exs file for the time being. Then we fix the error:

@spec my_cool_func(integer()) :: String.t()
def my_cool_func(an_integer) do
  new_integer = an_integer + 1
  Integer.to_string(new_integer)
end

We’ll run mix dialyzer again and see:

Total errors: 173, Skipped: 173, Unnecessary Skips: 1
done in 0m0.89s
done (passed successfully)

Notice that we now have an “unnecessary skip”, which is true because the previously ignored error is no longer an error. We can safely remove the new line we added in .dialyzer_ignore.exs without worry. If we had left that line in the ignore file, we could have accidentally reintroduced the error without causing mix dialyzer to fail. Dialyxir gives us the ability to prevent these unnecessary skips, forcing us to remove fixed errors from the .dialyzer_ignore.exs file in order to pass type checks.

# mix.exs
def project do
  [
    # ...
    dialyzer: [
      list_unused_filters: true
    ],
    # ...
  ]
end

Now when we have an unnecessary skip, mix dialyzer gives us the following error:

Total errors: 173, Skipped: 173, Unnecessary Skips: 1
Unused filters:
{"lib/phoenix.ex", :invalid_contract}
unused filters present
Halting VM with exit status 1

This is a slick way to prevent previously-fixed errors from sneaking back!

What Next?

Configuring CI

The next logical step is adding a Dialyzer check to our CI so that any PRs with new type errors are flagged and fixed before merging. The main concern is making sure that PLTs are cached properly between CI runs.

There is a section of the Dialyxir docs that explains how to configure various CI tools to work well with Dialyxir. And there’s the Fly GitHub Actions for Elixir CI/CD guide as well. We won’t go into too much other detail here.

Once we have Dialyzer integrated into our CI pipelines, we can prevent new type errors from showing up as well as starting to chip away at our .dialyzer_ignore.exs file by fixing existing errors!

When In Doubt, Ignore The Error

We’ve talked about how to ignore existing Dialyzer errors and prevent new ones from being added to a project, but it’s worth mentioning that ignoring new errors is completely valid as well. Sometimes a new error shows up (or you are trying to fix an existing ignored error) that neither you nor your colleagues can figure out. It might involve a massive nested struct or a series of complex function calls. The error message might span multiple pages of your terminal. Regardless, the purpose of a tool like Dialyzer is to improve developer experience/efficiency. If a tool starts to become more painful than helpful, escape hatches like Dialyzer’s .dialyzer_ignore.exs file are perfectly valid options. At the very least, adding a new error to your ignore file is a form of documentation of tech debt, not a failure on you or your colleagues’ part.

Types cannot catch every single bug or error; tests (unit, integration, or otherwise) are important as well, though they aren’t perfect either. We can raise our confidence in our code by combining types and tests; our goal should be productivity, not perfection!

What About Incremental Mode?

In 2022, Thomas Davies gave a talk titled “Incremental Dialyzer: How we made Dialyzer 7x Faster”. This new incremental mode for Dialyzer was released with OTP 26, and should drastically simplify and speed-up the addition of Dialyzer to large existing projects. It’s still super new, so few people (especially in the Elixir community) have had a chance to experiment with it. There is a GitHub issue about supporting this new --incremental mode in Dialyxir, but the work hasn’t been started yet.

This is an exciting enhancement that we have to look forward to!

Recommended Dialyzer Resources

You’ll notice that this post doesn’t cover “how to understand/fix Dialyzer errors”, as that is an entire topic worthy of its own post. Luckily, many people have written blog posts and given talks about this very topic. Here a collection of resources:

• Devon Estes - Decoding Dialyzer
• Brujo Benavides - Help Dialyzer Help You!
• Sean Cribbs - Chemanalysis: Dialyzing Elixir | Code BEAM SF 19
• Stavros Aronis - What does Dialyzer think about me? | Code BEAM STO 19

Aside: Language Servers

Up until now, all of our talk of type checking tools like Diayzer has involved running commands in our terminal, but many other programming languages often have tooling that will allow developers to interact with types and type-related errors in their favorite IDE or text editor. This is done using language servers. We won’t go too into depth, but a language server allows modern editors to hook into programming languages in order to provide things like intellisense, autocompletion, and type checking. These language servers are usually accompanied by an editor-specific extension to make use of the information they provide. Luckily, Erlang and Elixir both have solid options for language servers and extensions.

Erlang has Erlang LS, and many Elixir developers are familiar with ElixirLS. Both of these tools run Dialyzer in an incremental way. ElixirLS uses both Dialyzer and Dialyxir, but the latter is only used for formatting Dialyzer errors in a more readable fashion. ElixirLS stores its project PLT in .elixir_ls/dialyzer_manifest_$ERLANG_VERSION_elixir-$ELIXIR_VERSION_test file (without a .plt extension); if you ever run into issues with ElixirLS type errors not updating correctly, you can try to delete the PLT.

It’s also worth noting that alternative Elixir language servers exist, most notably Lexical and NextLS. Both Lexical and NextLS will support Dialyzer type-checking in the future; we look forward to seeing these tools develop!

Conclusion

We’ve seen how we can add Dialyzer late in the game to an Elixir project without getting overwhelmed. Tools like Dialyxir let us ignore all the legacy issues in our project today while helping us to keep it clean going forward. Hopefully we can keep in mind that our goal is for productivity, not perfection, and that it’s okay to ignore a type error so we can move forward today. We also touched on some interesting projects that may further improve our developer experience in future.

Finally, there is an ongoing research project by Giuseppe Castagna and Guillaume Duboc (and accompanying talk from ElixirConf EU 2023 by the project authors) looking to add a gradual type system to Elixir. That may change everything! Until that time, Dialyzer/Dialyxir can do a lot for our projects today. Hopefully we also saw that adding Dialyzer doesn’t have to be scary either.

Acknowledgements

• Everyone who participated in both of the mentioned Elixir Forum polls.
• Jeremy Huffman for maintaining Dialyxir, being very accommodating to my various issues/PRs, and reviewing this blog post.
• Jesper Eskilson for his talk, which was the catalyst in my Elixir type-checking quest.