Speed up your boot times with this one Dockerfile trick

Problem

You added Whisper text to speech transcription to your voice chat app and while it works great… the boot times on your deployment have slowed to a crawl. Or maybe you really love deploying your apps using single file elixir scripts but are finding the initial boot to be a little slower than you’d think. The common thread here is that they both will download and compile some dependencies at startup for you, what if we could do that in the build step?

Solution

The solution is actually pretty straightforward! In our Dockerfile we simply need to add a step where it runs this code and then stops. This will cause the files to be cached and available instantly at boot time. So let’s start with the Single File Elixir Scripts example:

Mix.install([
  :req,
  {:jason, "~> 1.0"}
])

# Dry run for copying cached mix install from builder to runner
if System.get_env("EXS_DRY_RUN") == "true" do
  System.halt(0)
  Process.sleep(:infinity)
else
    Req.get!("https://api.github.com/repos/elixir-lang/elixir").body["description"]
    |> dbg()
end

Here we added an environment check for “EXSDRYRUN”, and if so we halt, otherwise we do the thing we wanted to spin up a docker container for in the first place!

Now lets see what changes we need to make to our Dockerfile.

# syntax = docker/dockerfile:1
FROM "hexpm/elixir:1.14.2-erlang-25.2-debian-bullseye-20221004-slim"

# install dependencies
RUN apt-get update -y && apt-get install -y build-essential git libstdc++6 openssl libncurses5 locales \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

WORKDIR "/app"

# Copy our files over
COPY run.exs /app

# install hex + rebar if you plan on using Mix.install
RUN mix local.hex --force && \
    mix local.rebar --force

# force elixir to install your deps
ENV EXS_DRY_RUN=true
RUN elixir /app/run.exs

ENV EXS_DRY_RUN=false
CMD elixir /app/run.exs

What we’ve done here is run execute our code with EXS_DRY_RUN set to true during the docker build step, this will cache our dependencies in the docker container it’s self.

Just a couple extra lines and we’re good to go!

Bumblebee

Time for a more complex example using LiveBeats. Chris McCord semi-recently pushed an update that will transcribe your audio using bumblebee and the whisper model in real time. To set that up he has a function called load_serving that will configure, download and set up the model for you! Here is the code:

def load_serving do
    {:ok, whisper} = Bumblebee.load_model({:hf, "openai/whisper-tiny"})
    {:ok, featurizer} = Bumblebee.load_featurizer({:hf, "openai/whisper-tiny"})
    {:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "openai/whisper-tiny"})

    Bumblebee.Audio.speech_to_text(whisper, featurizer, tokenizer,
        compile: [batch_size: 10],
        max_new_tokens: 100,
        defn_options: [compiler: EXLA]
    )
end

He calls this during application start and the deploys take a super long time because the application won’t fully boot till this is ready. The models are being downloaded from HuggingFace directly and then compiled into EXLA on start. To fix this, we simply call this function in our Dockerfile and it will cache the model , so it won’t download on boot. The one downside about this is that our docker image will get significantly larger depending on the size of the model.

Below is the full Dockerfile from LiveBeats:

ARG BUILDER_IMAGE="hexpm/elixir:1.12.0-erlang-24.0.1-debian-bullseye-20210902-slim"
ARG RUNNER_IMAGE="debian:bullseye-20210902-slim"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"
ENV BUMBLEBEE_CACHE_DIR=/app/.bumblebee

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

# Compile the release
COPY lib lib

# note: if your project uses a tool like https://purgecss.com/,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets

# compile assets
RUN mix assets.deploy

RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

# NEW HERE
# Download the HuggingFace models to cache them
RUN /app/bin/live_beats eval 'LiveBeats.Application.load_serving()'

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8


WORKDIR "/app"
RUN chown nobody /app

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/.bumblebee/ ./.bumblebee
COPY --from=builder --chown=nobody:root /app/_build/prod/rel/live_beats ./

USER nobody

# Set the runtime ENV
ENV ECTO_IPV6="true"
ENV ERL_AFLAGS="-proto_dist inet6_tcp"
# NEW HERE
ENV BUMBLEBEE_CACHE_DIR=/app/.bumblebee
ENV BUMBLEBEE_OFFLINE=true

CMD /app/bin/server

The magic lines are highlighted with # NEW HERE mainly during the build stage we setup an environment variable for where Bumblebee should put our cached models, execute LiveBeats.Application.load_serving(),this will run our code without booting up the entire application, copy the those files into our runner and set ENV BUMBLEBEE_OFFLINE=true which makes sure that our app will still run even if HuggingFace goes down using the cached models.

And with that single line, we’ve done it! Now you have your cache built into your docker container!

Discussion

We showed two examples moving a boot-time step to a build-time step. This can be exceptionally handy when we deploy to many regions or want a fast loading application. In the case of BumbleBee, we also remove a risk that we won’t be able to boot if HuggingFace is not reachable.

Try to identify work that is happening on boot, and if its cached to the file system. If it is, move that work to the build step if possible!