How to publish custom Prometheus metrics from Elixir

At work I maintain a bunch of Elixir services, and every single one of them exposes a /metrics endpoint that Prometheus scrapes. For a long time I thought that was enough. The HTTP golden metrics (request rate, error rate, latency by controller and status) were there, the BEAM metrics were there, the dashboards were green, so life was good.

Then we had an incident where a downstream dependency started answering 422 to half of our calls, and absolutely nothing on the dashboards moved. Our own service kept answering 200 to its callers, because the code handled the failure gracefully and returned a degraded response. Green dashboards, unhappy users. That’s when it clicked for me: infrastructure metrics tell you the app is alive, but they say nothing about whether the product is doing its job.

So in this article I’ll show you how to publish your own custom Prometheus metrics from an Elixir app using PromEx, the patterns I ended up using in production code, and the two traps that had me reading wrong numbers on a dashboard for weeks. Let’s dive in!

Setting up PromEx

First, add the dependency to your mix.exs:

defp deps do
  [
    {:prom_ex, "~> 1.12"}
  ]
end

Then create the PromEx module for your app. This is the entry point that declares which plugins (collections of metrics) your app publishes:

defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      PromEx.Plugins.Phoenix,
      PromEx.Plugins.Ecto
    ]
  end

  @impl true
  def dashboards, do: []
end

Add it to your supervision tree in application.ex, before the endpoint:

children = [
  MyApp.PromEx,
  MyApp.Repo,
  MyAppWeb.Endpoint
]

And add a bit of config. The defaults are already sane, so you mostly need this to disable PromEx during tests:

# config/config.exs
config :my_app, MyApp.PromEx,
  manual_metrics_start_delay: :no_delay,
  drop_metrics_groups: [],
  grafana: :disabled,
  metrics_server: :disabled

Finally, expose the endpoint. Add the plug in your endpoint.ex, above the router plug:

plug PromEx.Plug, prom_ex_module: MyApp.PromEx

Start the app, hit http://localhost:4000/metrics, and you should already see a wall of Prometheus text with your BEAM, Phoenix and Ecto metrics. That’s it for the plumbing, the interesting part starts now.

Declaring your own metrics

Custom metrics live in a plugin. A plugin is just a module that declares a list of metric definitions, and PromEx wires them to :telemetry events for you:

defmodule MyApp.PromEx.BookingsPlugin do
  use PromEx.Plugin

  @impl true
  def event_metrics(_opts) do
    Event.build(
      :bookings_event_metrics,
      [
        counter(
          [:my_app, :bookings, :created, :count],
          event_name: [:my_app, :bookings, :created],
          description: "Booking creation outcomes, by result",
          tags: [:result]
        ),
        counter(
          [:my_app, :bookings, :notification_failures, :count],
          event_name: [:my_app, :bookings, :notification_failed],
          description: "Booking notification delivery failures, by type and reason",
          tags: [:notification_type, :reason]
        )
      ]
    )
  end
end

A few things worth knowing here:

• The first argument of counter/2 is the metric name, and it becomes my_app_bookings_created_count on the exported output. The parts are just joined with underscores.
• event_name is the :telemetry event that feeds the metric. It doesn’t have to match the metric name, and several metrics can listen to the same event (we’ll use that in a moment).
• tags are the Prometheus labels. Every tag you declare must be present in the event metadata, otherwise the sample gets dropped.
• use PromEx.Plugin imports counter/2, distribution/2, last_value/2 and sum/2 from Telemetry.Metrics, so you don’t need to import anything yourself. This is how each one lands on the Prometheus side:
  • counter/2 becomes a counter. It counts events and ignores the measurements entirely.
  • distribution/2 becomes a histogram, and it’s the only one that takes buckets.
  • last_value/2 becomes a gauge, holding the last measurement it saw.
  • sum/2 adds up a measurement, and is exported as a counter by default. If your value can also go down, pass reporter_options: [prometheus_type: :gauge], because a Prometheus counter is expected to only ever increase.

Then register the plugin in your PromEx module:

   def plugins do
     [
       PromEx.Plugins.Application,
       PromEx.Plugins.Beam,
       PromEx.Plugins.Phoenix,
-      PromEx.Plugins.Ecto
+      PromEx.Plugins.Ecto,
+      MyApp.PromEx.BookingsPlugin
     ]
   end

Emitting the metrics from your code

With Telemetry.Metrics you never touch the metric directly. You emit a :telemetry event, and whoever is listening (PromEx, in this case) decides what to do with it:

:telemetry.execute([:my_app, :bookings, :created], %{count: 1}, %{result: "success"})

The three arguments are the event name, the measurements, and the metadata (which is where your labels come from). A counter ignores the measurements and just counts events, so %{} would work too, but I like passing %{count: 1} because it makes the call site readable.

In production code I’ve seen three shapes of this, and all of them are fine depending on the case:

Option 1: One clause per outcome

The most direct one. A private function with a clause per result, called from wherever the outcome is known:

defp count_booking_created({:ok, _booking} = result) do
  :telemetry.execute([:my_app, :bookings, :created], %{count: 1}, %{result: "success"})
  result
end

defp count_booking_created({:error, %Ecto.Changeset{}} = result) do
  :telemetry.execute([:my_app, :bookings, :created], %{count: 1}, %{result: "validation_error"})
  result
end

defp count_booking_created({:error, _reason} = result) do
  :telemetry.execute([:my_app, :bookings, :created], %{count: 1}, %{result: "error"})
  result
end

Note how result is returned unchanged in all clauses. Instrumentation must never change the value that flows through it. If your metric code can alter the return value of a function, sooner or later it will.

Option 2: A facade module

Once you have more than two or three metrics, the event names start to spread all over the codebase, and your application services end up knowing about Prometheus. I prefer to hide them behind a small facade:

defmodule MyApp.Bookings.Metrics do
  @moduledoc """
  Facade over the booking business metrics.

  Keeps the application services free of Prometheus and telemetry details.
  Labels here are low cardinality and PII free.
  """

  @typedoc "Display-only field that degraded to nil on a failed lookup."
  @type enrichment_field :: :company_name | :category | :route_map

  @spec report_enrichment_degraded(enrichment_field()) :: :ok
  def report_enrichment_degraded(field) do
    :telemetry.execute(
      [:my_app, :bookings, :enrichment_degraded],
      %{count: 1},
      %{field: to_string(field)}
    )
  end
end

And the call site reads like domain code instead of instrumentation code:

defp company_name(%Booking{requester_id: requester_id}) when is_binary(requester_id) do
  with {:ok, %Account{client_id: client_id}} <- Accounts.get_account(requester_id),
       {:ok, name} <- Clients.get_client_name(client_id) do
    name
  else
    _other ->
      Metrics.report_enrichment_degraded(:company_name)
      nil
  end
end

This is exactly the incident I mentioned at the beginning. A page served with a nil company name still returns 200, so the golden HTTP metrics can’t see it. This counter is the only signal that something is degraded.

Option 3: A pipe friendly tap

When the thing you want to count is the last step of a pipeline, a tap-style function keeps it readable:

defp tap_count({:ok, _value} = result, metric_name) do
  :telemetry.execute([:my_app, metric_name], %{count: 1}, %{result: "success"})
  result
end

defp tap_count(result, metric_name) do
  :telemetry.execute([:my_app, metric_name], %{count: 1}, %{result: "error"})
  result
end

Which you then use like this:

opts
|> connection()
|> get("/billing/invoices/#{invoice_id}")
|> ResponseHandler.unwrap(file_name)
|> tap_count(:downloads)

Instrumenting calls to external dependencies

This is where I got the most value out of custom metrics. Every call to another service (HTTP, gRPC, whatever) goes through a single wrapper that emits one event, and that event feeds both a counter and a latency histogram.

First, the metric definitions. Note how the two of them share the same event_name:

defmodule MyApp.PromEx.ClientsPlugin do
  use PromEx.Plugin

  @impl true
  def event_metrics(_opts) do
    Event.build(
      :client_call_event_metrics,
      [
        counter(
          [:my_app, :client, :call, :count],
          event_name: [:my_app, :client, :call, :stop],
          description: "Wrapped client dependency calls, by client, operation and result",
          tags: [:client, :operation, :result]
        ),
        distribution(
          [:my_app, :client, :call, :duration, :milliseconds],
          event_name: [:my_app, :client, :call, :stop],
          measurement: :duration,
          description: "Latency of wrapped client dependency calls in milliseconds",
          tags: [:client, :operation, :result],
          unit: {:native, :millisecond},
          reporter_options: [buckets: [10, 50, 100, 250, 500, 1_000, 5_000, 10_000]]
        )
      ]
    )
  end
end

And then the wrapper itself:

defmodule MyApp.Infra.ClientInstrumentation do
  @moduledoc """
  Wraps a call to an external dependency and emits, per invocation, an
  invocation counter, a latency histogram and a structured log line.

  Returns `fun`'s result unchanged. On an exception, it records the error and
  re-raises with the original stacktrace.

  `operation` names the client call being wrapped (for example `:fetch_by_email`)
  and becomes a low cardinality label.
  """

  require Logger

  @event [:my_app, :client, :call, :stop]

  @spec instrument(atom(), atom(), (-> result)) :: result when result: term()
  def instrument(client, operation, fun) when is_atom(client) and is_function(fun, 0) do
    start = System.monotonic_time()

    try do
      fun.()
    rescue
      exception ->
        emit(client, operation, "exception", System.monotonic_time() - start)

        Logger.error("client call raised",
          client: client,
          operation: operation,
          error: Exception.message(exception)
        )

        reraise exception, __STACKTRACE__
    else
      value ->
        result = classify(value)
        emit(client, operation, result, System.monotonic_time() - start)
        value
    end
  end

  defp emit(client, operation, result, duration) do
    :telemetry.execute(@event, %{duration: duration}, %{
      client: to_string(client),
      operation: to_string(operation),
      result: result
    })
  end

  defp classify(:ok), do: "ok"
  defp classify({:ok, %Tesla.Env{status: status}}), do: http_result(status)
  defp classify({:ok, status, _body}) when status in 100..599, do: http_result(status)
  defp classify(result) when is_tuple(result) and elem(result, 0) == :ok, do: "ok"
  defp classify(_other), do: "error"

  defp http_result(status) when status < 400, do: "ok"
  defp http_result(_status), do: "error"
end

Wrapping a call is then a one liner, and you get the rate, the error ratio and the latency percentiles of every dependency for free:

ClientInstrumentation.instrument(:accounts, :get_account, fn ->
  AccountsClient.get(account_id)
end)

Those two little modules are where both of the traps I mentioned live, so let’s look at them.

The two traps I hit in production

Trap 1: durations are in native units

If you look at the wrapper again, you’ll see that I never convert the duration. I subtract two System.monotonic_time() calls and pass the raw difference, which is in native time units (the BEAM decides what those are, usually nanoseconds). The conversion to milliseconds happens in the metric definition, with unit: {:native, :millisecond}.

The first version of this code did the conversion by hand, and declared the unit too. So the value got converted twice, and every single call landed in the lowest bucket. The histogram was beautiful and completely fake, with a _sum that had been rounded down to zero. The exact same trap exists on prometheus.erl, which reads the unit from the _seconds name suffix and converts your bucket bounds to native units behind your back, so the observation has to arrive in native units as well.

The rule that got me out of it is easy to remember: measure in native units, declare the conversion, and write your buckets in the converted unit. Do the conversion in exactly one place.

Trap 2: an :ok tuple is not a successful HTTP call

This one is subtler, and it’s the reason the incident I opened with stayed invisible for so long.

For gRPC stubs, a failure comes back as {:error, %GRPC.RPCError{}}, so matching on the :ok tag is enough. HTTP is the opposite: a completed roundtrip is an :ok tuple whatever the server answered, with the status carried inside it. You get {:ok, %Tesla.Env{status: 422}} with raw Tesla, and {:ok, 422, body} with most generated OpenAPI clients.

So a naive classify/1 that only checks the tuple tag reports result="ok" for every failing dependency call in your system. The dashboards stay flat, the alerts never fire, and the only thing that tells you something is wrong is a customer complaining.

That’s why classify/1 reads the status out of both shapes, and treats anything >= 400 as an error. Including the 4xx that your code later maps to a legitimate domain outcome (a 404 becoming {:error, :not_found}, for example). The dependency did not return what we asked for, and the label should say so. If you need to separate expected 404s from real failures, add the status code as a label instead of lying with the result one.

Keep your labels boring

Two rules I’d tattoo on every metric spec module:

• Low cardinality. Every distinct combination of label values creates a new time series in Prometheus. A user_id or an email label is how you take down your monitoring stack. Stick to values from a small closed set: a result, a reason, an operation name, a status code.
• No PII, ever, and no secrets either. Metric endpoints get scraped, stored and shipped around, and they’re usually much less protected than your database. If a value is a secret (a public token used to access a page, for example), it must not appear as a label.

There’s also a third one that I learned by deleting code: don’t declare a metric that nothing emits. Empty series are worse than no series, because someone will eventually build a dashboard panel or an alert on top of them and read the silence as good news.

Testing your metrics

Two tests are enough to catch most regressions here.

The first one asserts that your plugin declares what you think it declares. It looks silly, but it’s what catches the accidental deletion of a metric that an alert depends on:

defmodule MyApp.PromEx.BookingsPluginTest do
  use ExUnit.Case, async: true

  alias MyApp.PromEx.BookingsPlugin
  alias PromEx.MetricTypes.Event

  test "declares the booking creation counter with a :result label" do
    %Event{metrics: metrics} = BookingsPlugin.event_metrics([])

    metric = Enum.find(metrics, &(&1.name == [:my_app, :bookings, :created, :count]))

    assert metric.tags == [:result]
    assert metric.event_name == [:my_app, :bookings, :created]
  end
end

The second one goes end to end through the endpoint, and proves that the whole chain (event, plugin, registry, plug) is actually wired:

defmodule MyAppWeb.MetricsTest do
  use MyAppWeb.ConnCase, async: false

  test "GET /metrics exposes the booking metrics" do
    :telemetry.execute([:my_app, :bookings, :created], %{count: 1}, %{result: "success"})

    conn = get(build_conn(), "/metrics")

    assert conn.status == 200
    assert conn.resp_body =~ "my_app_bookings_created_count"
  end
end

Remember to keep PromEx enabled in your test config if you want that second test to run (disabled: false), or start it only for that test file.

Bonus: what to actually do with them

Publishing the metric is only half of the job. Once it’s scraped, this is where it pays off:

• Alerts. A counter with a result label gives you an error ratio for free, which is the classic SLO shape. Something like sum(rate(my_app_bookings_created_count{result="error"}[5m])) / sum(rate(my_app_bookings_created_count[5m])). Then alert on multi-window burn rates rather than on a raw threshold, so a two minute blip doesn’t wake anybody up.
• Dashboards. The histogram gives you histogram_quantile(0.99, sum by (le, client, operation) (rate(my_app_client_call_duration_milliseconds_bucket[5m]))), which is the panel I check first when something feels slow.
• Business reporting. A counter of successful signups, bookings or payments per day is a business number your product people will genuinely use, and it costs you one line of code.

Every metric you add should have an answer to “what decision will I make with this?”. If you don’t have one, don’t add it.

Wrapping up

And that’s how I publish custom Prometheus metrics from Elixir. The setup with PromEx is honestly the easy part (a dependency, a module, a plug), and you can have it running in an afternoon. The hard part is choosing what to measure, keeping the labels boring, and making sure the numbers you export are the numbers you think you’re exporting.

If you take only one thing from this article, take the two traps. Measure durations in native units and convert once, and never trust an :ok tuple to mean that an HTTP call went well. Both of them produce dashboards that look perfectly healthy while your service quietly fails, and that’s the worst kind of bug in observability code.

Now go and add that one counter you’ve been postponing. Until next time!