imetrics is a lightweight library to allow you to easily instrument your Erlang applications, and expose the data through a consistent HTTP interface.
- No R16 support: We require the "newer" ets functions such as ets:update_counter/4.
- cowboy: The
cowboy
application (and its dependencies) must be started before you startimetrics
cowboy
depends onranch
,cowlib
,ssl
, andcrypto
(see here)
Build, launch, and run Erlang and imetrics:
rebar3 shell
To test imetrics:
rebar3 eunit
Next, decide if you want a counter or a gauge.
A counter is an integer that has the nice propertry of being monotonically increasing while imetrics is running. Why is this nice? Well, if your code is event-driven, like many Erlang apps, it's very easy to instrument counting such events. For instance, each time a client connects, we can increment a counter.
imetrics:add(client_connections).
Counters are also nice because they do not hide local extrema when quantized. In other words, the general shape of the time series is not hidden by a large delta-time (low resolution data).
imetrics also supports tags on counter metrics, which allow you to bundle many related counters under one named entry. When used in an http server, we can count the number of each HTTP response code.
imetrics:add(http_responses, #{ code => "200" }),
imetrics:add(http_responses, #{ code => "404" }).
A gauge is a number, any number. You are in charge of what value it's set to.
imetrics:set_gauge(velocity, 50.5).
And there are also tagged gauges.
imetrics:set_gauge(cpu_load_avg, #{ granularity => '1min' }, 0.1),
imetrics:set_gauge(cpu_load_avg, #{ granularity => '5min' }, 3.4).
Often, a gauge is a calculation based on some counter over time. For example,
consider tracking miles_driven
instead of velocity
. You may find that your
metrics are more flexible down the road.
imetrics
provides a module to help manage the collection of a gauge metric that occurs
across a set of Erlang processes. Please see the imetrics_actors
module for documentation
and an example.
Histograms are a way to take a continuous set of data and turn it into discrete "buckets," counting the number of elements that occur in a given range. Any value must be able to be accepted by a histogram, so the minimum bucket will count any elements <= to the value stored, and an additional "infinite" bucket will be added on top of any defined buckets to capture any values greater than the maximum value defined. Buckets can be defined manually by providing a list of cutoffs, or generated automatically, with an even distribution, by providing a min, max, and number of buckets. The maximum number of buckets a histogram can have is limited by an environment variable (see below).
%manually define bucket cutoffs
imetrics_hist_openmetrics:new(http_response_time, [0, 0.01, 0.05, 0.1, 0.2, 1, 10]),
%automatically generate evenly distributed cutoffs (in this case, 0, 4, 8, ..., 1000)
imetrics_hist_openmetrics:new(packet_size, [0, 1000], 251).
Histograms are incremented by indicating a continuous value, which will add one to the bucket in which that value would fall.
%using the buckets from above, add one to the 0.05 bucket
imetrics_hist_openmetrics:add(http_response_time, 0.03).
Histograms can also be tagged, just like Counters and Gauges.
imetrics_hist_openmetrics:new(http_response_time, #{code => "200"}, [0, 0.01, 0.05, 0.1, 0.2, 1, 10]),
imetrics_hist_openmetrics:new(http_response_time, #{code => "408"}, [0, 0.1, 1, 10, 100]).
Stats is a map tracking stats about a collection of values without storing the actual values themselves. It tracks number of values, min value, max value, sum, and squared sum. Designed more for a single bulk update instead of rapid updates in succession.
% retrieve/create named stats
Stats = imetrics:stats(ievent_expiration_jobs),
% for each expiration job, update stats in memory
Stats2 = imetrics_stats:update(ExecutionTime, Stats),
...
% store the result in ets
imetrics:set_stats(ievent_expiration_jobs, Stats2)
Exemplars are data points that can be added on top of a chart to provide information on a
specific example of an event to augment the higher-level information offered by metrics more broadly.
Exemplars are associated to a specific counter or histogram bucket (both name and tag, where applicable), and contain, at minimum, a value.
In addition, exemplars can attach labels and timestamps. Labels are user-defined fields of information, provided as quick reference to be displayed.
A trace id is a common label, as it allows linking to detailed information about that trace.
Labels are entered as a map, with label names as keys and label values as the values.
The combined length of label names and values should not exceed 128 UTF-8 characters if intended for use with systems using the OpenMetrics specifications.
Finally, exemplars can attach a timestamp. If left undefined, the timestamp will default to the current Erlang system time when imetrics:set_exemplar
is called.
Timestamps should be in seconds from Unix Epoch, and do not need to be integers.
If exemplars are not updated between queries, additional copies of the exemplar will not be generated,
so creating exemplars on infrequent events will not result in overpopulation of the data set.
Additionally, whenever set_exemplar
is called, the previous exemplar associated with the given
name and tags will be overwritten, meaning only the most recently passed exemplar will be stored at any given time.
imetrics:set_exemplar
, when called without a type identifier, or when called with a type of counter
, does not check to see if the added exemplar is associated with any actual values, and will always return true. imetrics_hist_openmetrics:set_exemplar
, or imetrics:set_exemplar
with type histogram
, works slightly differently, in that the value provided for the exemplar determines which bucket it is associated with in the histogram, and if the function is called for a histogram/tag combination that does not exist, this function will return {error, {badarg, check_ets}}
.
%Minimal set_exemplar
imetrics:set_exemplar(http_responses, 1),
%set_exemplar with all details included
imetrics:set_exemplar(http_responses, #{ code => "404" }, 1, #{traceid => "oHg5SJYRHA0"}, 1684267027.342, counter),
%Setting an exemplar on a histogram:
imetrics_hist_openmetrics:set_exemplar(http_response_times, #{code => "408"}, 15.3),
imetrics:set_exemplar(http_response_times, #{code => "400"}, 23, histogram).
The OpenMetrics standard specifies the Info type as a way to expose textual information that is expected to remain constant while a program is running, for example, a version number, a revision control commit, or a compiler version. imetrics supports info metrics with the set_info
function, which takes two arguments: a Name, and a map of tags to represent the data to be exposed. Info metrics are unique based on Name, so any time set_info
is called for a given name, the previous entry bearing that name will be overwritten with the new map.
imetrics:set_info(server, #{version => "1.7.10", otp => "25.2"}).
imetrics will normalize any Name and Tag Value inputs to the add and set_gauge
functions to a binary string. The Name and any Tag Values need to match one of these
guards (See imetrics:bin/1
):
is_atom(V)
is_list(V)
is_binary(V)
is_integer(V)
is_tuple(V) andalso tuple_size(V) =< 8
Otherwise, the add and set_gauge functions will return
{error, {function_clause, check_inputs}}
.
imetrics also includes adapters for common Erlang modules to enable collection of certain metrics automatically. Further documentation for these modules is available below:
- imetrics_cowboy - An imetrics adapter that collects information about Cowboy response handlers and can also annotate metrics with custom user data (docs)
- imetrics_cowboy_stream_h - An older adapter for Cowboy that doesn't collect any custom user data (docs TBD)
- imetrics_lager_backend - A Lager backend that counts log events segmented by log level (docs TBD)
The function imetrics:get_with_types/0
will return a proplist containing all
the metrics stored by imetrics with their tag dictionaries. Each metric will have
a map of tag permutations, with the value corresponding to each tag permutation.
> imetrics:get_with_types().
[{<<"client_connections">>,{counter,[{#{},1}]}},
{<<"http_responses">>,
{counter,[{#{code => <<"404">>},1},
{#{code => <<"200">>},1}]}},
{<<"cpu_load_avg">>,
{gauge,[{#{granularity => <<"1min">>},0.1},
{#{granularity => <<"5min">>},3.4}]}},
{<<"velocity">>,{gauge,[{#{},50.5}]}}]
imetrics starts an HTTP server that returns a plaintext representation of all the
metrics compatible with the OpenMetrics standard.
The URI for accessing this data is /metrics
.
$ curl localhost:8085/metrics
# TYPE client_connections counter
client_connections_total{} 1
# TYPE http_responses counter
http_responses_total{code="404"} 1
http_responses_total{code="200"} 1
# TYPE cpu_load_avg gauge
cpu_load_avg{granularity="1min"} 0.1
cpu_load_avg{granularity="5min"} 3.4
# TYPE velocity gauge
velocity{} 50.5
# EOF
For backwards compatibility with previous versions of imetrics, two older interfaces
still work, but they don't support all features. Namely, tagged metrics are unsupported.
(Only the legacy "mapped" metrics, which can store one tag at a time, are supported.
The APIs to create these "mapped" metrics are: imetrics:add_m/3
, imetrics:set_gauge_m/3
,
imetrics:update_gauge_m/3
, and imetrics:update_counter_dimension/2
.) Use of these
interfaces is strongly discouraged in favor of the more flexible tagging interface.
The function imetrics:get/0
will return a proplist containing all the metrics
stored by imetrics. Un-mapped metrics have the number value in pos 2 of each
entry. Mapped metrics have a nested proplist containing the metrics for each
associated key.
> imetrics:get().
[{<<"client_connections">>,1},
{<<"velocity">>,50.5},
{<<"http_responses">>,[{<<"200">>,1},{<<"404">>,1}]},
{<<"cpu_load_avg">>,[{<<"1min">>,0.1},{<<"5min">>,3.4}]}]
All the metric names and mapping keys are normalized as binaries.
imetrics starts a very simple HTTP server that returns an easily parseable
plaintext representation of all the metrics. The URI for accessing this data
is /imetrics/varz:get
. The somewhat strange URI format is necessary to make
proper use of the built-in HTTP server in Erlang. (See (Though
mod_esi
)mod_esi
is no longer used, this URI format is now maintained for backwards
compatibility.)
$ curl localhost:8085/imetrics/varz:get
client_connections 1
velocity 50.5
http_responses 200:1 404:1
cpu_load_avg 1min:0.1 5min:3.4
env var | default | desc |
---|---|---|
http_server_port |
8085 |
Listening port |
separator |
<<"_">> |
binary string used to separate tuple elements for Name, Key |
strict_openmetrics_compat |
false |
If set to true , metrics will only display on the old HTTP endpoint if they aren't compatible with the OpenMetrics endpoint. Metrics will only display on one endpoint or the other, never both. |
openmetrics_exemplar_compat |
false |
If set to true , counters will display with _total appended to their end, and exemplars will be displayed for counters. When false , counters will not get the appended suffix, and exemplars will not be displayed on counters. |
hist_max_buckets |
64 |
Upper limit on the number of buckets for histograms. Calls to create a histogram with more buckets than this value will fail. |
imetrics is currently undergoing an effort to update the format of metrics it
serves over HTTP. The old format is still served at /imetrics/varz:get
, and it
is a simple plaintext representation of the metrics. Work is ongoing to convert this
representation to the OpenMetrics standard.
This is currently served at /metrics
, and work continues on the feature/openmetrics
branch.