Instrumenting

Client libraries

Before you can monitor your services, you need to add instrumentation to their code via one of the Prometheus client libraries. These implement the Prometheus metric types.

Choose a Prometheus client library that matches the language in which your application is written. This lets you define and expose internal metrics via an HTTP endpoint on your application’s instance:

Unofficial third-party client libraries:

When Prometheus scrapes your instance's HTTP endpoint, the client library sends the current state of all tracked metrics to the server.

If no client library is available for your language, or you want to avoid dependencies, you may also implement one of the supported exposition formats yourself to expose metrics.

When implementing a new Prometheus client library, please follow the guidelines on writing client libraries. Note that this document is still a work in progress. Please also consider consulting the development mailing list. We are happy to give advice on how to make your library as useful and consistent as possible.

Scrape protocol content negotiation

This document specifies the protocol negotiation mechanism used by Prometheus when scraping metrics from targets. It defines the Accept header format, supported Content Types, and the negotiation process for determining the best available format for metric exposition.

Introduction

Prometheus supports multiple formats for scraping metrics, including both text-based and binary protobuf formats. Based on the value of the Accept header, the target will pick the best available Content Type for its reply.

Protocol Types

Supported Protocols

The following protocols are supported by Prometheus:

  1. PrometheusProto - Binary protobuf format
  2. PrometheusText0.0.4 - Prometheus text format version 0.0.4
  3. PrometheusText1.0.0 - Prometheus text format version 1.0.0
  4. OpenMetricsText0.0.1 - OpenMetrics text format version 0.0.1
  5. OpenMetricsText1.0.0 - OpenMetrics text format version 1.0.0

Protocol Headers

Each protocol MUST be associated with a specific MIME type and version:

Protocol MIME Type Parameters
PrometheusProto application/vnd.google.protobuf proto=io.prometheus.client.MetricFamily;encoding=delimited
PrometheusText0.0.4 text/plain version=0.0.4
PrometheusText1.0.0 text/plain version=1.0.0
OpenMetricsText0.0.1 application/openmetrics-text version=0.0.1
OpenMetricsText1.0.0 application/openmetrics-text version=1.0.0

Accept Header Construction

The Accept header is constructed by Prometheus to indicate what formats it supports.

Basic Format

The Accept header MUST be constructed as follows:

  1. For each protocol supported by the target:
    • The protocol's MIME type and parameters MUST be specified.
    • For protobuf protocols, an encoding of "delimited" MUST be specified.
    • For PrometheusText1.0.0 and OpenMetricsText1.0.0, the escaping scheme parameter SHOULD be appended.
    • A quality value (q) parameter SHOULD be appended.
  2. A catch-all */* with the lowest quality value SHOULD be appended.

Quality Values

Quality values SHOULD be assigned in descending order based on the protocol's position in the Accept header:

Escaping Scheme

For PrometheusText1.0.0 and OpenMetricsText1.0.0 protocols, the Accept header SHOULD include an escaping scheme parameter: escaping=<scheme>

Where <scheme> MUST be one of:

See Escaping Schemes spec for details on how the escaping schemes function.

Compression

The Accept-Encoding header SHOULD be set to:

Selection of Format

The scrape target SHOULD use the following process to select an appropriate Content-Type based on the list of protocols in the Accept header generated by Prometheus:

  1. It MUST use the protocol in the Accept header with the highest weighting that is supported by Prometheus.
  2. If no protocols are supported, the target MAY use a user-configured fallback scrape protocol.
  3. If no fallback is specified, the target MUST use PrometheusText0.0.4 as a last resort.

Content-Type Response

Targets SHOULD respond with a Content-Type header that matches one of the accepted formats. The Content-Type header MUST include:

  1. The appropriate MIME type.
  2. The version parameter.
  3. For text formats version 1.0.0 and above, the escaping scheme parameter.

Security Considerations

  1. Targets MUST validate the Accept header to prevent potential injection attacks
  2. The escaping scheme parameter MUST be validated to prevent protocol confusion
  3. Content-Type headers MUST be properly sanitized to prevent MIME type confusion

Examples

Default Accept Header

Accept: application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=1.0.0;escaping=allow-utf-8;q=0.3,text/plain;version=0.0.4;q=0.2,/;q=0.1

Protobuf-First Accept Header

Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.5,application/
openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.4,application/openmetrics-text;version=0.0.1;q=0.3,text/plain;version=1.0.0;escaping=allow-utf-8;q=0.2,text/plain;version=0.0.4;q=0.1,/;q=0.0

UTF-8 metric and label name escaping schemes

This document specifies the different escaping schemes used by Prometheus during generation of text exposition for metric and label names that contain characters outside the legacy character set. These schemes are negotiated during scraping via the escaping parameter in the Accept and Content-Type headers.

Introduction

Prometheus supports multiple escaping schemes to handle metric and label names in text exposition that contain characters outside the legacy character set (a-zA-Z0-9_:). The escaping scheme is negotiated during scraping and affects how metric producers should format their metric names.

Escaping Schemes

No Escaping (allow-utf-8)

Header Value: escaping=allow-utf-8

Behavior:

Underscore Escaping (underscores)

Header Value: escaping=underscores

Behavior:

Dots Escaping (dots)

Header Value: escaping=dots

Behavior:

Value Encoding Escaping (values)

Header Value: escaping=values

Behavior:

Default Behavior

If no escaping scheme is specified in the Accept header, underscores escaping SHOULD be used.

Security Considerations

  1. Targets MUST validate input names before applying escaping.
  2. The escaping scheme MUST be validated to prevent injection attacks.
  3. The allow-utf-8 scheme MUST only be used when both producer and consumer support UTF-8 names.

Exporters and integrations

There are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics. This is useful for cases where it is not feasible to instrument a given system with Prometheus metrics directly (for example, HAProxy or Linux system stats).

Third-party exporters

Some of these exporters are maintained as part of the official Prometheus GitHub organization, those are marked as official, others are externally contributed and maintained.

We encourage the creation of more exporters but cannot vet all of them for best practices. Commonly, those exporters are hosted outside of the Prometheus GitHub organization.

The exporter default port wiki page has become another catalog of exporters, and may include exporters not listed here due to overlapping functionality or still being in development.

The JMX exporter can export from a wide variety of JVM-based applications, for example Kafka and Cassandra.

Databases

Issue trackers and continuous integration

Messaging systems

Storage

HTTP

IoT

APIs

Logging

FinOps

Other monitoring systems

Miscellaneous

When implementing a new Prometheus exporter, please follow the guidelines on writing exporters Please also consider consulting the development mailing list. We are happy to give advice on how to make your exporter as useful and consistent as possible.

Software exposing Prometheus metrics

Some third-party software exposes metrics in the Prometheus format, so no separate exporters are needed:

The software marked direct is also directly instrumented with a Prometheus client library.

Other third-party utilities

This section lists libraries and other utilities that help you instrument code in a certain language. They are not Prometheus client libraries themselves but make use of one of the normal Prometheus client libraries under the hood. As for all independently maintained software, we cannot vet all of them for best practices.

Exposition formats

As of Prometheus version 2.0, all processes that expose metrics to Prometheus must use a text format, by default. An alternative, protobuf format may be used behind the HTTP negotiation.

There are various client libraries that implement those formats for you. If your preferred language doesn't have a client library you can create your own.

This document outlines officially supported exposition formats.

Prometheus Text Format

In this section you can find some basic information about this format as well as a more detailed breakdown of the format.

Basic info

Aspect Description
Inception April 2014
Supported in Prometheus version >=0.4.0
Transmission HTTP
Encoding UTF-8, \n line endings
HTTP Content-Type text/plain with parameters:
  • version=0.0.4 (A missing version value will lead to a fall-back to the most recent text format version.)
Optional HTTP Content-Encoding gzip
Advantages
  • Human-readable
  • Easy to assemble, especially for minimalistic cases (no nesting required)
  • Readable line by line (with the exception of metadata)
Limitations
  • Verbose
  • Types and docstrings not integral part of the syntax, meaning little-to-nonexistent metric contract validation
  • Parsing cost
Supported metric primitives
  • Counter
  • Gauge
  • Histogram
  • Summary
  • Untyped
Supported advanced features

Details

Prometheus' text-based format is line oriented. Lines are separated by a line feed character (\n). The last line must end with a line feed character. Empty lines are ignored.

Line format

Within a line, tokens can be separated by any number of blanks and/or tabs (and must be separated by at least one if they would otherwise merge with the previous token). Leading and trailing whitespace is ignored.

Comments, help text, and type information

Lines with a # as the first non-whitespace character are comments. They are ignored unless the first token after # is either HELP or TYPE. Those lines are treated as follows: If the token is HELP, at least one more token is expected, which is the metric name. All remaining tokens are considered the docstring for that metric name. HELP lines may contain any sequence of UTF-8 characters (after the metric name), but the backslash and the line feed characters have to be escaped as \\ and \n, respectively. Only one HELP line may exist for any given metric name.

If the token is TYPE, exactly two more tokens are expected. The first is the metric name, and the second is either counter, gauge, histogram, summary, or untyped, defining the type for the metric of that name. Only one TYPE line may exist for a given metric name. The TYPE line for a metric name must appear before the first sample is reported for that metric name. If there is no TYPE line for a metric name, the type is set to untyped. Metric names not corresponding to the legacy Prometheus metric name character set must be quoted and escaped.

The remaining lines describe samples (one per line) using the following syntax (EBNF):

metric_name_or_labels value [ timestamp ]

metric_name_or_labels = metric_name [ "{" labels "}" ] | "{" quoted_metric_name [ "," labels ] "}"

metric_name = identifier

quoted_metric_name = `"` escaped_string `"`

labels = [ label_pairs ]

label_pairs = label_pair { "," label_pair } [ "," ]

label_pair = label_name "=" `"` escaped_string `"`

label_name = identifier | `"` escaped_string `"`

In the sample syntax:

Grouping and sorting

All lines for a given metric must be provided as one single group, with the optional HELP and TYPE lines first (in no particular order). Beyond that, reproducible sorting in repeated expositions is preferred but not required, i.e. do not sort if the computational cost is prohibitive.

Each line must have a unique combination of a metric name and labels. Otherwise, the ingestion behavior is undefined.

Histograms and summaries

The histogram and summary types are difficult to represent in the text format. The following conventions apply:

Example

Below is an example of a full-fledged Prometheus metric exposition, including comments, HELP and TYPE expressions, a histogram, a summary, character escaping examples, and more.

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

# Escaping in label values:
msdos_file_access_time_seconds{path="C:\\DIR\\FILE.TXT",error="Cannot find file:\n\"FILE.TXT\""} 1.458255915e9

# UTF-8 metric and label names:
{"my.dotted.metric", "error.message"="Not Found"}

# Minimalistic line:
metric_without_timestamp_and_labels 12.47

# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045

# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

# Finally a summary, which has a complex representation, too:
# HELP rpc_duration_seconds A summary of the RPC duration in seconds.
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.01"} 3102
rpc_duration_seconds{quantile="0.05"} 3272
rpc_duration_seconds{quantile="0.5"} 4773
rpc_duration_seconds{quantile="0.9"} 9001
rpc_duration_seconds{quantile="0.99"} 76656
rpc_duration_seconds_sum 1.7560473e+07
rpc_duration_seconds_count 2693

OpenMetrics Text Format

OpenMetrics is an effort to standardize metric wire formatting built off of Prometheus text format. It is possible to scrape targets, and it is also available to use for federating metrics since Prometheus v2.23.0.

There are currently two versions of OpenMetrics:

Basic info

Aspect Description
Inception November 2020
Supported in Prometheus version >=2.5.0
Transmission HTTP
Encoding UTF-8, \n line endings, trailed with # EOF.
HTTP Content-Type application/openmetrics-text with parameters:
  • version=1.0.0 (a missing version value will lead to a fall-back to the most recent text OpenMetrics version.)
Optional HTTP Content-Encoding gzip
Advantages
  • Human-readable
  • Easy to assemble, especially for minimalistic cases (no nesting required)
  • Readable line by line (with the exception of metadata)
Limitations
  • Verbose
  • Types and docstrings not integral part of the syntax, meaning little-to-nonexistent metric contract validation
  • Parsing cost
Supported metric primitives
  • Counter
  • Gauge
  • Histogram
  • GaugeHistogram
  • Summary
  • Info
  • StateSet
  • Untyped
Supported advanced features
  • Unit metadata
  • Exemplars (Counters, Histogram, GaugeHistogram)
  • Created/Start Timestamps (Counter, Histogram, GaugeHistogram, Summary)
  • UTF-8 metric names and label names

Exemplars (Experimental)

Utilizing the OpenMetrics format allows for the exposition and querying of Exemplars. Exemplars provide a point in time snapshot related to a metric set for an otherwise summarized MetricFamily. Additionally, they may have a Trace ID attached to them which when used to together with a tracing system can provide more detailed information related to the specific service.

To enable this experimental feature you must have at least version v2.26.0 and add --enable-feature=exemplar-storage to your arguments.

Prometheus Protobuf Format

Prometheus officially supports protobuf exposition format in addition to the text representation.

The payload must be encoded as a set of Protobuf messages representing MetricFamily. Messages must be encoded in binary and prepended with their variadic unsigned-integer encoded size, which serves as a delimitation. The varint encoded size delimited encoding offers streaming capabilities, especially important for large scrape targets.

All string fields must be UTF-8 encoded.

Prometheus 3.0 prefers the text-based protocols by default unless:

In Prometheus 2.0, the Protobuf format was marked as deprecated, but since then this decision was reverted. From Prometheus 3.0, the Prometheus Proto is actively used and maintained, supplementing text-based formats.

Basic info

Aspect Description
Inception April 2014
Supported in Prometheus version >=0.4.0
Transmission HTTP
Encoding 32-Bit Varint-Encoded Record Length-Delimited Protocol Buffer Messages of io.prometheus.client.MetricFamily
HTTP Content-Type application/vnd.google.protobuf with parameters:
  • proto=io.prometheus.client.MetricFamily
  • encoding=delimited
Optional HTTP Content-Encoding gzip
Advantages
  • Cross-Platform
  • Size
  • Forward/Backward Compatibility
  • Strict Schema
  • Supports Concatenation and Streaming
  • Composite Values
Limitations
  • Not human readable
Supported metric primitives
  • Counter
  • Gauge
  • Histogram
  • GaugeHistogram
  • Summary
  • Untyped
Supported advanced features
  • Unit metadata
  • Exemplars (Counters, Histogram, GaugeHistogram)
  • Created/Start Timestamps (Counter, Histogram, GaugeHistogram, Summary)
  • Native (sparse and exponential) Histogram, GaugeHistogram
  • UTF-8 metric names and label names

When to use Proto over Text?

The text format is human-readable, compresses well and is efficient enough for programming use, but Prometheus community also maintains the protobuf formats, because:

You can learn more in the PromCon 2025 talk.

Versioning

At the moment Prometheus protobuf is stable, but explicitly unversioned. Instead, it follows Prometheus versioning as a reference.

Schema

Protobuf schema, identified as io.prometheus.client is maintained in Prometheus repository here. Schema is also available in buf registry.

HTTP Content-Type requirements

Starting with Prometheus 3.0, scrape targets must return a valid Content-Type header for the metrics endpoint. If the Content-Type is missing, unparsable, or not a supported media type, the scrape will fail. See changes in scrape protocols in the migration guide for details.

See each of the exposition format sections for the accurate HTTP content types.

ScrapeProtocols vs Content-Type

Prometheus scrape config offers scrape protocol negotiation based on the content-type using the scrape_protocols config. For the Prometheus user convenience the scrape protocols are referenced by a unique name that maps to the concrete content-type. See Protocol Headers for details.

However, the targets should expose metrics in the exposition format with the absolute, response content-type (e.g. application/openmetrics-text;version=1.0.0) and only one.

Historical versions

For details on historical format versions, see the legacy Client Data Exposition Format document.

The current version of the original Protobuf format (with the recent extensions for native histograms) is maintained in the prometheus/client_model repository.

Pushing metrics

Occasionally you will need to monitor components which cannot be scraped. The Prometheus Pushgateway allows you to push time series from short-lived service-level batch jobs to an intermediary job which Prometheus can scrape. Combined with Prometheus's simple text-based exposition format, this makes it easy to instrument even shell scripts without a client library.

Writing client libraries

This document covers what functionality and API Prometheus client libraries should offer, with the aim of consistency across libraries, making the easy use cases easy and avoiding offering functionality that may lead users down the wrong path.

There are 10 languages already supported at the time of writing, so we’ve gotten a good sense by now of how to write a client. These guidelines aim to help authors of new client libraries produce good libraries.

Conventions

MUST/MUST NOT/SHOULD/SHOULD NOT/MAY have the meanings given in https://www.ietf.org/rfc/rfc2119.txt

In addition ENCOURAGED means that a feature is desirable for a library to have, but it’s okay if it’s not present. In other words, a nice to have.

Things to keep in mind:

The common use cases are (in order):

Overall structure

Clients MUST be written to be callback based internally. Clients SHOULD generally follow the structure described here.

The key class is the Collector. This has a method (typically called ‘collect’) that returns zero or more metrics and their samples. Collectors get registered with a CollectorRegistry. Data is exposed by passing a CollectorRegistry to a class/method/function "bridge", which returns the metrics in a format Prometheus supports. Every time the CollectorRegistry is scraped it must callback to each of the Collectors’ collect method.

The interface most users interact with are the Counter, Gauge, Summary, and Histogram Collectors. These represent a single metric, and should cover the vast majority of use cases where a user is instrumenting their own code.

More advanced uses cases (such as proxying from another monitoring/instrumentation system) require writing a custom Collector. Someone may also want to write a "bridge" that takes a CollectorRegistry and produces data in a format a different monitoring/instrumentation system understands, allowing users to only have to think about one instrumentation system.

CollectorRegistry SHOULD offer register()/unregister() functions, and a Collector SHOULD be allowed to be registered to multiple CollectorRegistrys.

Client libraries MUST be thread safe.

For non-OO languages such as C, client libraries should follow the spirit of this structure as much as is practical.

Naming

Client libraries SHOULD follow function/method/class names mentioned in this document, keeping in mind the naming conventions of the language they’re working in. For example, set_to_current_time() is good for a method name in Python, but SetToCurrentTime() is better in Go and setToCurrentTime() is the convention in Java. Where names differ for technical reasons (e.g. not allowing function overloading), documentation/help strings SHOULD point users towards the other names.

Libraries MUST NOT offer functions/methods/classes with the same or similar names to ones given here, but with different semantics.

Metrics

The Counter, Gauge, Summary and Histogram metric types are the primary interface by users.

Counter and Gauge MUST be part of the client library. At least one of Summary and Histogram MUST be offered.

These should be primarily used as file-static variables, that is, global variables defined in the same file as the code they’re instrumenting. The client library SHOULD enable this. The common use case is instrumenting a piece of code overall, not a piece of code in the context of one instance of an object. Users shouldn’t have to worry about plumbing their metrics throughout their code, the client library should do that for them (and if it doesn’t, users will write a wrapper around the library to make it "easier" - which rarely tends to go well).

There MUST be a default CollectorRegistry, the standard metrics MUST by default implicitly register into it with no special work required by the user. There MUST be a way to have metrics not register to the default CollectorRegistry, for use in batch jobs and unittests. Custom collectors SHOULD also follow this.

Exactly how the metrics should be created varies by language. For some (Java, Go) a builder approach is best, whereas for others (Python) function arguments are rich enough to do it in one call.

For example in the Java Simpleclient we have:

class YourClass {
  static final Counter requests = Counter.build()
      .name("requests_total")
      .help("Requests.").register();
}

This will register requests with the default CollectorRegistry. By calling build() rather than register() the metric won’t be registered (handy for unittests), you can also pass in a CollectorRegistry to register() (handy for batch jobs).

Counter

Counter is a monotonically increasing counter. It MUST NOT allow the value to decrease, however it MAY be reset to 0 (such as by server restart).

A counter MUST have the following methods:

A counter is ENCOURAGED to have:

A way to count exceptions throw/raised in a given piece of code, and optionally only certain types of exceptions. This is count_exceptions in Python.

Counters MUST start at 0.

Gauge

Gauge represents a value that can go up and down.

A gauge MUST have the following methods:

Gauges MUST start at 0, you MAY offer a way for a given gauge to start at a different number.

A gauge SHOULD have the following methods:

A gauge is ENCOURAGED to have:

A way to track in-progress requests in some piece of code/function. This is track_inprogress in Python.

A way to time a piece of code and set the gauge to its duration in seconds. This is useful for batch jobs. This is startTimer/setDuration in Java and the time() decorator/context manager in Python. This SHOULD match the pattern in Summary/Histogram (though set() rather than observe()).

Summary

A summary samples observations (usually things like request durations) over sliding windows of time and provides instantaneous insight into their distributions, frequencies, and sums.

A summary MUST NOT allow the user to set "quantile" as a label name, as this is used internally to designate summary quantiles. A summary is ENCOURAGED to offer quantiles as exports, though these can’t be aggregated and tend to be slow. A summary MUST allow not having quantiles, as just _count/_sum is quite useful and this MUST be the default.

A summary MUST have the following methods:

A summary SHOULD have the following methods:

Some way to time code for users in seconds. In Python this is the time() decorator/context manager. In Java this is startTimer/observeDuration. Units other than seconds MUST NOT be offered (if a user wants something else, they can do it by hand). This should follow the same pattern as Gauge/Histogram.

Summary _count/_sum MUST start at 0.

Histogram

Histograms allow aggregatable distributions of events, such as request latencies. This is at its core a counter per bucket.

A histogram MUST NOT allow le as a user-set label, as le is used internally to designate buckets.

A histogram MUST offer a way to manually choose the buckets. Ways to set buckets in a linear(start, width, count) and exponential(start, factor, count) fashion SHOULD be offered. Count MUST include the +Inf bucket.

A histogram SHOULD have the same default buckets as other client libraries. Buckets MUST NOT be changeable once the metric is created.

A histogram MUST have the following methods:

A histogram SHOULD have the following methods:

Some way to time code for users in seconds. In Python this is the time() decorator/context manager. In Java this is startTimer/observeDuration. Units other than seconds MUST NOT be offered (if a user wants something else, they can do it by hand). This should follow the same pattern as Gauge/Summary.

Histogram _count/_sum and the buckets MUST start at 0.

Further metrics considerations

Providing additional functionality in metrics beyond what’s documented above as makes sense for a given language is ENCOURAGED.

If there’s a common use case you can make simpler then go for it, as long as it won’t encourage undesirable behaviours (such as suboptimal metric/label layouts, or doing computation in the client).

Labels

Labels are one of the most powerful aspects of Prometheus, but easily abused. Accordingly client libraries must be very careful in how labels are offered to users.

Client libraries SHOULD NOT allow users to have different label names for the same metric for Gauge/Counter/Summary/Histogram or any other Collector offered by the library.

Metrics from custom collectors should almost always have consistent label names. As there are still rare but valid use cases where this is not the case, client libraries should not verify this.

While labels are powerful, the majority of metrics will not have labels. Accordingly the API should allow for labels but not dominate it.

A client library MUST allow for optionally specifying a list of label names at Gauge/Counter/Summary/Histogram creation time. A client library SHOULD support any number of label names. A client library MUST validate that label names meet the documented requirements.

The general way to provide access to labeled dimension of a metric is via a labels() method that takes either a list of the label values or a map from label name to label value and returns a "Child". The usual .inc()/.dec()/.observe() etc. methods can then be called on the Child.

The Child returned by labels() SHOULD be cacheable by the user, to avoid having to look it up again - this matters in latency-critical code.

Metrics with labels SHOULD support a remove() method with the same signature as labels() that will remove a Child from the metric no longer exporting it, and a clear() method that removes all Children from the metric. These invalidate caching of Children.

There SHOULD be a way to initialize a given Child with the default value, usually just calling labels(). Metrics without labels MUST always be initialized to avoid problems with missing metrics.

Metric names

Metric names must follow the specification. As with label names, this MUST be met for uses of Gauge/Counter/Summary/Histogram and in any other Collector offered with the library.

Many client libraries offer setting the name in three parts: namespace_subsystem_name of which only the name is mandatory.

Dynamic/generated metric names or subparts of metric names MUST be discouraged, except when a custom Collector is proxying from other instrumentation/monitoring systems. Generated/dynamic metric names are a sign that you should be using labels instead.

Metric description and help

Gauge/Counter/Summary/Histogram MUST require metric descriptions/help to be provided.

Any custom Collectors provided with the client libraries MUST have descriptions/help on their metrics.

It is suggested to make it a mandatory argument, but not to check that it’s of a certain length as if someone really doesn’t want to write docs we’re not going to convince them otherwise. Collectors offered with the library (and indeed everywhere we can within the ecosystem) SHOULD have good metric descriptions, to lead by example.

Exposition

Clients MUST implement the text-based exposition format outlined in the exposition formats documentation.

Reproducible order of the exposed metrics is ENCOURAGED (especially for human readable formats) if it can be implemented without a significant resource cost.

Standard and runtime collectors

Client libraries SHOULD offer what they can of the Standard exports, documented below.

These SHOULD be implemented as custom Collectors, and registered by default on the default CollectorRegistry. There SHOULD be a way to disable these, as there are some very niche use cases where they get in the way.

Process metrics

These metrics have the prefix process_. If obtaining a necessary value is problematic or even impossible with the used language or runtime, client libraries SHOULD prefer leaving out the corresponding metric over exporting bogus, inaccurate, or special values (like NaN). All memory values in bytes, all times in unixtime/seconds.

Metric name Help string Unit
process_cpu_seconds_total Total user and system CPU time spent in seconds. seconds
process_open_fds Number of open file descriptors. file descriptors
process_max_fds Maximum number of open file descriptors. file descriptors
process_virtual_memory_bytes Virtual memory size in bytes. bytes
process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. bytes
process_resident_memory_bytes Resident memory size in bytes. bytes
process_heap_bytes Process heap size in bytes. bytes
process_start_time_seconds Start time of the process since unix epoch in seconds. seconds
process_threads Number of OS threads in the process. threads

Runtime metrics

In addition, client libraries are ENCOURAGED to also offer whatever makes sense in terms of metrics for their language’s runtime (e.g. garbage collection stats), with an appropriate prefix such as go_, hotspot_ etc.

Unit tests

Client libraries SHOULD have unit tests covering the core instrumentation library and exposition.

Client libraries are ENCOURAGED to offer ways that make it easy for users to unit-test their use of the instrumentation code. For example, the CollectorRegistry.get_sample_value in Python.

Packaging and dependencies

Ideally, a client library can be included in any application to add some instrumentation without breaking the application.

Accordingly, caution is advised when adding dependencies to the client library. For example, if you add a library that uses a Prometheus client that requires version x.y of a library but the application uses x.z elsewhere, will that have an adverse impact on the application?

It is suggested that where this may arise, that the core instrumentation is separated from the bridges/exposition of metrics in a given format. For example, the Java simpleclient simpleclient module has no dependencies, and the simpleclient_servlet has the HTTP bits.

Performance considerations

As client libraries must be thread-safe, some form of concurrency control is required and consideration must be given to performance on multi-core machines and applications.

In our experience the least performant is mutexes.

Processor atomic instructions tend to be in the middle, and generally acceptable.

Approaches that avoid different CPUs mutating the same bit of RAM work best, such as the DoubleAdder in Java’s simpleclient. There is a memory cost though.

As noted above, the result of labels() should be cacheable. The concurrent maps that tend to back metric with labels tend to be relatively slow. Special-casing metrics without labels to avoid labels()-like lookups can help a lot.

Metrics SHOULD avoid blocking when they are being incremented/decremented/set etc. as it’s undesirable for the whole application to be held up while a scrape is ongoing.

Having benchmarks of the main instrumentation operations, including labels, is ENCOURAGED.

Resource consumption, particularly RAM, should be kept in mind when performing exposition. Consider reducing the memory footprint by streaming results, and potentially having a limit on the number of concurrent scrapes.

Writing exporters

If you are instrumenting your own code, the general rules of how to instrument code with a Prometheus client library should be followed. When taking metrics from another monitoring or instrumentation system, things tend not to be so black and white.

This document contains things you should consider when writing an exporter or custom collector. The theory covered will also be of interest to those doing direct instrumentation.

If you are writing an exporter and are unclear on anything here, please contact us on IRC (#prometheus on libera) or the mailing list.

Maintainability and purity

The main decision you need to make when writing an exporter is how much work you’re willing to put in to get perfect metrics out of it.

If the system in question has only a handful of metrics that rarely change, then getting everything perfect is an easy choice, a good example of this is the HAProxy exporter.

On the other hand, if you try to get things perfect when the system has hundreds of metrics that change frequently with new versions, then you’ve signed yourself up for a lot of ongoing work. The MySQL exporter is on this end of the spectrum.

The node exporter is a mix of these, with complexity varying by module. For example, the mdadm collector hand-parses a file and exposes metrics created specifically for that collector, so we may as well get the metrics right. For the meminfo collector the results vary across kernel versions so we end up doing just enough of a transform to create valid metrics.

Configuration

When working with applications, you should aim for an exporter that requires no custom configuration by the user beyond telling it where the application is. You may also need to offer the ability to filter out certain metrics if they may be too granular and expensive on large setups, for example the HAProxy exporter allows filtering of per-server stats. Similarly, there may be expensive metrics that are disabled by default.

When working with other monitoring systems, frameworks and protocols you will often need to provide additional configuration or customization to generate metrics suitable for Prometheus. In the best case scenario, a monitoring system has a similar enough data model to Prometheus that you can automatically determine how to transform metrics. This is the case for Cloudwatch, SNMP and collectd. At most, we need the ability to let the user select which metrics they want to pull out.

In other cases, metrics from the system are completely non-standard, depending on the usage of the system and the underlying application. In that case the user has to tell us how to transform the metrics. The JMX exporter is the worst offender here, with the Graphite and StatsD exporters also requiring configuration to extract labels.

Ensuring the exporter works out of the box without configuration, and providing a selection of example configurations for transformation if required, is advised.

YAML is the standard Prometheus configuration format, all configuration should use YAML by default.

Metrics

Naming

Follow the best practices on metric naming.

Generally metric names should allow someone who is familiar with Prometheus but not a particular system to make a good guess as to what a metric means. A metric named http_requests_total is not extremely useful - are these being measured as they come in, in some filter or when they get to the user’s code? And requests_total is even worse, what type of requests?

With direct instrumentation, a given metric should exist within exactly one file. Accordingly, within exporters and collectors, a metric should apply to exactly one subsystem and be named accordingly.

Metric names should never be procedurally generated, except when writing a custom collector or exporter.

Metric names for applications should generally be prefixed by the exporter name, e.g. haproxy_up.

Metrics must use base units (e.g. seconds, bytes) and leave converting them to something more readable to graphing tools. No matter what units you end up using, the units in the metric name must match the units in use. Similarly, expose ratios, not percentages. Even better, specify a counter for each of the two components of the ratio.

Metric names should not include the labels that they’re exported with, e.g. by_type, as that won’t make sense if the label is aggregated away.

The one exception is when you’re exporting the same data with different labels via multiple metrics, in which case that’s usually the sanest way to distinguish them. For direct instrumentation, this should only come up when exporting a single metric with all the labels would have too high a cardinality.

Prometheus metrics and label names are written in snake_case. Converting camelCase to snake_case is desirable, though doing so automatically doesn’t always produce nice results for things like myTCPExample or isNaN so sometimes it’s best to leave them as-is.

Exposed metrics should not contain colons, these are reserved for user defined recording rules to use when aggregating.

Only [a-zA-Z0-9:_] are valid in metric names.

The _sum, _count, _bucket and _total suffixes are used by Summaries, Histograms and Counters. Unless you’re producing one of those, avoid these suffixes.

_total is a convention for counters, you should use it if you’re using the COUNTER type.

The process_ and scrape_ prefixes are reserved. It’s okay to add your own prefix on to these if they follow matching semantics. For example, Prometheus has scrape_duration_seconds for how long a scrape took, it's good practice to also have an exporter-centric metric, e.g. jmx_scrape_duration_seconds, saying how long the specific exporter took to do its thing. For process stats where you have access to the PID, both Go and Python offer collectors that’ll handle this for you. A good example of this is the HAProxy exporter.

When you have a successful request count and a failed request count, the best way to expose this is as one metric for total requests and another metric for failed requests. This makes it easy to calculate the failure ratio. Do not use one metric with a failed or success label. Similarly, with hit or miss for caches, it’s better to have one metric for total and another for hits.

Consider the likelihood that someone using monitoring will do a code or web search for the metric name. If the names are very well-established and unlikely to be used outside of the realm of people used to those names, for example SNMP and network engineers, then leaving them as-is may be a good idea. This logic doesn’t apply for all exporters, for example the MySQL exporter metrics may be used by a variety of people, not just DBAs. A HELP string with the original name can provide most of the same benefits as using the original names.

Labels

Read the general advice on labels.

Avoid type as a label name, it’s too generic and often meaningless. You should also try where possible to avoid names that are likely to clash with target labels, such as region, zone, cluster, availability_zone, az, datacenter, dc, owner, customer, stage, service, environment and env. If, however, that’s what the application calls some resource, it’s best not to cause confusion by renaming it.

Avoid the temptation to put things into one metric just because they share a prefix. Unless you’re sure something makes sense as one metric, multiple metrics is safer.

The label le has special meaning for Histograms, and quantile for Summaries. Avoid these labels generally.

Read/write and send/receive are best as separate metrics, rather than as a label. This is usually because you care about only one of them at a time, and it is easier to use them that way.

The rule of thumb is that one metric should make sense when summed or averaged. There is one other case that comes up with exporters, and that’s where the data is fundamentally tabular and doing otherwise would require users to do regexes on metric names to be usable. Consider the voltage sensors on your motherboard, while doing math across them is meaningless, it makes sense to have them in one metric rather than having one metric per sensor. All values within a metric should (almost) always have the same unit, for example consider if fan speeds were mixed in with the voltages, and you had no way to automatically separate them.

Don’t do this:

my_metric{label="a"} 1
my_metric{label="b"} 6
my_metric{label="total"} 7

or this:

my_metric{label="a"} 1
my_metric{label="b"} 6
my_metric{} 7

The former breaks for people who do a sum() over your metric, and the latter breaks sum and is quite difficult to work with. Some client libraries, for example Go, will actively try to stop you doing the latter in a custom collector, and all client libraries should stop you from doing the latter with direct instrumentation. Never do either of these, rely on Prometheus aggregation instead.

If your monitoring exposes a total like this, drop the total. If you have to keep it around for some reason, for example the total includes things not counted individually, use different metric names.

Instrumentation labels should be minimal, every extra label is one more that users need to consider when writing their PromQL. Accordingly, avoid having instrumentation labels which could be removed without affecting the uniqueness of the time series. Additional information around a metric can be added via an info metric, for an example see below how to handle version numbers.

However, there are cases where it is expected that virtually all users of a metric will want the additional information. If so, adding a non-unique label, rather than an info metric, is the right solution. For example the mysqld_exporter's mysqld_perf_schema_events_statements_total's digest label is a hash of the full query pattern and is sufficient for uniqueness. However, it is of little use without the human readable digest_text label, which for long queries will contain only the start of the query pattern and is thus not unique. Thus we end up with both the digest_text label for humans and the digest label for uniqueness.

Target labels, not static scraped labels

If you ever find yourself wanting to apply the same label to all of your metrics, stop.

There’s generally two cases where this comes up.

The first is for some label it would be useful to have on the metrics such as the version number of the software. Instead, use the approach described at https://www.robustperception.io/how-to-have-labels-for-machine-roles/.

The second case is when a label is really a target label. These are things like region, cluster names, and so on, that come from your infrastructure setup rather than the application itself. It’s not for an application to say where it fits in your label taxonomy, that’s for the person running the Prometheus server to configure and different people monitoring the same application may give it different names.

Accordingly, these labels belong up in the scrape configs of Prometheus via whatever service discovery you’re using. It’s okay to apply the concept of machine roles here as well, as it’s likely useful information for at least some people scraping it.

Types

You should try to match up the types of your metrics to Prometheus types. This usually means counters and gauges. The _count and _sum of summaries are also relatively common, and on occasion you’ll see quantiles. Histograms are rare, if you come across one remember that the exposition format exposes cumulative values.

Often it won’t be obvious what the type of metric is, especially if you’re automatically processing a set of metrics. In general UNTYPED is a safe default.

Counters can’t go down, so if you have a counter type coming from another instrumentation system that can be decremented, for example Dropwizard metrics then it's not a counter, it's a gauge. UNTYPED is probably the best type to use there, as GAUGE would be misleading if it were being used as a counter.

Help strings

When you’re transforming metrics it’s useful for users to be able to track back to what the original was, and what rules were in play that caused that transformation. Putting in the name of the collector or exporter, the ID of any rule that was applied and the name and details of the original metric into the help string will greatly aid users.

Prometheus doesn’t like one metric having different help strings. If you’re making one metric from many others, choose one of them to put in the help string.

For examples of this, the SNMP exporter uses the OID and the JMX exporter puts in a sample mBean name. The HAProxy exporter has hand-written strings. The node exporter also has a wide variety of examples.

Drop less useful statistics

Some instrumentation systems expose 1m, 5m, 15m rates, average rates since application start (these are called mean in Dropwizard metrics for example) in addition to minimums, maximums and standard deviations.

These should all be dropped, as they’re not very useful and add clutter. Prometheus can calculate rates itself, and usually more accurately as the averages exposed are usually exponentially decaying. You don’t know what time the min or max were calculated over, and the standard deviation is statistically useless and you can always expose sum of squares, _sum and _count if you ever need to calculate it.

Quantiles have related issues, you may choose to drop them or put them in a Summary.

Dotted strings

Many monitoring systems don’t have labels, instead doing things like my.class.path.mymetric.labelvalue1.labelvalue2.labelvalue3.

The Graphite and StatsD exporters share a way of transforming these with a small configuration language. Other exporters should implement the same. The transformation is currently implemented only in Go, and would benefit from being factored out into a separate library.

Collectors

When implementing the collector for your exporter, you should never use the usual direct instrumentation approach and then update the metrics on each scrape.

Rather create new metrics each time. In Go this is done with MustNewConstMetric in your Collect() method. For Python see https://github.com/prometheus/client_python#custom-collectors and for Java generate a List<MetricFamilySamples> in your collect method, see StandardExports.java for an example.

The reason for this is two-fold. Firstly, two scrapes could happen at the same time, and direct instrumentation uses what are effectively file-level global variables, so you’ll get race conditions. Secondly, if a label value disappears, it’ll still be exported.

Instrumenting your exporter itself via direct instrumentation is fine, e.g. total bytes transferred or calls performed by the exporter across all scrapes. For exporters such as the blackbox exporter and SNMP exporter, which aren’t tied to a single target, these should only be exposed on a vanilla /metrics call, not on a scrape of a particular target.

Metrics about the scrape itself

Sometimes you’d like to export metrics that are about the scrape, like how long it took or how many records you processed.

These should be exposed as gauges as they’re about an event, the scrape, and the metric name prefixed by the exporter name, for example jmx_scrape_duration_seconds. Usually the _exporter is excluded and if the exporter also makes sense to use as just a collector, then definitely exclude it.

Other scrape "meta" metrics should be avoided. For example, a counter for the number of scrapes, or a histogram of the scrape duration. Having the exporter track these metrics duplicate the automatically generated metrics of Prometheus itself. This adds to the storage cost of every exporter instance.

Machine and process metrics

Many systems, for example Elasticsearch, expose machine metrics such as CPU, memory and filesystem information. As the node exporter provides these in the Prometheus ecosystem, such metrics should be dropped.

In the Java world, many instrumentation frameworks expose process-level and JVM-level stats such as CPU and GC. The Java client and JMX exporter already include these in the preferred form via DefaultExports.java, so these should also be dropped.

Similarly with other languages and frameworks.

Deployment

Each exporter should monitor exactly one instance application, preferably sitting right beside it on the same machine. That means for every HAProxy you run, you run a haproxy_exporter process. For every machine with a Mesos worker, you run the Mesos exporter on it, and another one for the master, if a machine has both.

The theory behind this is that for direct instrumentation this is what you’d be doing, and we’re trying to get as close to that as we can in other layouts. This means that all service discovery is done in Prometheus, not in exporters. This also has the benefit that Prometheus has the target information it needs to allow users probe your service with the blackbox exporter.

There are two exceptions:

The first is where running beside the application you are monitoring is completely nonsensical. The SNMP, blackbox and IPMI exporters are the main examples of this. The IPMI and SNMP exporters as the devices are often black boxes that it’s impossible to run code on (though if you could run a node exporter on them instead that’d be better), and the blackbox exporter where you’re monitoring something like a DNS name, where there’s also nothing to run on. In this case, Prometheus should still do service discovery, and pass on the target to be scraped. See the blackbox and SNMP exporters for examples.

Note that it is only currently possible to write this type of exporter with the Go, Python and Java client libraries.

The second exception is where you’re pulling some stats out of a random instance of a system and don’t care which one you’re talking to. Consider a set of MySQL replicas you wanted to run some business queries against the data to then export. Having an exporter that uses your usual load balancing approach to talk to one replica is the sanest approach.

This doesn’t apply when you’re monitoring a system with master-election, in that case you should monitor each instance individually and deal with the "masterness" in Prometheus. This is as there isn’t always exactly one master, and changing what a target is underneath Prometheus’s feet will cause oddities.

Scheduling

Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers. That is, all scrapes should be synchronous.

Accordingly, you should not set timestamps on the metrics you expose, let Prometheus take care of that. If you think you need timestamps, then you probably need the Pushgateway instead.

If a metric is particularly expensive to retrieve, i.e. takes more than a minute, it is acceptable to cache it. This should be noted in the HELP string.

The default scrape timeout for Prometheus is 10 seconds. If your exporter can be expected to exceed this, you should explicitly call this out in your user documentation.

Pushes

Some applications and monitoring systems only push metrics, for example StatsD, Graphite and collectd.

There are two considerations here.

Firstly, when do you expire metrics? Collectd and things talking to Graphite both export regularly, and when they stop we want to stop exposing the metrics. Collectd includes an expiry time so we use that, Graphite doesn’t so it is a flag on the exporter.

StatsD is a bit different, as it is dealing with events rather than metrics. The best model is to run one exporter beside each application and restart them when the application restarts so that the state is cleared.

Secondly, these sort of systems tend to allow your users to send either deltas or raw counters. You should rely on the raw counters as far as possible, as that’s the general Prometheus model.

For service-level metrics, e.g. service-level batch jobs, you should have your exporter push into the Pushgateway and exit after the event rather than handling the state yourself. For instance-level batch metrics, there is no clear pattern yet. The options are either to abuse the node exporter’s textfile collector, rely on in-memory state (probably best if you don’t need to persist over a reboot) or implement similar functionality to the textfile collector.

Failed scrapes

There are currently two patterns for failed scrapes where the application you’re talking to doesn’t respond or has other problems.

The first is to return a 5xx error.

The second is to have a myexporter_up, e.g. haproxy_up, variable that has a value of 0 or 1 depending on whether the scrape worked.

The latter is better where there’s still some useful metrics you can get even with a failed scrape, such as the HAProxy exporter providing process stats. The former is a tad easier for users to deal with, as up works in the usual way, although you can’t distinguish between the exporter being down and the application being down.

Landing page

It’s nicer for users if visiting http://yourexporter/ has a simple HTML page with the name of the exporter, and a link to the /metrics page.

Port numbers

A user may have many exporters and Prometheus components on the same machine, so to make that easier each has a unique port number.

https://github.com/prometheus/prometheus/wiki/Default-port-allocations is where we track them, this is publicly editable.

Feel free to grab the next free port number when developing your exporter, preferably before publicly announcing it. If you’re not ready to release yet, putting your username and WIP is fine.

This is a registry to make our users’ lives a little easier, not a commitment to develop particular exporters. For exporters for internal applications we recommend using ports outside of the range of default port allocations.

Announcing

Once you’re ready to announce your exporter to the world, email the mailing list and send a PR to add it to the list of available exporters by editing this GitHub repository file.