Specs

Native Histograms
Native histograms were introduced as an experimental feature in November 2022.
They are a concept that touches almost every part of the Prometheus stack. The
first version of the Prometheus server supporting native histograms was
v2.40.0. The support had to be enabled via a feature flag
--enable-feature=native-histograms. Starting with v3.8.0, native histograms
are supported as a stable feature. However, scraping native histograms still has
to be activated explicitly via the 
scrape_native_histograms configuration
setting. To ease transition from the feature flag to the configuration setting,
setting the feature flag in v3.8 has the only remaining effect to set
scrape_native_histograms to 
true by default. Starting with v3.9, the feature
flag is a true no-op and explicitly setting 
scrape_native_histograms is
required. Sending over Remote-Write needs to be enabled with by the
send_native_histograms remote write config. (From v4 on, both
scrape_native_histograms and 
send_native_histograms will default to 
true.)
Due to the pervasive nature of the changes related to native histograms, the
documentation of those changes and explanation of the underlying concepts are
widely distributed over various channels (like the documentation of affected
Prometheus components, doc comments in source code, sometimes the source code
itself, design docs, conference talks, …). This document intends to gather all
these pieces of information and present them concisely in a unified context.
This document prefers to link existing detailed documentation rather than
restating it, but it contains enough information to be comprehensible without
referring to other sources. With all that said, it should be noted that this
document is neither suitable as an introduction for beginners nor does it focus
on the needs of developers. For the former, the plan is to provide an updated
version of the Best Practices article on histograms and
summaries. (TODO: And a blog post or maybe even a
series of them.) For the latter, there is Carrie
Edward's Developer’s Guide to Prometheus Native
Histograms.
While formal specifications are supposed to happen in their respective context
(e.g. OpenMetrics changes will be specified in the general OpenMetrics
specification), some parts of this document take the shape of a specification.
In those parts, the key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL
NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” are used as
described in RFC 2119.
This document still contains a lot of TODOs even though the feature is
considered stable and we don't expect breaking changes before v4.0.0. These
TODOs are reminders for completing the documentation, fixing minor issues and
additional features.
Introduction
The core idea of native histograms is to treat histograms as first class
citizens in the Prometheus data model. Elevating histograms to a “native”
sample type is the fundamental prerequisite for the key properties listed
below, which explains the choice of the name native histograms.
Prior to the introduction of native histograms, all Prometheus sample values
have been 64-bit floating point values (short float64 or just float). These
floats can directly represent gauges or counters. The Prometheus metric
types summary and (the classic version of) histogram, as they exist in
exposition formats, are broken down into float components upon ingestion: A
sum and a count component for both types, a number of quantile samples
for a summary and a number of bucket samples for a (classic) histogram.
With native histograms, a new structured sample type is introduced. A single
sample represents the previously known sum and count plus a dynamic set of
buckets. This is not limited to ingestion, but PromQL expressions may also
return the new sample type where previously it was only possible to return
float samples.
Native histograms have the following key properties:
A sparse bucket representation, allowing (near) zero cost for empty buckets.
Coverage of the full float64 range of values.
No configuration of bucket boundaries during instrumentation.
Dynamic resolution picked according to simple configuration parameters.
Sophisticated exponential bucketing schemas, ensuring mergeability between
all histograms using those schemas.
An efficient data representation for both exposition and storage.
These key properties are fully realized with standard bucketing schemas. There
are other schemas with different trade-offs that might only feature a subset of
these properties. See the Schema section below for details
Compared to the previously existing “classic” histograms, native histograms
(with standard bucketing schemas) allow a higher bucket resolution across
arbitrary ranges of observed values at a lower storage and query cost with very
little to no configuration required. Even partitioning histograms by labels is
now much more affordable.
Because the sparse representation (property 1 in the list above) is so crucial
for many of the other benefits of native histograms, sparse histograms was
a common name for native histograms early during the design process. However,
other key properties like the exponential bucketing schema or the dynamic
nature of the buckets are also very important, but not caught at all in the
term sparse histograms.
Design docs
These are the design docs that guided the development of native histograms.
Some details are obsolete now, but they describe rather well the underlying
concepts and how they evolved.
Sparse high-resolution histograms for
Prometheus,
the original design doc.
Prometheus Sparse Histograms and
PromQL,
more an exploratory document than a proper design doc about the handling of
native histograms in PromQL.
Conference talks
A more approachable way of learning about native histograms is to watch
conference talks, of which a selection is presented below. As an introduction,
it might make sense to watch these talks and then return to this document to
learn about all the details and technicalities.
Secret History of Prometheus
Histograms about the
classic histograms and why Prometheus kept them for so long.
Prometheus Histograms – Past, Present, and
Future
is the inaugural talk about the new approach that led to native histograms.
Better Histograms for
Prometheus explains why the
concepts work out in practice.
Native Histograms in
Prometheus
presents and explains native histograms after the actual implementation.
PromQL for Native
Histograms
explains the usage of native histograms in PromQL.
Prometheus Native Histograms in
Production provides an analysis
of performance and resource consumption.
Using OpenTelemetry’s Exponential Histograms in
Prometheus covers the
interoperability with the OpenTelemetry.
Glossary
A native histogram is an instance of the new complex sample type
representing a full histogram that this document is about. Where the context
is sufficiently clear, it is often just called a histogram below.
A classic histogram is an instance of the older sample type representing
a histogram with fixed buckets, formerly just called a histogram. It exists
as such in the exposition formats, but is broken into a number of float
samples upon ingestion into Prometheus.
Sparse histogram is an older, now deprecated name for native
histogram. This name might still be found occasionally in older
documentation. Sparse buckets remains a meaningful term for the buckets
of a native histogram.
Data model
This section describes the data model of native histograms in general. It
avoids implementation specifics as far as possible. This includes terminology.
For example, a list described in this section will become a repeated
message in a protobuf implementation and (most likely) a slice in a Go
implementation.
General structure
Similar to a classic histogram, a native histogram has a field for the count
of observations and a field for the sum of observations. While the count of
observation is generally non-negative (with the only exception being
intermediate results in PromQL), the
sum of observations might have any float64 value.
In addition, a native histogram contains the following components, which are
described in detail in dedicated sections below:
A schema to identify the method of determining the boundaries of any given
bucket with an index i.
A sparse representation of indexed buckets, mirrored for positive and
negative observations.
A zero bucket to count observations close to zero.
A (possibly empty) list of custom values.
Exemplars.
Flavors
Any native histogram has a specific flavor along each of two independent
dimensions:
Counter vs. gauge: Usually, a histogram is “counter like”, i.e. each of its
buckets acts as a counter of observations. However, there are also “gauge
like” histograms where each bucket is a gauge, representing arbitrary
distributions at a point in time. The concept of a gauge histogram was
previously introduced for classic histograms by
OpenMetrics.
Integer vs. floating point (short: float): The obvious use case of
histograms is to count observations, resulting in integer numbers of
observations ≥ 0 within each bucket, including the zero bucket, and for
the total count of observations, represented as unsigned 64-bit integers
(short: uint64). However, there are specific use cases leading to a
“weighted” or “scaled” histogram, where all of these values are represented
as 64-bit floating point numbers (short: float64). Note that the sum of
observations is a float64 in either case.
Float histograms are occasionally used in direct instrumentation for “weighted”
observations, for example to count the number of seconds an observed value was
falling into different buckets of a histogram. The far more common use case for
float histograms is within PromQL, though. PromQL generally only acts on float
values, so the PromQL engine converts every histogram retrieved from the TSDB
to a float histogram first, and any histogram stored back into TSDB via
recording rules is a float histogram. If such a histogram is effectively an
integer histogram (because the value of all non-sum fields can be represented
precisely as uint64), a TSDB implementation MAY convert them back to integer
histograms to increase storage efficiency. (As of Prometheus v3.00, the TSDB
implementation within Prometheus is not utilizing this option.) Note, however,
that the most common PromQL function applied to a counter histogram is 
rate,
which generally produces non-integer numbers, so that results of recording
rules will commonly be float histograms with non-integer values anyway.
PromQL expression may even create “negative” histograms (e.g. by multiplying a
histogram with -1). Those negative histograms are only allowed as intermediate
results and are otherwise considered invalid. They cannot be represented in any
of the exchange formats (exposition formats, remote-write, OTLP) and they
cannot be stored in the TSDB. Also see the detailed section about negative
histograms.
Treating native histograms explicitly as integer histograms vs. float histogram
is a notable deviation from the treatment of conventional simple numeric
samples, which are always treated as floats throughout the whole stack for the
sake of simplicity.
The main reason for the more involved treatment of histograms is the easy
efficiency gains in protobuf-based exposition formats. Protobuf uses varint
encoding for integers, which reduces the data size for small integer values
without requiring an additional compression layer. This benefit is amplified by
the delta encoding of integer buckets, which generally results in
smaller integer values. Floats, in contrast, always require 8 bytes in
protobuf. In practice, many integers in an integer histogram will fit in 1
byte, and most will fit in 2 bytes, so that the explicit presence of integer
histogram in a protobuf-exposition format results directly in a data size
reduction approaching 8x for histograms with many buckets. This is particularly
relevant as the overwhelming majority of histograms exposed by instrumented
targets are integer histograms.
For similar reasons, the representation of integer histograms in RAM and on
disk is generally more efficient than that of float histograms. This is less
relevant than the benefits in the exposition format, though. For one,
Prometheus uses Gorilla-style XOR encoding for floats, which reduces their
size, albeit not as much as the double-delta encoding used for integers. More
importantly, an implementation could always decide to internally use an integer
representation for histogram fields that are effectively integer values (see
above). (Historical note: Prometheus v1 used exactly this approach to improve
the compression of float samples, and Prometheus v3 might very well adopt this
approach again in the future.)
In a counter histogram, the total count of observation and the counts in the
buckets individually behave like Prometheus counters, i.e. they only go down
upon a counter reset. However, the sum of observation may decrease as a
consequence of the observation of negative values. PromQL implementations MUST
detect counter resets based on the whole histogram (see the counter reset
considerations section below for details).
(Note that this always has been a problem for the sum component of classic
histograms and summaries, too. The approach so far was to accept that counter
reset detection silently breaks for sum in those cases. Fortunately, negative
observations are a very rare use case for Prometheus histograms and summaries.)
Schema
The schema is a signed integer value with a size of 8 bits (short: int8). It
defines the way bucket boundaries are calculated. The currently valid values
are -53 and the range between and including -4 and +8 (with a larger range
between and including -9 and +52 being reserved, see below for details). More
schemas may be added in the future. -53 is a schema for so-called custom
bucket boundaries or short custom buckets, while the other schema numbers
represent the different standard exponential schemas (short: standard
schemas).
The standard schemas are mergeable with each other and are RECOMMENDED for
general use cases. Larger schema numbers correspond to higher resolutions.
Schema n has half the resolution of schema n+1, which implies that a
histogram with schema n+1 can be converted into a histogram with schema n
by merging neighboring buckets.
For any standard schema n, the boundaries of a bucket with index i
calculated as follows (using Python syntax):
The upper inclusive limit of a positive bucket:
(2**2**-n)**i
The lower exclusive limit of a positive bucket:
(2**2**-n)**(i-1)
The lower inclusive limit of a negative bucket:
-((2**2**-n)**i)
The upper exclusive limit of a negative bucket:
-((2**2**-n)**(i-1))
i is an integer number that may be negative.
There are exceptions to the rules above concerning the largest and smallest
finite values representable as a float64 (called 
MaxFloat64 and 
MinFloat64
in the following) and the positive and negative infinity values (
+Inf and
-Inf):
The positive bucket that contains 
MaxFloat64 (according to the boundary
formulas above) has an upper inclusive limit of 
MaxFloat64 (rather than the
limit calculated by the formulas above, which would overflow float64).
The next positive bucket (index i+1 relative to the bucket from the
previous item) has a lower exclusive limit of 
MaxFloat64 and an upper
inclusive limit of 
+Inf. (It could be called a positive overflow bucket.)
The negative bucket that contains 
MinFloat64 (according to the boundary
formulas above) has a lower inclusive limit of 
MinFloat64 (rather than the
limit calculated by the formulas above, which would underflow float64).
The next negative bucket (index i+1 relative to the bucket from the
previous item) has an upper exclusive limit of 
MinFloat64 and an lower
inclusive limit of 
-Inf. (It could be called a negative overflow bucket.)
Buckets beyond the 
+Inf and 
-Inf buckets described above MUST NOT be used.
There are more exceptions for values close to zero, see the zero bucket
section below.
The current limits of -4 for the lowest resolution and 8 for the highest
resolution have been chosen based on practical usefulness. Should a practical
need arise for even lower or higher resolution, an extension of the range will
be considered. However, a schema greater than 52 does not make sense as the
growth factor from one bucket to the next would then be smaller than the
difference between representable float64 numbers. Likewise, a schema smaller
than -9 does not make sense either, as the growth factor would then exceed the
largest float representable as float64. Therefore, the schema numbers between
(and including) -9 and +52 are reserved for future standard schemas (following
the formulas for bucket boundaries above) and MUST NOT be used for any other
schemas.
Receivers of native histograms MAY, upon ingestion, reduce the schema and
thereby the resolution of ingested histograms by merging buckets appropriately.
Receivers MAY accept schemas between 9 and 52 if they reduce the schema upon
ingestion to a valid number (i.e. between -4 and 8), following the formulas for
bucket boundaries above.
If, after this optional schema conversion, the schema is still unknown to the
receiver, there are the following options:
If a scrape (including federation) contains one or more histograms with an
unknown schema, the entire scrape MUST fail, following the Prometheus
practice of avoiding incomplete scrapes.
For any other ingestion paths (including replaying the WAL/WBL), the receiver
MAY ignore histograms with unknown schemas and SHOULD notify the user about
this omission in a suitable way.
When a TSDB implementation reads histograms from its permanent storage
(excluding replaying the WAL/WBL), similar guidelines apply: Schemas between 9
and 52 MAY be converted to valid schemas. Otherwise, unknown schemas MUST
return an error on retrieval, and the PromQL query that triggered the retrieval
MUST fail.
For schema -53, the bucket boundaries are set explicitly via custom values,
described in detail in the custom values section below. This
results in a native histogram with custom bucket boundaries (or short custom
buckets, often further abbreviated to NHCB). Such a histogram can be used to
represent a classic histogram as a native histogram. It can also be used if
the exponential bucketing featured by the standard schemas is a bad match for
the distribution to be represented by the histogram. Histograms with different
custom bucket boundaries are generally not mergeable with each other.
Therefore, schema -53 SHOULD only be used as an informed decision in specific
use cases.
Buckets
For standard schemas, buckets are represented as two lists, one for positive
buckets and one for negative buckets. For custom buckets (schema -53), only the
positive bucket list is used, but repurposed for all buckets.
Any unpopulated buckets MAY be excluded from the lists. (Which is the reason
why the buckets are often called sparse buckets.)
For float histograms, the elements of the lists are float64 and represent the
bucket population directly. Bucket populations are generally non-negative, with
the only exception being intermediate results in
PromQL.
For integer histograms, the elements of the lists are signed 64-bit integers
(short: int64), and each element represents the bucket population as a delta to
the previous bucket in the list. The first bucket in each list contains an
absolute population (which can also be seen as a delta relative to zero). The
deltas MUST NOT evaluate to a negative absolute bucket population.
To map buckets in the lists to the indices as defined in the previous section,
there are two lists of so-called spans, one for the positive buckets and one
for the negative buckets.
Each span consists of a pair of numbers, a signed 32-bit integer (short: int32)
called offset and an unsigned 32-bit integer (short: uint32) called length.
Only the first span in each list can have a negative offset. It defines the
index of the first bucket in its corresponding bucket list. (Note that for
NHCBs, the index is always positive, see the custom values
section below for details.) The length defines the number of
consecutive buckets the bucket list starts with. The offsets of the following
spans define the number of excluded (and thus unpopulated buckets). The lengths
define the number of consecutive buckets in the list following the excluded
buckets.
The sum of all length values in each span list MUST be equal to the length of
the corresponding bucket list.
Empty spans (with a length of zero) are valid and MAY be used, although they
are generally not useful and they SHOULD be eliminated by adding their offset
to the offset of the following span. Similarly, spans that are not the first
span in a list MAY have an offset of zero, although those offsets SHOULD be
eliminated by adding their length to the previous span. Both cases are allowed
so that producers of native histograms MAY pick whatever representation has the
best resource trade-offs at that moment. For example, if a histogram is
processed through various stages, it might be most efficient to only eliminate
redundant spans after the last processing stage.
In a similar spirit, there are situation where excluding every unpopulated
bucket from the bucket list is most efficient, but in other situations, it
might be better to reduce the number of spans by representing small numbers of
unpopulated buckets explicitly.
Note that future high resolution schemas might require offsets that are too
large to be represented with an int32. An extension of the data model will be
required in that case. (The current standard schema with the highest resolution
is schema 8, for which the bucket that contains 
MaxFloat64 has index 262144,
and thus the 
+Inf overflow bucket has index 262145, while the largest number
representable with int32 is 2147483647. The highest standard schema that would
still work with int32 offsets would be schema 20, corresponding to a growth
factor from bucket to bucket of only ~1.000000661.)
Examples
An integer histogram has the following positive buckets (index→population):
-2→3, -1→5, 0→0, 1→0, 2→1, 3→0, 4→3, 5→2
They could be represented in this way:
Positive bucket list: 
[3, 2, -4, 2, -1]
Positive span list: 
[[-2, 2], [2,1], [1,2]]
The second and third span could be merged into one if the single unpopulated
bucket with index 3 is represented explicitly, leading to the following result:
Positive bucket list: 
[3, 2, -4, -1, 3, -1]
Positive span list: 
[[-2, 2], [2,4]]
Or merge all the spans into one by representing all unpopulated buckets above
explicitly:
Positive bucket list: 
[3, 2, -5, 0, 1, -1, 3, -1]
Positive span list: 
[[-2, 8]]
Zero bucket
Observations of exactly zero do not fit into any bucket as defined by the
standard schemas above. They are counted in a dedicated bucket called the zero
bucket.
The number of observations in the zero bucket is tracked by a single uint64
(for integer histograms) or float64 (for float histograms). As for regular
buckets, this number is generally non-negative.
The zero bucket has an additional parameter called the zero threshold, which
is a float64 ≥ 0. If the threshold is set to zero, only observations of exactly
zero go into the zero bucket, which is the case described above. If the
threshold has a positive value, all observations within the closed interval
[-threshold, +threshold] go to the zero bucket rather than a regular bucket.
This has two use cases:
Noisy observations close to zero tend to populate a high number of buckets.
Those observations might happen due to numerical inaccuracies or if the
source of the observations are actual physical measurements. A zero bucket
with a relatively small threshold redirects those observations into a single
bucket.
If the user is more interested in the long tail of a distribution, far away
from zero, a relatively large threshold of the zero bucket helps to avoid
many high resolution buckets for a range that is not of interest.
The threshold of the zero bucket SHOULD coincide with a boundary of a regular
bucket, which avoids the complication of the zero bucket overlapping with parts
of a regular bucket. However, if such an overlap is happening, the observations
that are counted in the regular bucket overlapping with the zero bucket MUST be
outside of the [-threshold, +threshold] interval.
To merge histograms with the same zero threshold, the two zero buckets are
simply added. If the zero thresholds in the source histograms are different,
however, the largest threshold in any of the source histograms is chosen. If
that threshold happens to be within any populated bucket in the other source
histograms, the threshold is increased until one of the following is true for
each source histogram:
The new threshold coincides with the boundary of a populated bucket.
The new threshold is not within any populated bucket.
Then the source zero buckets and any source buckets now inside the new
threshold are added up to yield the population of the new zero bucket.
The zero bucket is not used if the schema is -53 (custom buckets).
Custom values
The list of custom values is unused for standard schemas. It is used by
non-standard schemas in a custom way in case there is need to store additional
data.
The only currently defined schema for which custom values are used is -53
(custom buckets). The remaining part of this section describes the usage of the
custom values in more detail for this specific case.
The custom values represent the upper inclusive boundaries of the custom
buckets. They are sorted in ascending fashion. The custom buckets themselves
are stored using the positive bucket list and the positive span list, although
their boundaries, as determined via the custom values, can be negative. The
index of each of those “positive” buckets defines the zero-based position of
their upper boundary within the custom values list.
The lower exclusive boundary is defined by the custom value preceding the upper
boundary. For the first custom value (at position zero in the list), there is
no preceding value, in which case the lower boundary is considered to be 
-Inf
inclusively. Therefore, the custom bucket with index zero counts all
observations between (and including) 
-Inf and the first custom value. In the
common case that only positive observations are expected, the custom bucket
with index zero SHOULD have an upper boundary of zero to clearly mark if there
have been any observations at zero or below. (If there are indeed only positive
observations, the custom bucket with index zero will stay unpopulated and
therefore will never be represented explicitly. The only cost is the additional
zero element at the beginning of the custom values list.)
Custom values MUST NOT be 
+Inf. Observations greater than the last custom
value go into an overflow bucket with an upper boundary of 
+Inf. This
overflow bucket is added with an index equal to the length of the custom values
list. As a consequence, the upper boundary of the 
+Inf bucket often included
in classic histograms is not represented explicitly in the custom values.
Custom values MUST NOT be 
NaN. This is explicitly excluded in OpenMetrics,
but other exposition formats could, in principle, feature upper boundaries of
NaN in classic histograms (presumably as a result of some error – such a
boundary would not make any sense). Such a classic histogram MUST be rejected
and cannot be converted into an NHCB.
Exemplars
A native histogram sample can have zero, one, or more exemplars. They work in
the same way as conventional exemplars, but they are organized in a list (as
there can be more than one), and they MUST have a timestamp.
Exemplars exposed as part of a classic histogram MAY be used by native
histograms, if they have a timestamp.
Special cases of observed values
Instrumented code SHOULD avoid observing values of 
NaN and 
±Inf because
they make limited sense in the context of a histogram. However, those values
MUST still be handled properly, as described in the following.
The sum of observations is calculated as usual by adding the observation to the
sum of observations, following normal floating point arithmetic. (For example,
an observation of 
NaN will set the sum to 
NaN. An observation of 
+Inf
will set the sum to 
+Inf, unless it is already 
NaN or 
-Inf, in which case
the sum is set to 
NaN.)
An observation of 
NaN goes into no bucket, but increments the count of
observations. This implies that the count of observations can be greater than
the sum of all buckets (negative, positive, and zero buckets), and the
difference is the number of 
NaN observations. (For an integer histogram
without any 
NaN observations, the sum of all buckets is equal to the count of
observations. Within the usual floating point precision limits, the same is
true for a float histogram without any 
NaN observations.)
An observation of 
+Inf or 
-Inf increments the count of observations and
increments a bucket chosen in the following way:
With a standard schema, a 
+Inf observation increments the
positive overflow bucket as described above.
With a standard schema, a 
-Inf observation increments the
negative overflow bucket as described above.
With schema -53 (custom buckets), a 
+Inf observation increments the
bucket with an index equal to the length of the custom values list.
With schema -53 (custom buckets), a 
-Inf observation increments the
bucket with index zero.
OpenTelemetry interoperability
Prometheus (Prom) native histograms with a standard schema can be
easily mapped into an OpenTelemetry (OTel) exponential histogram and vice
versa, as detailed in the following.
The Prom schema is equal to the scale in OTel, with the restriction that
OTel allows lower values than -4 and higher values than +8. As described above,
Prom has reserved more schema numbers to extend its range, should it ever by
required in practice.
The index is offset by one, i.e. a Prom bucket with index n has index n-1
for OTel.
OTel has a dense rather than a sparse representation of buckets. One might see
OTel as “Prom with only one span”.
The Prom zero bucket is called zero count in OTel. (Prom also uses zero
count to name the field storing the count of observations in the zero bucket).
Both work the same, including the existence of a zero threshold. Note that
OTel implies a threshold of zero if none is given.
(TODO: The OTel spec reads: “When zero_threshold is unset or 0, this bucket
stores values that cannot be expressed using the standard exponential formula
as well as values that have been rounded to zero.” Double-check if this really
creates the same behavior. If there are problems close to zero, we could make
Prom's spec more precise. If OTel counts NaN in the zero bucket, we have to add
a note here.)
OTel exponential histograms only support standard exponential bucketing schemas
(as the name suggests). Therefore, NHCBs (or native histograms with other future
bucketing schemas) cannot be cleanly converted to OTel exponential histograms.
However, conversion to a conventional OTel histogram with fixed buckets is
still possible.
OTel histograms of any kind have optional fields for the minimum and maximum
value observed in the histogram. These fields have no equivalent concept in
Prometheus because counter histograms accumulate data over a long and
unpredictable timespan and can be scraped at any time, so that tracking a
minimum and maximum value is either infeasible or of limited use. Note, though,
that native histograms enable a fairly accurate estimation of the maximum and
minimum observation during arbitrary timespans, see the PromQL
section.
Exposition formats
Metrics exposition in the classic Prometheus use case is dominated by strings
because all the metric names, label names, and label values take much more
space than the float64 sample values, even if the latter are represented in a
potentially more verbose text form. This was one of the reasons why abandoning
protobuf-based exposition seemed advantageous in the past.
In contrast, a native histogram, following the data model described above,
consists of a lot more numerical data. This amplifies the advantages of a
protobuf based format. Therefore, the previously abandoned protobuf-based
exposition was revived to efficiently expose and scrape native histograms.
Classic Prometheus formats
At the time native histograms were conceived, OpenMetrics adoption was still
lacking, and in particular, the protobuf version of OpenMetrics had no known
applications at all. Therefore, the initial approach was to extend the classic
Prometheus protobuf format to support native histograms. (An additional
practical consideration was that the Go instrumentation
library was still using the
classic protobuf spec as its internal data model, simplifying the initial
development.)
The classic Prometheus text form was not extended for native histograms, and
such an extension is not planned. (See also the OpenMetrics
section below.)
There is a proto2 and a proto3 version of the protobuf specification, which
both create the same wire format:
proto2
proto3
These files have comprehensive comments, which should enable an easy mapping
from the proto spec to the data model described above.
Here are relevant parts from the proto3 file:
// [...]
message Histogram {
  uint64 sample_count       = 1;
  double sample_count_float = 4; // Overrides sample_count if > 0.
  double sample_sum         = 2;
  // Buckets for the classic histogram.
  repeated Bucket bucket = 3 [(gogoproto.nullable) = false]; // Ordered in increasing order of upper_bound, +Inf bucket is optional.
  google.protobuf.Timestamp start_timestamp = 15;
  // Everything below here is for native histograms (also known as sparse histograms).
  // schema defines the bucket schema. Currently, valid numbers are -4 <= n <= 8.
  // They are all for base-2 bucket schemas, where 1 is a bucket boundary in each case, and
  // then each power of two is divided into 2^n logarithmic buckets.
  // Or in other words, each bucket boundary is the previous boundary times 2^(2^-n).
  // In the future, more bucket schemas may be added using numbers < -4 or > 8.
  sint32 schema           = 5;
  double zero_threshold   = 6; // Breadth of the zero bucket.
  uint64 zero_count       = 7; // Count in zero bucket.
  double zero_count_float = 8; // Overrides sb_zero_count if > 0.
  // Negative buckets for the native histogram.
  repeated BucketSpan negative_span = 9 [(gogoproto.nullable) = false];
  // Use either "negative_delta" or "negative_count", the former for
  // regular histograms with integer counts, the latter for float
  // histograms.
  repeated sint64 negative_delta = 10; // Count delta of each bucket compared to previous one (or to zero for 1st bucket).
  repeated double negative_count = 11; // Absolute count of each bucket.
  // Positive buckets for the native histogram.
  // Use a no-op span (offset 0, length 0) for a native histogram without any
  // observations yet and with a zero_threshold of 0. Otherwise, it would be
  // indistinguishable from a classic histogram.
  repeated BucketSpan positive_span = 12 [(gogoproto.nullable) = false];
  // Use either "positive_delta" or "positive_count", the former for
  // regular histograms with integer counts, the latter for float
  // histograms.
  repeated sint64 positive_delta = 13; // Count delta of each bucket compared to previous one (or to zero for 1st bucket).
  repeated double positive_count = 14; // Absolute count of each bucket.
  // Only used for native histograms. These exemplars MUST have a timestamp.
  repeated Exemplar exemplars = 16;
}
message Bucket {
  uint64   cumulative_count       = 1; // Cumulative in increasing order.
  double   cumulative_count_float = 4; // Overrides cumulative_count if > 0.
  double   upper_bound            = 2; // Inclusive.
  Exemplar exemplar               = 3;
}
// A BucketSpan defines a number of consecutive buckets in a native
// histogram with their offset. Logically, it would be more
// straightforward to include the bucket counts in the Span. However,
// the protobuf representation is more compact in the way the data is
// structured here (with all the buckets in a single array separate
// from the Spans).
message BucketSpan {
  sint32 offset = 1; // Gap to previous span, or starting point for 1st span (which can be negative).
  uint32 length = 2; // Length of consecutive buckets.
}
// A BucketSpan defines a number of consecutive buckets in a native
// histogram with their offset. Logically, it would be more
// straightforward to include the bucket counts in the Span. However,
// the protobuf representation is more compact in the way the data is
// structured here (with all the buckets in a single array separate
// from the Spans).
message BucketSpan {
  sint32 offset = 1; // Gap to previous span, or starting point for 1st span (which can be negative).
  uint32 length = 2; // Length of consecutive buckets.
}
// [...]
Note the following:
Both native histograms and classic histograms are encoded by the same
Histogram proto message, i.e. the existing 
Histogram message got extended
with fields for native histograms.
The fields for the sum and the count of observations and the
start_timestamp are shared between classic and native histograms and keep
working in the same way for both.
The format originally did not support classic float histograms. While
extending the format for native histograms, support for classic float
histograms was added as a byproduct (see fields 
sample_count_float,
cumulative_count_float).
The 
Bucket field and the 
Bucket message are used for the buckets of a
classic histogram. It is perfectly possible to create a 
Histogram message
that represents both a classic and a native version of the same histogram.
Parsers have the freedom to pick either or both versions (see also the
scrape configuration section).
The bucket population is encoded as absolute numbers in case of float
histograms, and as deltas to the previous bucket (or to zero for the first
bucket) in case of integer histograms. The latter leads to smaller numbers,
which encode to a smaller message size because protobuf uses varint encoding
for the 
sint64 type.
A native histogram that has received no observations yet and a classic
histogram that has no buckets configured would look exactly the same as a
protobuf message. Therefore, a 
Histogram message that is meant to be parsed
as a native histogram MUST contain a “no-op span”, i.e. a 
BucketSpan with
offset and 
length set to 0, in the repeated 
positive_span field.
Any number of exemplars for native histograms MAY be added in the repeated
Exemplar field of the 
Histogram message, but each one MUST have a
timestamp. If there are no exemplars provided in this way, a parser MAY use
timestamped exemplars provided for classic buckets (as at most one exemplar
per bucket in the 
Exemplar field of the 
Bucket message).
The number and distribution of native histogram exemplars SHOULD fit the use
case at hand. Generally, the exemplar payload SHOULD NOT be much larger than
the remaining part of the 
Histogram message, and the exemplars SHOULD fall
into different buckets and cover the whole spread of buckets approximately
evenly. (This is generally preferred over an exemplar distribution that
proportionally represents the distribution of observations, as the latter
will rarely yield exemplars from the long tail of a distribution, which are
often the most interesting exemplars to look at.)
There is no representation for the custom values needed for NHCBs. NHCBs are
never directly exposed, but presented as classic histograms, to be converted
(back) to NHCB upon ingestion. This is also true for
federation. We might still add fields for the custom values in
the future, should the need arise, e.g. for future schemas that also utilize
custom values.
OpenMetrics
Currently (2024-11-03), OpenMetrics does not support native histograms.
Adding support to the protobuf version of OpenMetrics is relatively
straightforward due to its similarity to the classic Prometheus protobuf
format. A proposal in the form of a
PR is under review.
Adding support to the text version of OpenMetrics is harder, but also highly
desirable because there are many situations where the generation of protobuf is
infeasible. A text format has to make a trade-off between readability for
humans and efficient handling by machines (encoding, transport, decoding). Work
on it is in progress. See the design
doc
for more details.
(TODO: Update section as progress is made.)
Instrumentation libraries
The protobuf specification enables low-level
creation of metrics exposition including native histograms using the language
specific bindings created by the protobuf compiler. However, for direct code
instrumentation, an instrumentation library is needed.
Currently (2026-06-15), there are three official Prometheus instrumentation
libraries supporting native histograms:
Go: source –
documentation
Java: source –
documentation
Rust: source –
documentation
Adding native histogram support to other instrumentation libraries is
relatively easy if the library already supports protobuf exposition. For purely
text based libraries, the completion of a text based exposition
format is a prerequisite. (TODO: Update this as needed.)
This section does not cover details of how to use individual instrumentation
libraries (see the documentation linked above for that) but focuses on the
common usage patterns and also provides general guidelines how to implement
native histogram support as part of an instrumentation library. The already
existing Go implementation is
used for examples. The sections about the data model and the
exposition formats are highly relevant for the
implementation of instrumentation libraries (but not restated in this
section!).
The actual instrumentation API for histograms does not change for native
histograms. Both classic histograms and native histograms receive observations
in the same way (with subtle differences concerning exemplars, see next
paragraph). Instrumentation libraries can even maintain a classic and a native
version of the same histogram and expose them in parallel so that the scraper
can choose which version to ingest (see the section about exposition
formats for details). The user chooses whether to expose
classic and/or native histograms via configuration settings.
Exemplars for classic histograms are usually tracked by storing and exposing
the most recent exemplar for each bucket. As long as classic buckets are
defined, an instrumentation library MAY expose the same exemplars for the
native version of the same histogram, as long as each exemplar has a timestamp.
(In fact, a scraper MAY use the exemplars provided with the classic version of
the histogram even if it is otherwise only ingesting the native version, see
details in the exposition formats section.) However, a
native histogram MAY be assigned any number of exemplars, and an
instrumentation library SHOULD use this liberty to meet the best practices for
exemplars as described in the exposition formats
section.
An instrumentation library SHOULD offer the following configuration parameters
for native histograms following standard schemas. Names are
examples from the Go library – they have to be adjusted to the idiomatic style
in other languages. The value in parentheses is the default value that the
library SHOULD offer.
NativeHistogramBucketFactor (1.1): A float greater than one to determine
the initial resolution. The library picks a starting schema that results in a
growth of the bucket width from one bucket to the next by a factor not larger
than the provided value. See table below for example values.
NativeHistogramZeroThreshold (2-128): A float of value zero or
greater to set the initial threshold for the zero bucket.
The resolution is set via a growth factor rather than providing the schema
directly because most users will not know the mathematics behind the schema
numbers. The notion of an upper limit for the growth factor from bucket to
bucket is understandable without knowing about the internal workings of native
histograms. The following table lists an example factor for each valid schema.
NativeHistogramBucketFactor
resulting schema
65536
-4
256
-3
16
-2
4
-1
2
0
1.5
1
1.2
2
1.1
3
1.05
4
1.03
5
1.02
6
1.01
7
1.005
8
Limiting the bucket count
Buckets of native histograms are created dynamically when they are populated
for the first time. An unexpectedly broad distribution of observed values can
lead to an unexpectedly high number of buckets, requiring more memory than
anticipated. If the distribution of observed values can be manipulated from the
outside, this could even be used as a DoS attack vector via exhausting all the
memory available to the program. Therefore, an instrumentation library SHOULD
offer a bucket limitation strategy. It MAY set one by default, depending on the
typical use cases the library is used for. (TODO: Maybe we should say that a
strategy SHOULD be set by default. The Go library is currently not limiting the
buckets by default, and no issues have been reported with that so far.)
The following describes the bucket limitation strategy implemented by the Go
instrumentation library. Other libraries MAY follow this example, but other
strategies might be feasible as well, depending on the typical usage pattern of
the library.
The strategy is defined by three parameters: an unsigned integer
NativeHistogramMaxBucketNumber, a duration 
NativeHistogramMinResetDuration,
and a float 
NativeHistogramMaxZeroThreshold. If
NativeHistogramMaxBucketNumber is zero (which is the default), buckets are
not limited at all, and the other two parameters are ignored. If
NativeHistogramMaxBucketNumber is set to a positive value, the library
attempts to keep the bucket count of each histogram to the provided value. A
typical value for the limit is 160, which is also the default value used by
OTel exponential histograms in a similar strategy. (Note that partitioning by
labels will create a number of histograms. The limit applies to each of them
individually, not to all of them in aggregate.) If the limit would be exceeded,
a number of remedies are applied in order until the number of buckets is within
the limit again:
If at least 
NativeHistogramMinResetDuration has passed since the last
reset of the histogram (which includes the creation of the histogram), the
whole histogram is reset, i.e. all buckets are deleted and the sum and count
of observations as well as the zero bucket are set to zero. Prometheus
handles this as a normal counter reset, which means that some observations
will be lost between scrapes, so resetting should happen rarely compared to
the scraping interval. Additionally, frequent counter resets might lead to
less efficient storage in the TSDB (see the TSDB section for
details). A 
NativeHistogramMinResetDuration of one hour is a value that
should work well in most situations.
If not enough time has passed since the last reset (or if
NativeHistogramMinResetDuration is set to zero, which is the default
value), no reset is performed. Instead, the zero threshold is increased to
merge buckets close to zero into the zero bucket, reducing the number of
buckets in that way. The increase of the threshold is limited by
NativeHistogramMaxZeroThreshold. If this value is already reached (or it
is set to zero, which is the default), nothing happens in this step.
If the number of buckets still exceeds the limit, the resolution of the
histogram is reduced by converting it to the next lower schema, i.e. by
merging neighboring buckets, thereby doubling the width of the buckets. This
is repeated until the bucket count is within the configured limit or schema
-4 is reached.
If step 2 or 3 have changed the histogram, a reset will be performed once
NativeHistogramMinResetDuration has passed since the last reset, not only to
remove the buckets but also to return to the initial values for the zero
threshold and the bucket resolution. Note that this is treated like a reset for
other reasons in all aspects, including updating the start
timestamp.
It is tempting to set a very low 
NativeHistogramBucketFactor (e.g. 1.005)
together with a reasonable 
NativeHistogramMaxBucketNumber (e.g. 160). In this
way, each histogram always has the highest possible resolution that is
affordable within the given bucket count “budget”. (This is the default
strategy used by the OTel exponential histogram. It starts with an even higher
schema (20), which is currently not even available in Prometheus native
histograms.) However, this strategy is generally not recommended for the
Prometheus use case. The resolution will be reduced quite often after creation
and after each reset as observations come in. This creates churn both in the
instrumented program as well as in the TSDB, which is particularly problematic
for the latter. All of this effort is mostly in vain because the typical
queries involving histograms require many histograms to get merged, during
which the lowest common resolution is used so that the user ends up with a
lower resolution anyway. The TSDB can be protected against the churn by
limiting the resolution upon ingestion (see
below), but if a reasonably low
resolution will be enforced upon ingestion anyway, it is more straightforward
to set this resolution during instrumentation already. However, this strategy
might be worth the resource overhead within the instrumented program in
specific cases where a reasonable resolution cannot be assumed at
instrumentation time, and the scraper should have the flexibility to pick the
desired resolution at scrape time.
Partitioning by labels
While partitioning of a classic histogram with many buckets by labels has to be
done judiciously, the situation is more relaxed with native histograms.
Partitioning a native histograms still creates a multiplicity of individual
histograms. However, the resulting partitioned histograms will often populate
fewer buckets each than the original unpartitioned histogram. (For example, if
a histogram tracking the duration of HTTP requests is partitioned by HTTP
status code, the individual histogram tracking requests responded by status
code 404 might have a very sharp bucket distribution around the typical
duration it takes to identify an unknown path, populating only a few buckets.)
The total number of populated buckets for all partitioned histograms will still
go up, but by a smaller factor than the number of partitioned histograms. (For
example, if adding labels to an already quite heavy classic histogram results
in 100 labeled histograms, the total cost will go up by a factor of 100. In
case of a native histogram, the cost for the single histogram might already be
lower if the classic histogram featured a high resolution. After partitioning,
the total number of populated buckets in the labeled native histograms will be
significantly smaller than 100 times the number of buckets in the original
native histogram.)
NHCB
Currently (2024-11-03), instrumentation libraries offer no way to directly
configure native histograms with custom bucket boundaries (NHCBs). The use case
for NHCBs is to allow native-histogram enabled scrapers to convert classic
histograms to NHCBs upon ingestion (see next section).
However, there are valid use cases where custom buckets are desirable directly
during instrumentation. In those cases, the current approach is to instrument
with a classic histogram and configure the scraper to convert it to an NHCB
upon ingestion. However, a more direct treatment of NHCBs in instrumentation
libraries might happen in the future.
Scrape configuration
To enable the Prometheus server to scrape native histograms, set
scrape_native_histograms: true in individual scrape configs, or in the
global settings. Enabling 
scrape_native_histograms also changes the content
negotiation to prefer classic protobuf-based exposition format over the
OpenMetrics 1.x text format.
Fine-tuning content negotiation
It is possible to fine-tune the scrape protocol negotiation globally or per
scrape config via the 
scrape_protocols config setting. It is a list defining
the content negotiation priorities. Its value depends on what feature flags are
enabled (for example 
--enable-feature=created-timestamp-zero-ingestion), what
value the user sets in it directly and lastly whether
scrape_native_histograms is enabled.
If 
scrape_native_histograms is enabled and 
scrape_protocols is not set by
a feature flag or the user globally or per scrape config, then its effective
value for a scrape config is changed to
[ PrometheusProto, OpenMetricsText1.0.0,OpenMetricsText0.0.1, PrometheusText0.0.4 ]
to enable scraping native histograms.
The 
scrape_protocols setting can be used to configure protobuf scrapes without
ingesting native histograms or enforce a non-protobuf format for certain targets
even with 
scrape_native_histograms enabled. As long as the classic
Prometheus protobuf format (
PrometheusProto in the configured list) is the
only format supporting native histograms, both 
scrape_native_histograms and
negotiation of protobuf is required to actually ingest native histograms.
NOTE: Switching the used exposition format between text-based and
protobuf-based has some non-obvious implications. Most importantly, certain
implementation details result in the counter-intuitive effect that scraping
with a text-based format is generally much less resource demanding than
scraping with a protobuf-based format (see tracking
issue for details).
Even more subtle is the effect on the formatting of label values for 
quantile
labels (used in summaries) and 
le labels (used in classic histograms). This
problem only affects v2 of the Prometheus server (v3 has consistent formatting
under all circumstances) and is not directly related to native histograms, but
might show up in the same context because enabling native histograms requires
the protobuf exposition format. See details in the documentation for the
native-histograms feature
flag
for v2.55.
Limiting bucket count and resolution
While instrumentation libraries SHOULD offer
configuration options to limit the resolution and bucket count of a native
histogram, there is still a need to enforce those limits upon ingestion. Users
might be unable to change the instrumentation of a given program, or a program
might be deliberately instrumented with high-resolution histograms to give
different scrapers the option to reduce the resolution as they see fit.
The Prometheus scrape config offers two settings to address this need:
The 
native_histogram_bucket_limit sets an upper inclusive limit for the
number of buckets in an individual histogram. If the limit is exceeded, the
resolution of a histogram with a standard schema is repeatedly
reduced (by doubling the width of the buckets, i.e. decreasing the schema)
until the limit is reached. In case an NHCB exceeds the limit, or in the rare
case that the limit cannot be satisfied even with schema -4, the scrape
fails.
The 
native_histogram_min_bucket_factor sets a lower inclusive limit for
the growth factor from bucket to bucket. This setting is only relevant for
standard schemas and has no effect on NHCBs. Again, if the limit
is exceeded, the resolution of the histogram is repeatedly reduced (by
doubling the width of the buckets, i.e. decreasing the schema) until the
limit is reached. However, once schema -4 is reached, the scrape will still
succeed, even if a higher growth factor has been specified.
Both settings accept zero as a valid value, which implies “no limit”. In case
of the bucket limit, this means that the number of buckets are indeed not
checked at all. In the case of the bucket factor, Prometheus will still ensure
that a standard schema will not exceed the capabilities of the used storage
backend. Prometheus currently stores histograms with standard exponential
schemas of at most 8. However, it accepts exponential schemas greater than 8 up
to the reserved limit of 52 but reduces their resolution upon
ingestion so that schema 8 is reached (or a lower one if required by the
native_histogram_bucket_limit or 
native_histogram_min_bucket_factor
settings).
If both settings have a non-zero values, the schema is decreased sufficiently
to satisfy both limits.
Note that the bucket factor set during
instrumentation is an upper limit (exposed bucket
growth factor ≤ configured value), while the bucket factor set in the
scrape config is a lower limit (ingested bucket growth factor ≥ configured
value). The schemas resulting from certain limits are therefore slightly
different. Some examples:
native_histogram_min_bucket_factor
resulting max schema
65536
-4
256
-3
16
-2
4
-1
2
0
1.4
1
1.1
2
1.09
3
1.04
4
1.02
5
1.01
6
1.005
7
1.002
8
General considerations about setting the limits:
native_histogram_bucket_limit is suitable to set a hard limit for the cost of
an individual histogram. The same cannot be accomplished by
native_histogram_min_bucket_factor because histograms can have many buckets
even with a low resolution if the distribution of observations is sufficiently
broad. 
native_histogram_min_bucket_factor is well suited to avoid needless
overall resource costs. For example, if the use case at hand only requires a
certain resolution, setting a corresponding
native_histogram_min_bucket_factor for all histograms might free up enough
resources to accept a very high bucket count on a few histograms with broad
distributions of observed values. Another example is the case where some
histograms have low resolution for some reason (maybe already on the
instrumentation side). If aggregations regularly include those low resolution
histograms, the outcome will have that same low resolution (see the PromQL
details below). Storing other histograms
regularly aggregated with the low resolution histograms at higher resolution
might not be of much use.
Scraping both classic and native histograms
As described above, a histogram exposed by an
instrumented program might contain both a classic and a native histograms, and
some parts are even shared (like the count and sum of observations). This
section explains which parts will be scraped by Prometheus, and how to control
the behavior.
If 
scrape_native_histograms is 
false (default in v3) in the scrape config,
Prometheus will completely ignore the native histogram parts during scraping. If
scrape_native_histograms is 
true (default in v4+), Prometheus will prefer
the native histogram parts over the classic histogram parts, even if both are
exposed for the same histogram. Prometheus will still scrape the classic
histogram parts for histograms with no native histogram data.
In situations like migration scenarios, it might
be desired to scrape both versions, classic and native, for the same histogram,
provided both versions are exposed by the instrumented program. To enable this
behavior, there is a boolean setting 
always_scrape_classic_histograms in the
scrape config. It defaults to false, but if set to true, both versions of each
histogram will be scraped and ingested, provided there is at least one classic
bucket and at least one native bucket span (which might be a no-op span). This
will not cause any conflicts in the TSDB because classic histograms are
ingested as a number of suffixed series, while native histograms are ingested
as just one series with their unmodified name. (Example: A histogram called
rpc_latency_seconds results in a native histogram series named
rpc_latency_seconds and in a number of series for the classic part, namely
rpc_latency_seconds_sum, 
rpc_latency_seconds_count, and a number of
rpc_latency_seconds_bucket series with different 
le labels.)
Scraping classic histograms as NHCBs
The aforementioned NHCB is capable of modeling a classic histogram as a native
histogram. Via the boolean scrape config option
convert_classic_histograms_to_nhcb, Prometheus can be configured to ingest
classic histograms as NHCBs.
While NHCBs support automatic reconciliation between different bucket
layouts, their mergeability is still
fundamentally limited. The reconciliation only retains exact matches of bucket
boundaries between the involved NHCBs. This yields useful results, if most
bucket boundaries match. However, arbitrary changes in the bucket layout can
easily create a situation where none of the boundaries match, resulting in a
histogram with only one bucket (the overflow bucket).
A key advantage of NHCBs is that they are generally much less expensive to
store. In particular, the incremental cost of adding additional buckets is
relatively low, which allows affordable ingestion of classic histograms with
many buckets.
TSDB
NOTE: This section provides a high level overview of storing native histograms
in the TSDB and also explains some important individual aspects that might be
easy to miss. It is not meant to explain implementation details, define on-disk
formats, or guide through the code base. There is a detailed documentation of
the various storage
formats
and of course the usual generated GoDoc, with the tsdb
package and the
storage package
as suitable starting points. A helpful resource is also the aforementioned
Developer’s Guide to Prometheus Native
Histograms.
Integer histograms vs. float histograms
The TSDB stores integer histograms and float histograms differently. Generally,
integer histograms are expected to compress better, so a TSDB implementation
MAY store a float histogram as an integer histogram if all bucket counts and
the count of observations have an integer value within the int64 range so that
the conversion to an integer histogram creates a numerically precise
representation of the original float histogram. (Note that the Prometheus TSDB
is not utilizing this option yet.)
Encoding
Native histograms require two new chunk encodings (Go type 
chunkenc.Encoding)
in the TSDB: 
chunkenc.EncHistogram (string representation 
histogram,
numerical value 2) for integer histograms, and 
chunkenc.EncFloatHistogram
(string representation 
floathistogram, numerical value 3) for float
histograms.
Similarly, there are two new record types for the WAL and the in-memory
snapshot (Go type 
record.Type): 
record.HistogramSamples (string
representation 
histogram_samples, numerical value 9) for integer histograms,
and 
record.FloatHistogramSamples (string representation
float_histogram_samples, numerical value 10) for float histograms. For
backwards compatibility reasons, there are two more histogram record types:
record.HistogramSamplesLegacy (
histogram_samples_legacy, 7) and
record.FloatHistogramSamplesLegacy (
float_histogram_samples_legacy, 8).
They were used prior to the introduction of custom values needed for NHCB. They
are supported so that reading old WALs is still possible.
Prometheus identifies time series just by their labels. Whether a sample in a
series is a float (and as such a counter or a gauge) or a histogram (no matter
what flavor) does not contribute to the series's identity. Therefore, a series
MAY contain a mix of samples of different types and flavors. Changes of the
sample type within a time series are expected to be very rare in practice. They
usually happen after changes in the instrumentation of a target (in the rare
case that the same metric name is used for e.g. a gauge float prior to the
change and a counter histogram after the change) or after a change of a
recording rule (e.g. where the old version of a rule created a gauge float and
the new version of the rule now creates a gauge histogram while retaining its
name). Frequent changes of the sample type are usually the consequence of a
misconfiguration (e.g. two different recording rules creating different sample
types feeding into the same series). Therefore, a TSDB implementation MUST
handle a change in sample type, but it MAY do so in a relatively inefficient
way. When the Prometheus TSDB encounters a sample type that cannot be written
to the currently used chunk, it closes that chunk and starts a new one with the
appropriate encoding. (A time series that switches sample types back and forth
for each sample will lead to a new chunk for each sample, which is indeed very
inefficient.)
Histogram chunks use a number of custom encodings for numerical values, in
order to reduce the data size by encoding common values in fewer bits than less
common values. The details of each custom encoding are described in the low
level chunk format
documentation
(and ultimately in the code linked from there). The following three encodings
are used for a number of different fields and are therefore named here for
later reference:
varbit-int is a variable bitwidth encoding for signed integers. It uses
between 1 bit and 9 bytes. Numbers closer to zero need fewer bits. This is
similar to the timestamp encoding in chunks for float samples, but with a
different bucketing of the various bit lengths, optimized for the value
distribution commonly encountered in native histograms.
varbit-uint is a similar encoding, but for unsigned integers.
varbit-xor is a variable bitwidth encoding for a sequence of floats. It is
based on XOR'ing the current and the previous float value in the sequence. It
uses between 1 bit and 77 bits per float. This is exactly the same encoding
the TSDB already uses for float samples.
Histogram chunks start as usual with the number of samples in the chunk (as a
uint16), followed by one byte describing if the histogram is a gauge histogram
or a counter histogram and providing counter reset information for the latter.
See the corresponding section below for
details. This is followed by the so called chunk layout, which contains the
following information, shared by all histograms in the chunk:
The threshold of the zero bucket, using a custom encoding that encodes common
values (zero or certain powers of two) in just one byte, but requires 9 bytes
for arbitrary values.
The schema, encoded as varbit-int.
The positive spans, encoded as the number of spans (varbit-uint), followed by
the length (varbit-uint) and the offset (varbit-int) of each span in a
repeated sequence.
The negative spans in the same way.
Only for schema -53 (NHCB) the custom values, encoded as the number of custom
values (varbit-uint), followed by the custom values in a repeated sequence,
using a custom encoding.
The chunk layout is followed by a repeated sequence of sample data. The sample
data is different for integer histograms and float histograms. For an integer
histogram, the data of each sample contains the following:
The timestamp, encoded as varbit-int, with an absolute value in the 1st
sample, a delta between the 1st and 2nd sample for the 2nd sample, and a
“delta of deltas” for any further samples (i.e. the same “double delta”
encoding used for timestamps in conventional float chunks, just with a
different bit bucketing for the varbit-int encoding).
The count of observations, encoded as varbit-uint for the 1st sample and as
varbit-int for any further samples, using the same “delta of deltas” approach
as for timestamps.
The zero bucket population, encoded as varbit-uint for the 1st sample and as
varbit-int for any further samples, using the same “delta of deltas” approach
as for timestamps.
The sum of observations, encoded as a float64 for the 1st sample and as
varbit-xor for any further samples (XOR'ing between the current and previous
sample).
The bucket populations of the positive buckets, each as a delta to the
previous bucket (or as the absolute population in the 1st bucket), encoded as
varbit-int, using the same “delta of deltas” approach as for timestamps. (In
other words, the “double delta” encoding is applied to values that are
already deltas on their own, which is the reason why this is sometimes called
“triple delta“ encoding.)
The bucket populations of the negative buckets in the same way.
The sample data of a float histogram has the following differences:
The count of observations and the zero bucket populations are floats now and
therefore encoded in the same way as the sum of observations (float64 in the
1st sample, varbit-xor for any further samples).
The bucket population are not only floats now, but also absolute population
counts rather than deltas between buckets. In the 1st sample, all bucket
populations are represented as plain float64's, while they are encoded as
varbit-xor for all further samples, XOR'ing corresponding buckets from the
current and the previous sample.
The following events trigger cutting a new chunk (for the reasons described in
parentheses):
A change of sample type between integer histogram and float histogram
(because both require different chunk encodings altogether).
A change of sample type between gauge histogram and counter histogram
(because the leading byte has to denote the different type).
A counter reset for a counter histogram (to be stored in the leading byte as
counter reset information, see details below).
A schema change (which means we need a new chunk layout, and a chunk can only
have one chunk layout).
A change of the zero threshold (which changes the chunk layout, see above).
A change of the custom values (which changes the chunk layout, see above).
A staleness marker is followed by a regular sample (which does not strictly
require a new chunk, but it can be assumed that most histograms will change
so much when they go away and come back that cutting a new chunk is the best
option).
The chunk size limit is exceeded (see details below).
Differences in the spans would also change the chunk layout, but they are
reconciled by adding (explicitly represented) unpopulated buckets as needed so
that all histograms in a chunk share the same span structure. This is
straightforward if a bucket disappears, because the missing bucket is simply
added to the new histogram as an unpopulated bucket while the histogram is
appended to the chunk. However, disappearance of a formerly populated bucket
constitutes a counter reset (see below), so
this case can only happen for gauge histograms (which do not feature counter
resets). The far more common case is that buckets exist in a newly appended
histogram that did not exist in the previously appended histograms. In this
case, these buckets have to be added as explicitly unpopulated buckets to all
previously appended histograms. This requires a complete re-encoding of the
entire chunk. (There is some optimization potential in only re-encoding the
affected parts. Implementing this would be quite complicated. So far, the
performance impact of the full re-encoding did not stick out as problematic.)
Staleness markers
NOTE: To understand the following section, it is important to recall how
staleness markers work in the TSDB. Staleness markers in float series are
represented by one specific bit pattern among the many that can be used to
represent the 
NaN value. This very specific float value is called “special
stale 
NaN value” in the following section. It is (almost certainly) never
returned by the usual arithmetic float operations and as such different from a
“naturally occurring” 
NaN value, including those discussed in Special cases
of observed values. In fact, the special
stale 
NaN value is never returned directly when querying the TSDB, but it is
handled internally before it reaches the caller.
To mark staleness in histogram series, the usual special stale 
NaN value
could be used. However, this would require cutting a new chunk, just for the
purpose of marking the series as stale, because a float value following a
histogram value has to be stored in a different chunk (see above). Therefore,
there is also a histogram version of a stale marker where the field for the sum
of observations is set to the special stale 
NaN value. In this case, all
other fields are ignored, which enables setting them to values suitable for
efficient storage (as the histogram version of a stale marker is essentially
just a storage optimization). This works for both float and integer histograms
(as the sum field is a float value even in an integer histogram), and the
appropriate version can be used to avoid cutting a new chunk. All version of a
stale marker (float, integer histogram, float histogram) MUST be treated as
equivalent by the TSDB.
Chunk size limit
The size of float chunks is limited to 1024 bytes. The same size limitation is
generally used for histogram chunks, too. However, individual histograms can
become very large if they have many buckets, so blindly enforcing the size
limit could lead to chunks with very few histograms. (In the most extreme case,
a single histogram could even take more than 1024 bytes so that the size limit
could not be enforced at all.) With very few histograms per chunk, the
compression ratio becomes worse. Therefore, a minimum number of 10 histograms
per chunks has to be reached before the size limit of 1024 bytes kicks in. This
implies that histogram chunks can be much larger than 1024 bytes.
Requiring a minimum of 10 histograms per chunk is an initial, very simplistic
approach, which might be improved in the future to find a better trade-off
between chunk size and compression ratio.
Counter reset considerations
Generally, Prometheus considers a counter to have reset whenever its value
drops from one sample to the next (but see also the next section about the
start timestamp). The situation is more complex
when detecting a counter reset between two histogram samples.
First of all, gauge histograms and counter histograms are explicitly different
(whereas Prometheus generally treats all float samples equally after ingestion,
no matter if they were ingested as a gauge or a counter metric). Counter resets
do not apply to gauge histograms.
If a gauge histogram is followed by a counter histogram in a time series, a
counter reset is assumed to have happened, because a change from gauge to
counter is considered equivalent to the gauge being deleted and the counter
being newly created from zero.
The most common case is a counter histogram being followed by another counter
histogram. In this case, a possible counter reset is detected by the following
procedure:
If the two histograms are both using a standard schema, but differ in schema or
in the zero bucket width, these changes could be part of a compatible
resolution reduction (which happens regularly to reduce the bucket count of a
histogram). Both of the following is true for a
compatible resolution reduction:
If the schema has changed, its number has decreased from one standard
exponential schema to another standard schema.
If the zero bucket width has changed, any populated regular bucket in the first
histogram is either completely included in the zero bucket of the second
histogram or not at all (i.e. no partial overlap of old regular buckets with
the new zero bucket).
If any of the conditions are not met, the change is not a compatible resolution
reduction. Because such a change is only possible by resetting or newly
creating a histogram, it is considered a counter reset and the detection
procedure is concluded.
If both conditions are met, the first histogram has to be converted so that its
schema and zero bucket width matches those of the second histogram. This
happens in the same way as previously described:
Neighboring buckets are merged to reduce the schema, and regular buckets are
merged with the zero bucket to increase the width of the zero bucket.
If both histograms are NHCBs (schema -53), any difference in their custom
values is reconciled as described below.
At this point in the procedure, both histograms have the same schema and zero
bucket width, either because this was the case from the beginning, or because
one of the histograms was converted accordingly. (Note that NHCBs do not use
the zero bucket. Their zero bucket widths and population counts are considered
equal for the sake of this procedure.) In this situation, any of the following
constitutes a counter reset:
A drop in the count of observations (but notably not a drop in the sum of
observations).
A drop in the population count of any bucket, including the zero bucket. This
includes the case where a populated bucket disappears, because a
non-represented bucket is equivalent to a bucket with a population of zero.
If none of the above is the case, there is no counter reset.
As this whole procedure is relatively involved, the counter reset detection
preferably happens once during ingestion, with the result being persisted for
later use. Counter reset detection during ingestion has to happen anyway
because a counter reset is one of the triggers to cut a new chunk.
Cutting a new chunk after a counter reset aims to improve the compression
ratio. A counter reset sets all bucket populations to zero, so there are fewer
buckets to represent. A chunk, however, has to represent the superset of all
buckets of all histograms in the chunk, so cutting a new chunk enables a
simpler set of buckets for the new chunk.
This in turn implies that there will never be a counter reset after the first
sample in a chunk. Therefore, the only counter reset information that has to be
persisted is that of the 1st histogram in a chunk. This happens in the
so-called histogram flags, a single byte stored directly after the number
of samples in the chunk. This byte is currently only used for the counter reset
information, but it may be used for other flags in the future. The counter
reset information uses the first two bits. The four possible bit patterns are
represented as Go constants of type 
CounterResetHeader in the 
chunkenc
package. Their names and meanings are the following:
GaugeType (bit pattern 
11): The chunk contains gauge histograms.
Counter resets are irrelevant for gauge histograms.
CounterReset (bit pattern 
10): A counter reset happened between the last
histogram of the previous chunk and the 1st histogram of this chunk. (It is
likely that the counter reset was actually the reason why the new chunk was
cut.)
NotCounterReset (bit pattern 
01): No counter reset happened between the
last histogram of the previous chunk and the 1st histogram of this chunk.
(This commonly happens if a new chunk is cut because the previous chunk hit
the size limit.)
UnknownCounterReset (bit pattern 
00): It is unknown if there was a
counter reset between the last histogram of the previous chunk and the 1st
histogram of this chunk.
UnknownCounterReset is always a safe choice. It does not prevent counter
reset detection, but merely requires that the counter reset detection procedure
has to be performed (again) whenever counter reset information is needed.
The counter reset information is propagated to the caller when querying the
TSDB (in the Go code as a field of type 
CounterResetHint in the Go types
Histogram and 
FloatHistogram, using enumerated constants with the same
names as the bit pattern constants above).
For gauge histogram, the 
CounterResetHint is always 
GaugeType. Any other
CounterResetHint value implies that the histogram in question is a counter
histogram. In this way, queriers (including the PromQL engine, see
below) obtain the information if a histogram is a gauge or a counter
(which is notably different from float samples).
As long as counter histograms are returned in order from a single chunk, the
CounterResetHint for the 2nd and following histograms in a chunk is set to
NotCounterReset. (Overlapping blocks and out-of-order ingestion may lead to
histogram sequences coming from multiple chunks, which requires special
treatment, see below.)
When returning the 1st histogram from a counter histogram chunk, the
CounterResetHint MUST be set to 
UnknownCounterReset unless the TSDB
implementation can ensure that the previously returned histogram was indeed the
same histogram that was used as the preceding histogram to detect the counter
reset at ingestion time. Only in the latter case, the counter reset information
from the chunk MAY be used directly as the 
CounterResetHint of the returned
histogram.
This precaution is needed because there are various ways how chunks might get
removed or inserted (e.g. deletion via tombstones or adding blocks for
backfilling). A counter reset, while attributed to one sample, is in fact
happening between the marked sample and the preceding sample. Removing the
preceding sample or inserting another sample in between the two samples
invalidates the previously performed counter reset detection.
TODO: Currently, the Prometheus TSDB has no means of ensuring that the
preceding chunk is still the same chunk as during ingestion. Therefore,
Prometheus currently returns 
UnknownCounterReset for all 1st histograms
from a counter histogram chunk. See tracking
issue for efforts to
change that.
As already implied above, the querier MUST perform the counter reset detection
procedure (again), if the 
CounterResetHint is set to 
UnknownCounterReset.
Special caution has to be applied when processing overlapping blocks or
out-of-order samples (for querying or during compaction). Both overdetection
and underdetection of counter resets may happen in these cases, as illustrated
by the following examples:
Example for underdetection: One chunk contains samples ABC, without counter
resets. Another chunk contains samples DEF, again without counter resets. The
chunks are overlapping and refer to the same series. When querying them
together, the temporal order of samples turns out to be ADBECF. There might
now very well be a counter reset between some or even all of those samples.
This is in fact likely if the two samples are actually from unrelated series
and got merged into the same series by accident. However, even accidental
merges like this have to be handled correctly by the TSDB. If the overlapping
chunks are compacted into a new chunk, a new counter reset detection has to
happen, catching the new counter resets. If querying the overlapping chunks
directly (without prior compaction), a 
CounterResetHint of
UnknownCounterReset has to be set for each sample that comes from a
different chunk than the previously returned sample, which mandates a counter
reset detection by the querier (utilizing the safe fallback described above).
Example for overdetection: There is a sequence of samples ABCD with a
counter reset happening between B and C. However, the initial ingestion
missed B and C so that only A and D were ingested, with a counter reset
detected between A and D. Later, B and C are ingested (via out-of-order
ingestion or as separate chunks later added to the TSDB as a separate block),
with a counter reset detected between B and C. In this case, each sample goes
into its own chunk, so when assembling all the chunks, they do not even
overlap. However, when returning the counter reset hints according to the
rules above, both C and D will be returned to the querier with a
CounterResetHint of 
CounterReset, although there is now no counter reset
between C and D. Similar to the situation in the previous example, a new
counter reset detection has to be performed between A and B, and another one
between C and D. Or both B and D have to be returned with a
CounterResetHint of 
UnknownCounterReset.
In summary, whenever the TSDB cannot safely establish that a counter reset
detection between two samples has happened upon ingestion, it either has to
perform another counter reset detection or it has to return a
CounterResetHint of 
UnknownCounterReset for the second sample.
Note that there is the possibility of counter resets that are not detected by
the procedure described above, namely if the counts in the reset histogram have
increased quickly enough so that the 1st sample after the counter reset has no
counts that have decreased compared to the last sample prior to the counter
reset. (This is also a problem for float counters, where it is actually more
likely to happen.) With the mechanisms explained above, it is possible to store
a counter reset even in this case, provided that the counter reset was detected
by other means. However, due to the complications caused by insertion and
removal of chunks, out-of-order samples, and overlapping blocks (as explained
above), this information MAY get lost if a second round of counter reset
detection is required. (TODO: Currently, this information is reliably lost, see
TODO above.) A better way to safely mark a counter reset is via start
timestamps (see next section).
Start timestamp handling
OpenMetrics introduced the so-called created timestamps for counters, summaries,
and classic counter histograms. (The term is probably short for “created-at
timestamp”. The more appropriate term might have been “creation timestamp” or
“reset timestamp”.) The term has since been renamed to "start timestamp".
The start timestamp provides the most recent time the metric was created or
reset. A design
doc
describes how Prometheus handles start timestamps.
Start timestamps are also useful for native histograms. In the same way a
synthetic zero sample is inserted for float counters, a zero value of a
histogram sample is inserted for counter histograms. A zero value of a
histogram has no populated buckets, and the sum of observations, the count of
observations, and the zero bucket population are all zero. Schema, zero bucket
width, custom values, and the float vs. integer flavor of the histogram SHOULD
match the sample that directly follows the synthetic zero sample (to not
trigger the detection of a spurious counter reset).
The counter reset information of the synthetic zero sample is always set to
CounterReset.
Exemplars
Exemplars for native histograms are attached to the histogram sample as a
whole, not to individual buckets. (See also the exposition formats
section.) Therefore, it is allowed (and in fact the
common case) that a single native histogram sample comes with multiple
exemplars attached.
Exemplars may or may not change from one scrape to the next. Scrapers SHOULD
detect unchanged exemplars to avoid storing many duplicate exemplars. Duplicate
detection is potentially expensive, though, given that a single sample might
have many exemplars, of which any subset could be repeated exemplars from the
last scrape. The TSDB MAY rely on the assumption that any new exemplar has a
more recent timestamp than any of the previously exposed exemplars. (Remember
that exemplars of native histograms MUST have a timestamp.) Duplicate detection
is then possible in an efficient way:
The exemplars of a newly ingested native histogram are sorted by the
following fields: first timestamp, then value, then labels.
The exemplars are appended to the exemplar storage in the sorted order.
The append fails for exemplars that would be sorted before or are equal to
the last successfully appended exemplar (which might be from the previous
scrape for the same metric).
The append succeeds for exemplars that would be sorted after the last
successfully appended exemplar.
Exemplars are only counted as out of order if all exemplars of an ingested
histogram would be sorted before the last successfully appended exemplar. This
does not detect out-of-order exemplars that are mixed with newer exemplars or
with a duplicate of the last successfully appended exemplar, which is
considered acceptable.
PromQL
This section describes how PromQL handles native histograms. It focuses on
general concepts rather than every single detail of individual operations. For
the latter, refer to the PromQL documentation about
operators
and
functions.
Annotations
The introduction of native histograms creates certain situations where a PromQL
expression returns unexpected results, most commonly the case where some or all
elements in the output vector are unexpectedly missing. To help users detect
and understand those situations, operations acting on native histograms often use
annotations. Annotations can have warn and info level and describe possible
issues encountered during the evaluation. Warn level is used to mark situations
that are most likely an actual problem the user has to act on. Info level is
used for situations that might also be deliberate, but are still unusual enough
to flag them.
Integer histograms vs. float histograms
PromQL always acts on float histograms. Native histograms that are stored as
integer histograms are automatically converted to float histograms when
retrieved from the TSDB.
Compatibility between histograms
When an operator or function acts on two or more native histograms, the
histograms involved need to have the same schema, the same zero bucket width,
and (if applicable) the same custom values. Within certain limits, histograms
can be converted on the fly to meet these compatibility criteria:
NHCBs (schema -53) are only compatible with each other. Different custom
values need to be reconciled by conversion in the following way:
Identify the custom values that are shared by all of the original NHCBs.
These are the new reconciled custom values.
Convert each original NHCB to the new custom values by merging its buckets
into the unified bucket set described by the new custom values.
Note that it is easily possible that the original NHCBs do not share any
custom values. In this case, the new bucket set will only consist of the
overflow bucket, taking all observations from all of the original buckets.
Any query requiring reconciliation of custom values is flagged with an
info-level annotation.
Histograms with standard schemas can always be converted to the
smallest (i.e. lowest resolution) common schema by decreasing the resolution
of the histograms with greater schemas (i.e. higher resolution). This happens
in the usual way by merging neighboring buckets into the larger buckets of
the smaller schema.
Different zero bucket widths are handled by expanding the smaller zero
buckets, merging any populated regular bucket into the expanded zero bucket
as appropriate. If the greatest common width happens to end up in the middle
of any populated bucket, it is further expanded to coincide with the bucket
boundary of that bucket. (See more details in the zero bucket section
above.)
If incompatibility prevents an operation, a warn-level annotation is added to
the result.
Counter resets
Counter resets are defined as described above.
Counter reset hints returned from the TSDB MAY be taken into account to avoid
explicit counter reset detection and to correctly process counter resets that
are not detectable by the usual procedure. (This implies that these counter
resets are only taken into account on a best effort basis. However, the same is
true for the TSDB itself, see above.) A notable difference to the counter reset
handling for classic histograms and summaries is that a decrease of the sum of
observations does not constitute a counter reset by itself. (For example,
calculating the rate of a native histogram will still work correctly even if
the histogram has observed negative values.)
Note that the counter reset hints of counter histograms returned by sub-queries
MUST NOT be taken into account to avoid explicit counter reset detection,
unless the PromQL engine can safely detect that consecutive counter histograms
returned from the sub-query are also consecutive in the TSDB.
Gauge histograms vs. counter histograms
Via the counter reset hint returned from the TSDB, PromQL is aware if a native
histogram is a gauge or a counter histogram. To mirror PromQL's treatment of
float samples (where it cannot reliably distinguish between float counters and
gauges), functions that act on counters will still process gauge histograms,
and vice versa, but a warn-level annotation is returned with the result. Note
that explicit counter reset detection has to be performed on a gauge histogram
in that case, treating it as if it were a counter histogram.
Interpolation within a bucket
When estimating quantiles or fractions of observations in a histogram, or when
removing observations from a histogram, PromQL has to apply interpolation
within a bucket. In classic histograms, this interpolation happens in a linear
fashion. It is based on the assumption that observations are equally
distributed within the bucket. In reality, this assumption might be far off.
(For example, an API endpoint might respond to almost all request with a
latency of 110ms. The median latency and maybe even the 90th percentile latency
would then be close to 110ms. If a classic histogram has bucket boundaries at
100ms and 200ms, it would see most observations in that range and estimate the
median at 150ms and the 90th percentile at 190ms.) The worst case is an
estimation at one end of a bucket where the actual value is at the other end of
the bucket. Therefore, the maximum possible error is the whole width of a
bucket. Not doing any interpolation and using some fixed midpoint within a
bucket (for example the arithmetic mean or even the harmonic mean) would
minimize the maximum possible error (which would then be half of the bucket
width in case of the arithmetic mean), but in practice, the linear
interpolation yields an error that is lower on average. Since the interpolation
has worked well over many years of classic histogram usage, interpolation is
also applied for native histograms.
For NHCBs, PromQL applies the same linear interpolation method as for classic
histograms to keep results consistent. (The main use case for NHCBs is a
drop-in replacement for classic histograms.) This includes the following
special cases:
If all custom values are positive, the lower boundary of
the lowest bucket is assumed to be zero. (Note that it is recommended to
include a custom value of zero to avoid this heuristics.)
If at least one custom value is zero or negative, the lower boundary of the
lowest bucket is assumed to be -Inf. However, for purposes of interpolation,
all observations in the lowest buckets are assumed to be equal to the upper
boundary of the bucket.
Similarly, all observation in the overflow bucket are assumed to be equal to
the last custom value (i.e. the upper bound of the highest regular bucket).
For an NHCB without any custom values, all observations are assumed to be of
value zero.
For standard exponential schemas, linear interpolation can be seen as a misfit.
While exponential schemas primarily intend to minimize the relative error of
quantile estimations, they also benefit from a balanced usage of buckets, at
least over certain ranges of observed values. The basic assumption is that for
most practically occurring distributions, the density of observations tends to
be higher for smaller observed values. Therefore, PromQL uses exponential
extrapolation for the standard schemas, which models the assumption that
dividing a bucket into two when increasing the schema number by one (i.e.
doubling the resolution) will on average see similar populations in both new
buckets. A more detailed explanation can be found in the PR implementing the
interpolation method.
A special case is interpolation within the zero bucket. The zero bucket breaks
the exponential bucketing schema. Therefore, linear interpolation is applied
within the zero bucket. Furthermore, if all populated regular buckets of a
histogram are positive, it is assumed that all observations in the zero bucket
are also positive, i.e. the interpolation is done between zero and the upper
bound of the zero bucket. In the case of a histogram where all populated
regular buckets are negative, the situation is mirrored, i.e. the interpolation
within the zero bucket is done between the lower bound of the zero bucket and
zero.
Mixed series
As already discussed above, neither the sample type nor the flavor of a native
histogram is part of the identity of a series. Therefore, one and the same
series might contain a mix of different sample types and flavors.
A mix of counter histograms and gauge histograms doesn't prevent any PromQL
operation, but a warn-level annotation is returned with the result if some of
the input samples have an inappropriate flavor (see
above).
A mix of float samples and histogram samples is more problematic. Many
functions that operate on range vectors will remove elements from the result
where the input elements contain a mix of floats and histograms. If this
happens, a warn-level annotation is added to the result. Concrete examples can
be found below.
Unary minus and negative histograms
The unary minus can be used on native histograms. It returns a histogram where
all bucket populations and the count and the sum of observations have their
sign inverted. The counter reset hint is set to 
GaugeType in any case.
Everything else stays the same. Enforcing 
GaugeType is needed because
explicit counter reset detection will be thrown off by the inverted sign.
Generally, histograms with negative bucket populations or a negative count of
observations do not really make sense on their own and are only supposed to act
as intermediate results inside other expressions. They are always considered
gauge histograms within PromQL. They cannot be persisted as a result of a
recording rule. (A rule evaluating to a negative histogram results in an
error.) It is impossible to represent negative histograms in any of the
exchange formats (exposition formats, remote-write, OTLP).
Binary operators
Most binary operators do not work between two histograms or between a histogram
and a float or between a histogram and a scalar. If an operator processes such
an impossible combination, the corresponding element is removed from the output
vector and an info-level annotation is added to the result. (This situation is
somewhat similar to label matching, where the sample type plays a role similar
to a label. Therefore, such a mismatch might be known and deliberate, which is
the reason why the level of the annotation is only info.)
The following describes all the operations that actually do work.
Addition and subtraction
Addition (
+) and subtraction (
-) work between two compatible histograms.
These operators add or subtract all matching bucket populations and the count
and the sum of observations. Missing buckets are assumed to be empty and
treated accordingly.
Generally, both operands should be gauges. Adding and subtracting counter
histograms requires caution, but PromQL allows it. Adding a gauge histogram and
a counter histogram results in a gauge histogram. Adding two counter histograms
results in a counter histogram. If the two operands share the same counter
reset hint, the resulting counter histogram retains that counter reset hint.
Otherwise, the resulting counter reset hint is set to 
UnknownCounterReset.
The result of a subtraction is always marked as a gauge histogram because it
might result in negative histograms, see notes
above. Adding or subtracting two counter
histograms with directly contradicting counter reset hints (i.e. 
CounterReset
and 
NotCounterReset) triggers a warn-level annotation. (TODO: As described
above, the TSDB currently does not return
NotCounterReset, so this annotation will only happen under specific
circumstances involving the 
HistogramStatsIterator, which includes additional
counter reset tracking. See tracking
issue.)
Multiplication
Multiplication (
*) works between a float sample or a scalar on the one side
and a histogram on the other side, in any order. It multiplies all bucket
populations and the count and the sum of observations by the float (sample or
scalar). This will lead to “scaled” and sometimes even negative histograms,
which is usually only useful as intermediate results inside other expressions
(see also notes above).
Multiplication works for both counter histograms and gauge histograms, and
their flavor is left unchanged by the operation, with the exception that
multiplying by a negative value always results in a gauge histogram.
Division
Division (
/) works between a histogram on the left hand side and a float
sample or a scalar on the right hand side. It is equivalent to multiplication
with the inverse of the float (sample or scalar). Division by zero results in a
histogram with no regular buckets and the zero bucket population and the count
and sum of observations all set to 
+Inf, 
-Inf, or 
NaN, depending on their
values in the input histogram (positive, negative, or zero/
NaN,
respectively).
Equality and inequality
Equality (
==) and inequality (
!=) work between two histograms, both in
their filtering version as well as with the 
bool modifier. They compare the
schema, the custom values, the zero threshold, all bucket populations, and the
sum and count of observations. Whether the histograms have counter or gauge
flavor is irrelevant for the comparison. (A counter histogram could be equal to
a gauge histogram.)
Logical and set operators
The logical/set binary operators (
and, 
or, 
unless) work as expected even
if histogram samples are involved. They only check for the existence of a
vector element and don't change their behavior depending on the sample type or
flavor of an element (float or histogram, counter or gauge).
Trim operators
The “trim upper” (
</) and “trim lower” (
>/) operators were introduced
specifically for native histograms. They only work for a histogram on the left
hand side and a float sample or a scalar on the right hand side. (An info-level
annotation is returned in all other cases.)
These operators remove observations from the left hand side histogram that are
greater or smaller than the float value on the right hand side, respectively,
and return the resulting histogram.
The removal is only precise if the threshold coincides with a bucket boundary.
Otherwise, interpolation within the affected buckets has to be used, as
described above with the following exceptions
for NHCBs:
If the lowest bucket has a lower boundary of -Inf, all observation in that
bucket are considered to be of value -Inf (rather than the upper boundary).
If the highest bucket has an upper boundary of +Inf, all observation in that
bucket are considered to be of value +Inf (rather than the lower boundary).
However, in the pathologic edge case where a histogram has only a single
bucket with a lower limit of -Inf and an upper limit of +Inf, all
observations are still considered to be of value zero.
If any observations have been removed, the sum of all observations in the
resulting histogram is estimated from the remaining buckets. The value of each
observation in a given bucket is estimated to be the following:
The upper boundary of the lowest bucket of an NHCB if that upper boundary is
negative or zero. (This follows again the original interpolation
heuristics, despite using a different one
for trimming, as explained above).
The lower boundary for the overflow bucket of an NHCB. (Again switching back to
the original interpolation heuristics.)
-Inf for the negative overflow bucket of a standard exponential histogram.
+Inf for the positive overflow bucket of a standard exponential histogram.
The arithmetic mean for all other buckets of an NHCB and for the zero bucket
of a standard exponential histogram, taking into account the known heuristics
for the following special cases:
The lowest bucket of an NHCB is considered to have a lower boundary of zero if
its upper boundary is positive.
The lower boundary of the zero bucket of a standard exponential histogram is
considered to be zero if the histograms has no populated negative buckets.
The upper boundary of the zero bucket of a standard exponential histogram is
considered to be zero if the histograms has no populated positive buckets.
The geometric mean for all other buckets of a standard exponential histogram.
Additionally, the boundaries used for the (arithmetic or geometric) mean
calculation for the bucket that was only partially removed (using the
interpolation described above) is modified in the following way: The relevant
upper boundary (for 
</) or lower boundary (for 
>/) is considered to be
equal to the cutoff threshold provided as the 2nd operand.
Note that this estimation of the sum of observations is inaccurate (even in the
case where the cutoff threshold conicides with a bucket boundary), up to a
point where it could yield results that are obviously wrong. For example, after
removing some positive observation, the estimated sum of observations could be
larger than the sum of observations in the original histogram. The estimation
algorithm could be refined to take such cases into account, but it is kept
deliberately simple to make it easier to reason with. In general, the
histograms resulting from trim operations are meant to be used for quantile
estimation or filtering, where the sum of observations is irrelevant anyway.
The trim operators are also a viable alternative to the 
histogram_fraction
function (explained below) in cases where the desired outcome is not a fraction
of observations but a count of observations. For example, using
histogram_count(h >/ 0.2 </ 0.5) instead of 
histogram_fraction(0,2, 0.5, h) * histogram_count(h) is simpler and also yields 
0 instead of 
NaN in case 
h
has no observations. (The 
histogram_fraction calculated from an empty
histogram is always 
NaN.)
The trim operators preserve the counter vs. gauge flavor of the histogram.
Aggregation operators
The following aggregation operators work in the same way with float and
histogram samples (for the reason stated in parentheses):
group (The result of this aggregation does not depend on the sample values.)
count (The result of this aggregation does not depend on the sample values.)
count_values (The text representation as produced by the Go
FloatHistogram.String method is used as the value of histograms.)
limitk (The sampled elements are returned unchanged.)
limit_ratio (The sampled elements are returned unchanged.)
The 
sum aggregation operator works with native histograms by summing up the
histogram to be aggregated in the same way as described for the 
+ operator
above. The 
avg aggregation operator works in the same way, but divides the
sum by the number of aggregated histograms (in the same way as described for
the 
/ operator above).
Both 
sum and 
avg remove elements from the output vector that would require
the aggregation of float samples with histogram samples. Such a removal is
flagged by a warn-level annotation.
Both 
sum and 
avg should only be applied to gauge histograms. PromQL allows
to aggregate counter histograms (and even a mix of both), but it requires
caution to do so in a meaningful way. The implications for the gauge vs.
counter flavor and the resulting counter reset hint are derived from those for
the 
+ operator above:
If all aggregated histograms share the same counter reset hint, the result
retains that same counter reset hint.
If there is at least one gauge histogram among the aggregated histograms, the
result is a gauge histogram.
In all other cases, the counter reset hint of the result is set to
UnknownCounterReset.
In any case, any directly contradicting counter reset hints (i.e.
CounterReset and 
NotCounterReset) among the aggregated histograms trigger
a warn-level annotation.
All other aggregation operators do not work with native histograms.
Histograms in the input vector are simply ignored, and an info-level annotation
is added for each ignored histogram.
Functions
The following functions operate on range vectors of native histograms by
applying the usual float operation individually to matching buckets (including
the zero bucket) and the sum and count of observations, resulting in a new
native histogram:
delta() (For gauge histograms.)
increase() (For counter histograms.)
rate() (For counter histograms.)
idelta() (For gauge histograms.)
irate() (For counter histograms.)
These functions SHOULD be applied to either gauge histograms or counter
histograms as noted above. However, they all work with both flavors, but if at
least one histogram of an unsuitable flavor is contained in the range vector, a
warn-level annotation is added to the result.
delta(), 
increase(), and 
rate() return no result for series that contain
a mix of float samples and histogram samples within the range. 
idelta() and
irate() return no result for series where the last two samples within the
range are a mix of a float sample and a histogram sample. In either case, a
warn-level annotation is added for each output element missing for these
reasons.
All these functions return gauge histograms as results.
As usual, these functions attempt to reconcile different schemas by converting
the schema to a common one as far as possible. However, the functions applied
to counters (
increase(), 
rate(), 
irate()) do not perform this conversion
for the 1st sample if there is a counter reset between the 1st and 2nd sample.
In this case, the 1st sample is not included in the calculation, so an
incompatible bucket layout between the 1st sample and the other samples is
simply ignored silently.
To prevent extrapolation below zero, the same heuristics is applied as for
float counters,
but solely based on the count of observations. Therefore, individual buckets
might still be extrapolated below zero in some cases. An alternative could have
been to find the smallest extrapolation where neither the count nor any bucket
would be extrapolated below zero. However, this does not necessarily lead to a
better heuristics while inflicting a significant cost in complexity. In the
common and important case where the first sample in the range is a synthetic
zero sample derived from the created-at timestamp, the limited extrapolation
will actually work perfectly precise, because the count and all buckets are
zero at precisely the timestamp of the synthetic sample, which is also the
point in time to which the extrapolation is limited. Note that classic
histogram apply the heuristics independently to each bucket and the count and
the sum (as they are all separate series). This is known to lead to
inconsistencies. NHCBs do not reproduce this problem and work in the same way
as other native histograms, which means that the result of 
rate() and
increase() may be slightly different when comparing classic histograms and
equivalent NHCBs.
avg_over_time() and 
sum_over_time() work with native histograms in a way
that corresponds to the respective aggregation operators. In particular, if a
series contains a mix of float samples and histogram samples within the range,
the corresponding result is removed entirely from the output vector. Such a
removal is flagged by a warn-level annotation.
The 
changes() and the 
resets() function work with native histogram samples
in the same way as with float samples. They even work with a mix of float
samples and histogram samples within the same series. In this case, a change
from a float sample to a histogram sample and vice versa counts as a change for
changes() and as a reset for 
resets(). A change in flavor from counter
histogram to gauge histogram and vice versa does not count as a change for
changes(). 
resets() SHOULD only be applied to counter floats and counter
histograms, but the function still works with gauge histograms, applying
explicit counter reset detection in this case. Furthermore, a change from
counter histogram to gauge histogram and vice versa is counted as a reset.
The 
histogram_quantile() function has a very special role as it is the only
function that treats a specific “magic” label specially, namely the 
le label
used by classic histograms. 
histogram_quantile() also works for native
histograms in a similar way, but without the special role of the 
le label.
The function keeps treating float samples in the known way, while it uses the
new “native” way for native histogram samples.
An example for a typical query for classic histograms (including 
rate and
aggregation):
histogram_quantile(0.9, sum by (job, le) (rate(http_request_duration_seconds_bucket[10m])))
This is the corresponding query for a native histograms:
histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[10m])))
As with classic histograms, an estimation of the maximum and minimum
observation in a histogram can be performed using 1 and 0, respectively, as the
first parameter of 
histogram_quantile. However, native histograms with
standard schemas enable much more useful results, not only because of the
usually higher resolution of native histograms, but even more so because native
histograms with standard schemas sustain the same resolution across the whole
range of float64 numbers. With a classic histogram, the odds are that the
maximum observation is in the +Inf bucket, so that the estimation simply
returns the upper limit of the last bucket before the +Inf bucket. Similarly,
the minimum observation will often be in the lowest bucket.
histogram_quantile treats observations of value 
NaN (which SHOULD NOT
happen, see above) effectively as
observations of 
+Inf. This follows the rationale that 
NaN is never less
than any value that 
histogram_quantile returns. As long as the result
falls into an existing bucket we return the result calculated as if
NaN observations were observed as 
+Inf and also issue an info level
annotation to let the user know that results are skewed due to 
NaN. This is
consistent with how classic histograms usually treat 
NaN observations (which
end up in the 
+Inf bucket in most implementations). If the result falls above
all existing buckets, we return 
NaN. For a detailed explanation why, see
histogram_fraction below; intuitively this case means that we don't have a
number that is greater than all observations in the quantile as 
NaN is not
comparable to any number. We also return an info level annotation specific to
this case.
The following functions have been introduced specifically for native
histograms:
histogram_avg()
histogram_count()
histogram_fraction()
histogram_sum()
histogram_stddev()
histogram_stdvar()
All these functions silently ignore float samples as input. Each function
returns a vector of float samples.
histogram_count() and 
histogram_sum() return the count of observations or
the sum of observations, respectively, that are contained in a native histogram.
As they are normal functions, their result cannot be used in a range selector.
Instead of using sub-queries, the recommended way to calculate a rate of the
count or the sum of observations is to first rate the histogram and then apply
histogram_count() or 
histogram_sum() to the result. For example, the
following query calculates the rate of observations (in this case corresponding
to “requests per second”) from a native histogram:
histogram_count(rate(http_request_duration_seconds[10m]))
Note that the special counter reset detection for native histograms doesn't
apply when using a sub-query on the result of 
histogram_sum(), i.e. negative
observations may result in spurious counter resets.
histogram_avg() returns the arithmetic average of the observed values
in a native histogram. (This is notably different from applying the 
avg
aggregation operator to a number of native histograms. The latter returns an
averaged histogram.)
Similarly, 
histogram_stddev() and 
histogram_stdvar() return the estimated
standard deviation or standard variance, respectively, of the observations in a
native histogram. For this estimation, all observations in a bucket are assumed
to have the value of the mean of the bucket boundaries. For the zero bucket and
for buckets with custom boundaries, the arithmetic mean is used. For standard
exponential buckets, the geometric mean is used.
histogram_fraction(lower, upper, histogram) returns the estimated fraction of
observations in 
histogram between the provided boundaries, the scalar values
lower and 
upper. The error of the estimation depends on the resolution of
the underlying native histogram and how closely the provided boundaries are
aligned with the bucket boundaries in the histogram. 
+Inf and 
-Inf are
valid boundary values and useful to estimate the fraction of all observations
above or below a certain value. However, observations of value 
NaN are always
considered to be outside of the specified boundaries (even 
+Inf and 
-Inf).
Whether the provided boundaries are inclusive or exclusive is only relevant if
the provided boundaries are precisely aligned with bucket boundaries in the
underlying native histogram. In this case, the behavior depends on the precise
definition of the schema of the histogram.
The value of 
q = histogram_fraction(-Inf, x, histogram) means that the
fraction of observations less or equal to 
x is 
q. On the other hand
y = histogram_quantile(q, histogram) means that 
q fraction of observations
are less or equal to 
y. Since 
histogram_quantile calculates the approximate
smallest value for 
y, it follows that 
y<=x in general. Consider the case
when 90% of the observations are 
NaN. Then the maximum value of
histogram_fraction is 
0.1 since 
histogram_fraction considers 
NaN
observations outside any bucket. If for example
histogram_quantile(0.5, histogram) returned any real number 
y, then
according to the argument above, we should find some number 
x for which 
y<=x
and 
histogram_fraction(-Inf, x, histogram) is equal to 
0.5, however this
doesn't happen for any 
y, which is the reason we return 
NaN if the result of
histogram_quantile would be outside all buckets.
The following functions do not interact directly with sample values and
therefore work with native histogram samples in the same way as they work with
float samples:
absent()
absent_over_time()
count_over_time()
info()
label_join()
label_replace()
last_over_time()
present_over_time()
sort_by_label()
sort_by_label_desc()
timestamp()
All remaining functions not mentioned in this section do not work with native
histograms. Histogram elements in the input vector are silently ignored. For
deriv(), 
double_exponential_smoothing(), 
predict_linear(), and all the
<aggregation>_over_time() functions not mentioned before, native histogram
samples are removed from the input range vector. In case any series contains a
mix of float samples and histogram samples within the range, the removal of
histograms is flagged by an info-level annotation.
Recording rules
Recording rules MAY result in native histogram values. They are stored back
into the TSDB as during normal ingestion, including whether the histogram is a
gauge histogram or a counter histogram. In the latter case, a counter reset
explicitly marked by the counter reset hint is also stored, while a new counter
reset detection is initiated during ingestion otherwise.
TSDB implementations MAY convert the float histograms created by recording
rules to integer histograms if this conversion precisely represents all the
float values in the original histogram.
Alerting rules
Alerts work as usual with native histograms. However, it is RECOMMENDED to
avoid native histograms as output values for alerts. If native histogram
samples are used in templates, they are rendered in their simple text
form (as produced by the Go 
FloatHistogram.String
method), which is hard to read for humans.
Testing framework
The PromQL testing framework has been extended so that both PromQL unit tests
as well as rules unit tests via 
promtool can include native histograms. The
histogram sample notation is complex and explained in the documentation for
rules unit
testing.
In the unit test framework there is an alternative 
load command called
load_with_nhcb, which converts classic histograms to NHCBs and loads both the
float series of the classic histogram as well as the NHCB series resulting from
the conversion.
Not specific to native histograms, but very useful in their context, is the
expect keyword in the unit test framework that can define expectations about
the info- and warn-level annotations.
Optimizations
As usual, PromQL implementations MAY apply any optimizations they see fit as
long as the behavior stays the same. Decoding native histograms can be quite
expensive with the potentially many buckets. Similarly, deep-copying a
histogram sample within the PromQL engine is much more expensive than copying a
simple float sample. This creates a huge potential for optimization compared to
a naive approach of always decoding everything and always copying everything.
Prometheus currently tries to avoid needless copies (TODO: but a proper CoW
like approach still has to be implemented, as it would be much cleaner and less
bug prone) and skips decoding of the buckets for special cases where only the
sum and count of observations is required.
Prometheus query API
The query API
documentation
includes native histogram support. This section focuses on the parts relevant
for native histograms and provides a bit of context not part of the API
documentation.
Instant and range queries
To return native histograms in the JSON response of instant (
query endpoint)
and range (
query_range endpoint) queries, both the 
vector and 
matrix
result type needs an extension by a new key.
The 
vector result type gets a new key 
histogram at the same level as the
existing 
value key. Both these keys are mutually exclusive, i.e. each element
in a 
vector has either a 
value key (for a float result) or a 
histogram
key (for a histogram result). The value of the 
histogram key is structured
similarly to the value of the 
value key (a two-element array), with the
difference that the string representing the float sample value is replaced by a
specific histogram object described below.
The 
matrix result type gets a new key 
histograms at the same level as the
existing 
values key. These keys are not mutually exclusive. A series may
contain both float values and histogram values, but for a given timestamp,
there must be only one sample, either a float or a histogram. The value of the
histograms key is structured similarly to the value of the 
values key (an
array of n two-element arrays), with the difference that the strings
representing float sample values are replaced by specific histogram objects
described below.
Note that a better naming of the keys would be 
float/
histogram and
floats/
histograms because both float values and histogram values are
values. The current naming has historical reasons. (In the past, there was only
one value type, namely floats, so calling the keys simply 
value and 
values
was the obvious choice.) The intention here is to not break existing consumers
that do not know about native histograms.
The histogram object mentioned above has the following structure:
{
  "count": "<count_of_observations>",
  "sum": "<sum_of_observations>",
  "buckets": [ [ <boundary_rule>, "<left_boundary>", "<right_boundary>", "<count_in_bucket>" ], ... ]
}
count and 
sum directly correspond to the histogram fields of the same name.
Each bucket is represented explicitly with its boundaries and count, including
the zero bucket. Spans and the schema are therefore not part of the response,
and the structure of the histogram object does not depend on the used schema.
The 
<boundary_rule> placeholder is an integer between 0 and 3 with the
following meaning:
0: “open left” (left boundary is exclusive, right boundary in inclusive)
1: “open right” (left boundary is inclusive, right boundary in exclusive)
2: “open both” (both boundaries are exclusive)
3: “closed both” (both boundaries are inclusive)
For standard schemas, positive buckets are “open left”, negative buckets are
“open right”, and the zero bucket (with a negative left boundary and a positive
right boundary) is “closed both”. For NHCBs, all buckets are “open left”
(mirroring the behavior of classic histograms). Future schemas might utilize
different boundary rules.
Metadata
For the 
series endpoint, series containing native histograms are included in
the same way as conventional series containing only floats. The endpoint does
not provide any information what sample types are included (and in fact, any
series may contain either or both sample types). Note in particular that a
histogram exposed by a target under the name 
request_duration_seconds will
lead to a series called 
request_duration_seconds if it is exposed and
ingested as a native histogram, but if it is exposed and ingested as a classic
histogram, it will lead to a set of series called
request_duration_seconds_sum, 
request_duration_seconds_count, and
request_duration_seconds_bucket. If the histogram is ingested as both a
native histogram and a classic
histogram, all of the series
names above will be returned by the 
series endpoint.
The target and metric metadata (endpoints 
targets/metadata and 
metadata)
work a bit differently, as they are acting on the original name as exposed by
the target. This means that a classic histogram called
request_duration_seconds will be represented by these metadata endpoints
only as 
request_duration_seconds (and not 
request_duration_seconds_sum,
request_duration_seconds_count, or 
request_duration_seconds_bucket). A
native histogram 
request_duration_seconds will also be represented under this
name. Even in the case where 
request_duration_seconds is ingested as both a
classic and a native histogram, there will be no collision as the metadata
returned is actually the same (most notably the returned 
type will be
histogram). In other words, there is currently no way of distinguishing
native from classic histograms via the metadata endpoints alone. An additional
look-up via the 
series endpoint is required. There are no plans to change
this, as the existing metadata endpoints are anyway severely limited (no
historical information, no metadata for metrics created by rules, limited
ability to handle conflicting metadata between different targets). There are
plans, though, to improve metadata handling in Prometheus in general. Those
efforts will also take into account how to support native histograms properly.
(TODO: Update as progress is made.)
Prometheus UI
This section describes the rendering of histograms by Prometheus's own UI. This
MAY be used as a guideline for third party graphing frontends.
In the Table view, a histogram data point is rendered graphically as a bar
graph together with a textual representation of all the buckets with their
lower and upper limit and the count and sum of observations. Each bar in the
bar graph represents a bucket. The position of each bar on the x axis is
determined by the lower and upper limit of the corresponding bucket. The area
of each bar is proportional to the population of the corresponding bucket
(which is a core principle of rendering histograms in general).
The graphical histogram allows a choice between an exponential and a linear x
axis. The former is the default. It is a good fit for the standard schemas.
(TODO: Consider linear as a default for non-exponential schemas.) Conveniently,
all regular buckets of an exponential schema have the same width on an
exponential x axis. This means that the y axis can display actual bucket
populations without violating the above principle that the area (not the
height) of a bar is representative for the bucket population. The zero bucket
is an exception to that. Technically, it has an infinite width. Prometheus
simply renders it with the same width as the regular exponential buckets (which
in turn means that the x axis is not strictly exponential around the zero
point). (TODO: How to do the rendering for non-exponential schemas.)
With a linear x axis, the buckets generally have varying width. Therefore,
the y axis displays the bucket population divided by its width. The
Prometheus UI does not render values on the y axis as they would be hard to
interpret for humans anyway. The population can still be inspected in the text
representation.
In the Graph view, Prometheus displays a heatmap (TODO: not yet, see below),
which could be seen as a series of histograms over time, rotated by 90 degrees
and encoding the bucket population as a color rather than the height of a bar.
The typical query to render a counter-like histogram as a heatmap would be a
rate query. A heatmap is an extremely powerful representation that allows
humans to easily spot characteristics of distributions as they change over
time.
TODO: Heatmaps are not implemented yet. Instead, the UI plots just the sum of
observations as a conventional graph. See tracking
issue. The same issue
also discusses how to deal with the rendering of range vectors in the Table
view.
Template expansion
Native histograms work in template expansion. They are rendered in a text
representation inspired by the mathematical notation of open and closed
intervals. (This is generated by the 
FloatHistogram.String method in Go.) As
native histograms can have a lot of buckets and bucket boundaries tend to have
boundaries with a lot of decimal places, the representation isn't necessarily
very readable. Use native histograms in template expansion judiciously.
Example for the text representation of a float histogram:
{count:3493.3, sum:2.349209324e+06, [-22.62741699796952,-16):1000, [-16,-11.31370849898476):123400, [-4,-2.82842712474619):3, [-2.82842712474619,-2):3.1, [-0.01,0.01]:5.5, (0.35355339059327373,0.5]:1, (1,1.414213562373095]:3.3, (1.414213562373095,2]:4.2, (2,2.82842712474619]:0.1}
Remote-write & remote-read
The protobuf specs for remote-write &
remote-read were
extended for native histograms. Receivers not capable of processing native
histograms will simply ignore the newly added fields. Nevertheless, Prometheus
has to be configured to send native histograms via remote-write (by setting the
send_native_histograms remote-write config setting to true).
In remote-write v2,
native histograms are a stable feature.
It might appear tempting to convert classic histograms to NHCBs while sending
or receiving them. However, this does not overcome the known consistency
problems classic histograms suffer from when transmitted via remote-write.
Instead, classic histograms SHOULD be converted to NHCBs during scraping.
Similarly, explicit OTel histograms SHOULD be converted to NHCBs during OTLP
ingestion already.
TODO: A remaining possible problem with remote-write is what to do if multiple
exemplars originally ingested for the same native histogram are sent in
different remote-write requests.
Federation
Federation of native histograms works as expected, provided the federation
scrape uses the protobuf format. A federation via OpenMetrics text format will
be possible, at least in principle, once native histograms are supported in
that format, but federation via protobuf is preferred for efficiency reasons
anyway.
TODO: Update once OM supports NH.
NHCBs are rendered as classic float histograms when exposed via the federation
endpoint. Scrapers have the option of converting them back to NHCBs or ingest
them as classic histograms. The latter could lead to naming collisions, though.
Note that OpenMetrics v1 does not support classic float histograms.
Fortunately, Prometheus federation does not use OpenMetrics v1 anyway, but
either the protobuf format or the classic text format.
OTLP
The OTLP receiver built into Prometheus converts incoming OTel exponential
histograms to Prometheus native histograms utilizing the compatibility
described above. The resolution of a
histogram using a schema (“scale” in OTel lingo) greater than 8 will be reduced
to match schema 8. (In the unlikely case that a schema smaller than -4 is used,
the ingestion will fail.)
Explicit OTel histograms are the equivalent of Prometheus's classic histograms.
Prometheus therefore converts them to classic histograms by default, but
optionally offers direct conversion to NHCBs.
Pushgateway
Native histogram support has been gradually added to the
Pushgateway. Full support was
reached in v1.9. The Pushgateway always has been based on the classic protobuf
format as its internal data model, which made the necessary changes easy
(mostly UI concerns). Combined histograms (with classic and native buckets) can
be pushed and will be exposed as such via the 
/metrics endpoint. (However,
the query API, which can be used to query the pushed metrics as JSON, will only
be able to return one kind of buckets and will prefer native buckets if
present.)
promtool
This section describes 
promtool commands added or changed to support native
histograms. Commands not mentioned explicitly do not directly interact with
native histograms and require no changes.
The 
promtool query ... commands work with native histograms. See the query
API documentation to learn about the output format.
A new command 
promtool query analyze was specifically added to analyze
classic and native histogram usage patterns returned by the query API.
The rules unit testing via 
promtool test rules works with native histograms,
using the format described above.
promtool tsdb analyze and 
promtool tsdb list work normally with native
histograms. The 
--extended output of the former has specific sections for
histogram chunks.
promtool tsdb dump uses the usual text representation of native histograms
(as produced by the Go method 
FloatHistogram.String).
promtool tsdb create-blocks-from rules works with rules that emit native
histograms.
The 
promtool promql ... commands support all the PromQL features added for
native histograms.
While 
promtool tsdb bench write could in principle include native histograms,
such a support is not planned at the moment.
The following commands depend on the OpenMetrics text format and
therefore cannot support native histograms as long as there is no native
histogram support in OpenMetrics:
promtool check metrics
promtool push metrics
promtool tsdb dump-openmetrics
promtool tsdb create-blocks-from openmetrics
TODO: Update as progress is made. See tracking
issue.
prom2json
prom2json is a small tool that
scrapes a Prometheus 
/metrics endpoint, converts the metrics to a bespoke
JSON format, which it dumps to stdout. This is convenient for further
processing with tools handling JSON, for example 
jq.
prom2json v1.4 added support for native histograms. If a histogram in the
exposition contains at least one bucket span, 
prom2json will replace the
usual classic bucket in the JSON output with the buckets of the native
histogram, following a format inspired by the Prometheus query
API.
Migration considerations
When migrating from classic to native histograms, there are three important
sources of issues to consider:
Querying native histograms works differently from querying classic
histograms. In most cases, the changes are minimal and straightforward, but
there are tricky edge cases, which make it hard to perform a reliable
auto-conversion.
Classic and native histograms cannot be aggregated with each other. A change
from classic to native histograms at a certain point in time makes it hard
to create dashboards that work across the transition point, and range
vectors that contain the transition point will inevitably be incomplete
(i.e. a range vector selecting classic histograms will only contain data
points in the earlier part of the range, and a range vector selecting native
histograms will only contain data points in the later part of the range).
A classic histogram might be tailored to have bucket boundaries precisely at
the points of interest. Native histograms with a standard schema can have a
high resolution, but do not allow to set bucket boundaries at arbitrary
values. In those cases, the user experience with native histograms might
actually be worse.
To address (3), it is of course possible to not migrate the classic histogram
in question and leave things as they are. Another option is to leave the
instrumentation the same but convert classic histograms to NHCBs upon
ingestion. This leverages the increased storage performance of native
histograms, but still requires to address (1) and (2) in the same way as for a
full migration to native histograms (see next paragraphs).
The conservative way of addressing (1) and (2) is to allow a long transition
period, which comes at the cost of collecting and storing classic and native
histograms in parallel for a while.
The first step is to update the instrumentation to expose classic and native
histograms in parallel. (This step can be skipped if the plan is to stick with
classic histogram in the instrumentation and simply convert them to NHCBs
during scraping.)
Then configure Prometheus to scrape both classic and native histograms, see
section about scraping both classic and native
histograms above. (If needed,
also activate the conversion of classic histograms to
NHCB.)
The existing queries involving classic histograms will continue to work, but
from now on, users can start working with native histograms and start to change
queries in dashboards, alerts, recording rules,… As already mentioned above, it
is important to pay attention to queries with longer range vectors like
histogram_quantile(0.9, rate(rpc_duration_seconds[1d])). This query
calculates the 90th percentile latency over the last day. Hoewever, if native
histograms haven't been collected for at least one day, the query will only
cover that shorter period. Thus, the query should only be used once native
histograms have been collected for at least 1d. For a dashboard that displays
the daily 90th percentile latency over the last month, it is tempting to craft
a query that correctly switches from classic to native histograms at the right
moment. While that is in principle possible, it is tricky. If feasible, the
transition period during which classic and native histograms are collected in
parallel, can be quite long to minimize the necessity to implement tricky
switch-overs. For example, once classic and native histograms have been
collected in parallel for a month, any dashboard not looking farther back than
a month can simply be switched over from a classic histogram query to a native
histogram query without any consideration about the right switch-over.
Once there is confidence that all queries have been migrated correctly,
configure Prometheus to only scrape native histograms (which is the “normal”
setting). (It is also possible to incrementally remove classic histograms with
relabel rules in the scrape config.) If everything still works, it is time to
remove classic histograms from the instrumentation.
The Grafana Mimir documentation contains a detailed migration
guide
following the same philosophy as described in this section.

OpenMetrics 1.0
Version: 1.0
Status: Published
Date: November 2020
Authors: Richard Hartmann, Ben Kochie, Brian Brazil, Rob Skillington
Created in 2012, Prometheus has been the default for cloud-native observability since 2015. A central part of Prometheus' design is its text metric exposition format, called the Prometheus exposition format 0.0.4, stable since 2014. In this format, special care has been taken to make it easy to generate, to ingest, and to understand by humans. As of 2020, there are more than 700 publicly listed exporters, an unknown number of unlisted exporters, and thousands of native library integrations using this format. Dozens of ingestors from various projects and companies support consuming it.
With OpenMetrics, we are cleaning up and tightening the specification with the express purpose of bringing it into IETF. We are documenting a working standard with wide and organic adoption while introducing minimal, largely backwards-compatible, and well-considered changes. As of 2020, dozens of exporters, integrations, and ingestors use and preferentially negotiate OpenMetrics already.
Given the wide adoption and significant coordination requirements in the ecosystem, sweeping changes to either the Prometheus exposition format 0.0.4 or OpenMetrics 1.0 are considered out of scope.
NOTE: OpenMetrics 2.0 development is in progress. Read here on how to join the Prometheus OM 2.0 work group.
Overview
Metrics are a specific kind of telemetry data. They represent a snapshot of the current state for a set of data. They are distinct from logs or events, which focus on records or information about individual events.
OpenMetrics is primarily a wire format, independent of any particular transport for that format. The format is expected to be consumed on a regular basis and to be meaningful over successive expositions.
Implementers MUST expose metrics in the OpenMetrics text format in response to a simple HTTP GET request to a documented URL for a given process or device. This endpoint SHOULD be called "/metrics". Implementers MAY also expose OpenMetrics formatted metrics in other ways, such as by regularly pushing metric sets to an operator-configured endpoint over HTTP.
Metrics and Time Series
This standard expresses all system states as numerical values; counts, current values, enumerations, and boolean states being common examples. Contrary to metrics, singular events occur at a specific time. Metrics tend to aggregate data temporally. While this can lose information, the reduction in overhead is an engineering trade-off commonly chosen in many modern monitoring systems.
Time series are a record of changing information over time. While time series can support arbitrary strings or binary data, only numeric data is in scope for this RFC.
Common examples of metric time series would be network interface counters, device temperatures, BGP connection states, and alert states.
Data Model
This section MUST be read together with the ABNF section. In case of disagreements between the two, the  ABNF's restrictions MUST take precedence. This reduces repetition as the text wire format MUST be supported.
Data Types
Values
Metric values in OpenMetrics MUST be either floating points or integers. Note that ingestors of the format MAY only support float64. The non-real values NaN, +Inf and -Inf MUST be supported. NaN MUST NOT be considered a missing value, but it MAY be used to signal a division by zero.
Booleans
Boolean values MUST follow 
1==true, 
0==false.
Timestamps
Timestamps MUST be Unix Epoch in seconds. Negative timestamps MAY be used.
Strings
Strings MUST only consist of valid UTF-8 characters and MAY be zero length. NULL (ASCII 0x0) MUST be supported.
Label
Labels are key-value pairs consisting of strings.
Label names beginning with underscores are RESERVED and MUST NOT be used unless specified by this standard. Label names MUST follow the restrictions in the ABNF section.
Empty label values SHOULD be treated as if the label was not present.
LabelSet
A LabelSet MUST consist of Labels and MAY be empty. Label names MUST be unique within a LabelSet.
MetricPoint
Each MetricPoint consists of a set of values, depending on the MetricFamily type.
Exemplars
Exemplars are references to data outside of the MetricSet. A common use case are IDs of program traces.
Exemplars MUST consist of a LabelSet and a value, and MAY have a timestamp. They MAY each be different from the MetricPoints' LabelSet and timestamp.
The combined length of the label names and values of an Exemplar's LabelSet MUST NOT exceed 128 UTF-8 character code points. Other characters in the text rendering of an exemplar such as 
",= are not included in this limit for implementation simplicity and for consistency between the text and proto formats.
Ingestors MAY discard exemplars.
Metric
Metrics are defined by a unique LabelSet within a MetricFamily. Metrics MUST contain a list of one or more MetricPoints. Metrics with the same name for a given MetricFamily SHOULD have the same set of label names in their LabelSet.
MetricPoints SHOULD NOT have explicit timestamps.
If more than one MetricPoint is exposed for a Metric, then its MetricPoints MUST have monotonically increasing timestamps.
MetricFamily
A MetricFamily MAY have zero or more Metrics. A MetricFamily MUST have a name, HELP, TYPE, and UNIT metadata. Every Metric within a MetricFamily MUST have a unique LabelSet.
Name
MetricFamily names are a string and MUST be unique within a MetricSet. Names SHOULD be in snake_case. Metric names MUST follow the restrictions in the ABNF section.
Colons in MetricFamily names are RESERVED to signal that the MetricFamily is the result of a calculation or aggregation of a general purpose monitoring system.
MetricFamily names beginning with underscores are RESERVED and MUST NOT be used unless specified by this standard.
Suffixes
The name of a MetricFamily MUST NOT result in a potential clash for sample metric names as per the ABNF with another MetricFamily in the Text Format within a MetricSet. An example would be a gauge called "foo_created" as a counter called "foo" could create a "foo_created" in the text format.
Exposers SHOULD avoid names that could be confused with the suffixes that text format sample metric names use.
Suffixes for the respective types are:
Counter: 
_total, 
_created
Summary: 
_count, 
_sum, 
_created, `` (empty)
Histogram: 
_count, 
_sum, 
_bucket, 
_created
GaugeHistogram: 
_gcount, 
_gsum, 
_bucket
Info: 
_info
Gauge: `` (empty)
StateSet: `` (empty)
Unknown: `` (empty)
Type
Type specifies the MetricFamily type. Valid values are "unknown", "gauge", "counter", "stateset", "info", "histogram", "gaugehistogram", and "summary".
Unit
Unit specifies MetricFamily units. If non-empty, it MUST be a suffix of the MetricFamily name separated by an underscore. Be aware that further generation rules might make it an infix in the text format.
Help
Help is a string and SHOULD be non-empty. It is used to give a brief description of the MetricFamily for human consumption and SHOULD be short enough to be used as a tooltip.
MetricSet
A MetricSet is the top level object exposed by OpenMetrics. It MUST consist of MetricFamilies and MAY be empty.
Each MetricFamily name MUST be unique. The same label name and value SHOULD NOT appear on every Metric within a MetricSet.
There is no specific ordering of MetricFamilies required within a MetricSet. An exposer MAY make an exposition easier to read for humans, for example sort alphabetically if the performance tradeoff makes sense.
If present, an Info MetricFamily called "target" per the "Supporting target metadata in both push-based and pull-based systems" section below SHOULD be first.
Metric Types
Gauge
Gauges are current measurements, such as bytes of memory currently used or the number of items in a queue. For gauges the absolute value is what is of interest to a user.
A MetricPoint in a Metric with the type gauge MUST have a single value.
Gauges MAY increase, decrease, or stay constant over time. Even if they only ever go in one direction, they might still be gauges and not counters. The size of a log file would usually only increase, a resource might decrease, and the limit of a queue size may be constant.
A gauge MAY be used to encode an enum where the enum has many states and changes over time, it is the most efficient but least user friendly.
Counter
Counters measure discrete events. Common examples are the number of HTTP requests received, CPU seconds spent, or bytes sent. For counters how quickly they are increasing over time is what is of interest to a user.
A MetricPoint in a Metric with the type Counter MUST have one value called Total. A Total is a non-NaN and MUST be monotonically non-decreasing over time, starting from 0.
A MetricPoint in a Metric with the type Counter SHOULD have a Timestamp value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before.
A MetricPoint in a Metric's Counter's Total MAY reset to 0. If present, the corresponding Created time MUST also be set to the timestamp of the reset.
A MetricPoint in a Metric's Counter's Total MAY have an exemplar.
StateSet
StateSets represent a series of related boolean values, also called a bitset. If ENUMs need to be encoded this MAY be done via StateSet.
A point of a StateSet metric MAY contain multiple states and MUST contain one boolean per State. States have a name which are Strings.
A StateSet Metric's LabelSet MUST NOT have a label name which is the same as the name of its MetricFamily.
If encoded as a StateSet, ENUMs MUST have exactly one Boolean which is true within a MetricPoint.
This is suitable where the enum value changes over time, and the number of States isn't much more than a handful.
MetricFamilies of type StateSets MUST have an empty Unit string.
Info
Info metrics are used to expose textual information which SHOULD NOT change during process lifetime. Common examples are an application's version, revision control commit, and the version of a compiler.
A MetricPoint of an Info Metric contains a LabelSet. An Info MetricPoint's LabelSet MUST NOT have a label name which is the same as the name of a label of the LabelSet of its Metric.
Info MAY be used to encode ENUMs whose values do not change over time, such as the type of a network interface.
MetricFamilies of type Info MUST have an empty Unit string.
Histogram
Histograms measure distributions of discrete events. Common examples are the latency of HTTP requests, function runtimes, or I/O request sizes.
A Histogram MetricPoint MUST contain at least one bucket, and SHOULD contain Sum, and Created values. Every bucket MUST have a threshold and a value.
Histogram MetricPoints MUST have one bucket with an +Inf threshold. Buckets MUST be cumulative. As an example for a metric representing request latency in seconds its values for buckets with thresholds 1, 2, 3, and +Inf MUST follow value_1 <= value_2 <= value_3 <= value_+Inf. If ten requests took 1 second each, the values of the 1, 2, 3, and +Inf buckets MUST equal 10.
The +Inf bucket counts all requests. If present, the Sum value MUST equal the Sum of all the measured event values. Bucket thresholds within a MetricPoint MUST be unique.
Semantically, Sum, and buckets values are counters so MUST NOT be NaN or negative.
Negative threshold buckets MAY be used, but then the Histogram MetricPoint MUST NOT contain a sum value as it would no longer be a counter semantically. Bucket thresholds MUST NOT equal NaN. Count and bucket values MUST be integers.
A Histogram MetricPoint SHOULD have a Timestamp value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before.
A Histogram's Metric's LabelSet MUST NOT have a "le" label name.
Bucket values MAY have exemplars. Buckets are cumulative to allow monitoring systems to drop any non-+Inf bucket for performance/anti-denial-of-service reasons in a way that loses granularity but is still a valid Histogram.
Each bucket covers the values less and or equal to it, and the value of the exemplar MUST be within this range. Exemplars SHOULD be put into the bucket with the highest value. A bucket MUST NOT have more than one exemplar.
GaugeHistogram
GaugeHistograms measure current distributions. Common examples are how long items have been waiting in a queue, or size of the requests in a queue.
A GaugeHistogram MetricPoint MUST have one bucket with an +Inf threshold, and SHOULD contain a Gsum value. Every bucket MUST have a threshold and a value.
The buckets for a GaugeHistogram follow all the same rules as for a Histogram.
The bucket and Gsum of a GaugeHistogram are conceptually gauges, however bucket values MUST NOT be negative or NaN. If negative threshold buckets are present, then sum MAY be negative. Gsum MUST NOT be NaN. Bucket values MUST be integers.
A GaugeHistogram's Metric's LabelSet MUST NOT have a "le" label name.
Bucket values can have exemplars.
Each bucket covers the values less and or equal to it, and the value of the exemplar MUST be within this range. Exemplars SHOULD be put into the bucket with the highest value. A bucket MUST NOT have more than one exemplar.
Summary
Summaries also measure distributions of discrete events and MAY be used when Histograms are too expensive and/or an average event size is sufficient.
They MAY also be used for backwards compatibility, because some existing instrumentation libraries expose precomputed quantiles and do not support Histograms. Precomputed quantiles SHOULD NOT be used, because quantiles are not aggregatable and the user often can not deduce what timeframe they cover.
A Summary MetricPoint MAY consist of a Count, Sum, Created, and a set of quantiles.
Semantically, Count and Sum values are counters so MUST NOT be NaN or negative.
Count MUST be an integer.
A MetricPoint in a Metric with the type Summary which contains Count or Sum values SHOULD have a Timestamp value called Created. This can help ingestors discern between new metrics and long-running ones it did not see before. Created MUST NOT relate to the collection period of quantile values.
Quantiles are a map from a quantile to a value. An example is a quantile 0.95 with value 0.2 in a metric called myapp_http_request_duration_seconds which means that the 95th percentile latency is 200ms over an unknown timeframe. If there are no events in the relevant timeframe, the value for a quantile MUST be NaN.
A Quantile's Metric's LabelSet MUST NOT have "quantile" label name.
Quantiles MUST be between 0 and 1 inclusive. Quantile values MUST NOT be negative. Quantile values SHOULD represent the recent values. Commonly this would be over the last 5-10 minutes.
Unknown
Unknown SHOULD NOT be used. Unknown MAY be used when it is impossible to determine the types of individual metrics from 3rd party systems.
A point in a metric with the unknown type MUST have a single value.
Data transmission & wire formats
The text wire format MUST be supported and is the default. The protobuf wire format MAY be supported and MUST ONLY be used after negotiation.
The OpenMetrics formats are Regular Chomsky Grammars, making writing quick and small parsers possible.
The text format compresses well, and protobuf is already binary and efficiently encoded.
Partial or invalid expositions MUST be considered erroneous in their entirety.
Protocol Negotiation
All ingestor implementations MUST be able to ingest data secured with TLS 1.2 or later. All exposers SHOULD be able to emit data secured with TLS 1.2 or later. ingestor implementations SHOULD be able to ingest data from HTTP without TLS. All implementations SHOULD use TLS to transmit data.
Negotiation of what version of the OpenMetrics format to use is out-of-band. For example for pull-based exposition over HTTP standard HTTP content type negotiation is used, and MUST default to the oldest version of the standard (i.e. 1.0.0) if no newer version is requested.
Push-based negotiation is inherently more complex, as the exposer typically initiates the connection. Producers MUST use the oldest version of the standard (i.e. 1.0.0) unless requested otherwise by the ingestor.
Text format
ABNF
ABNF as per RFC 5234
"exposition" is the top level token of the ABNF.
exposition = metricset HASH SP eof [ LF ]
metricset = *metricfamily
metricfamily = *metric-descriptor *metric
metric-descriptor = HASH SP type SP metricname SP metric-type LF
metric-descriptor =/ HASH SP help SP metricname SP escaped-string LF
metric-descriptor =/ HASH SP unit SP metricname SP *metricname-char LF
metric = *sample
metric-type = counter / gauge / histogram / gaugehistogram / stateset
metric-type =/ info / summary / unknown
sample = metricname [labels] SP number [SP timestamp] [exemplar] LF
exemplar = SP HASH SP labels SP number [SP timestamp]
labels = "{" [label *(COMMA label)] "}"
label = label-name EQ DQUOTE escaped-string DQUOTE
number = realnumber
; Case insensitive
number =/ [SIGN] ("inf" / "infinity")
number =/ "nan"
timestamp = realnumber
; Not 100% sure this captures all float corner cases.
; Leading 0s explicitly okay
realnumber = [SIGN] 1*DIGIT
realnumber =/ [SIGN] 1*DIGIT ["." *DIGIT] [ "e" [SIGN] 1*DIGIT ]
realnumber =/ [SIGN] *DIGIT "." 1*DIGIT [ "e" [SIGN] 1*DIGIT ]
; RFC 5234 is case insensitive.
; Uppercase
eof = %d69.79.70
type = %d84.89.80.69
help = %d72.69.76.80
unit = %d85.78.73.84
; Lowercase
counter = %d99.111.117.110.116.101.114
gauge = %d103.97.117.103.101
histogram = %d104.105.115.116.111.103.114.97.109
gaugehistogram = gauge histogram
stateset = %d115.116.97.116.101.115.101.116
info = %d105.110.102.111
summary = %d115.117.109.109.97.114.121
unknown = %d117.110.107.110.111.119.110
BS = "\"
EQ = "="
COMMA = ","
HASH = "#"
SIGN = "-" / "+"
metricname = metricname-initial-char 0*metricname-char
metricname-char = metricname-initial-char / DIGIT
metricname-initial-char = ALPHA / "_" / ":"
label-name = label-name-initial-char *label-name-char
label-name-char = label-name-initial-char / DIGIT
label-name-initial-char = ALPHA / "_"
escaped-string = *escaped-char
escaped-char = normal-char
escaped-char =/ BS ("n" / DQUOTE / BS)
escaped-char =/ BS normal-char
; Any unicode character, except newline, double quote, and backslash
normal-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
Overall Structure
UTF-8 MUST be used. Byte order markers (BOMs) MUST NOT be used. As an important reminder for implementers, byte 0 is valid UTF-8 while, for example, byte 255 is not.
The content type MUST be:
application/openmetrics-text; version=1.0.0; charset=utf-8
Line endings MUST be signalled with line feed (\n) and MUST NOT contain carriage returns (\r). Expositions MUST end with EOF and SHOULD end with 
EOF\n.
An example of a complete exposition:
# TYPE acme_http_router_request_seconds summary
# UNIT acme_http_router_request_seconds seconds
# HELP acme_http_router_request_seconds Latency though all of ACME's HTTP request router.
acme_http_router_request_seconds_sum{path="/api/v1",method="GET"} 9036.32
acme_http_router_request_seconds_count{path="/api/v1",method="GET"} 807283.0
acme_http_router_request_seconds_created{path="/api/v1",method="GET"} 1605281325.0
acme_http_router_request_seconds_sum{path="/api/v2",method="POST"} 479.3
acme_http_router_request_seconds_count{path="/api/v2",method="POST"} 34.0
acme_http_router_request_seconds_created{path="/api/v2",method="POST"} 1605281325.0
# TYPE go_goroutines gauge
# HELP go_goroutines Number of goroutines that currently exist.
go_goroutines 69
# TYPE process_cpu_seconds counter
# UNIT process_cpu_seconds seconds
# HELP process_cpu_seconds Total user and system CPU time spent in seconds.
process_cpu_seconds_total 4.20072246e+06
# EOF
Escaping
Where the ABNF notes escaping, the following escaping MUST be applied
Line feed, 
\n (0x0A) -> literally 
\\n (Bytecode 0x5c 0x6e)
Double quotes -> 
\\" (Bytecode 0x5c 0x22)
Backslash -> 
\\\\ (Bytecode 0x5c 0x5c)
A double backslash SHOULD be used to represent a backslash character.
A single backslash SHOULD NOT be used for undefined escape sequences.
As an example, 
\\\\a is equivalent and preferable to 
\\a.
Numbers
Integer numbers MUST NOT have a decimal point. Examples are 
23, 
0042, and 
1341298465647914.
Floating point numbers MUST be represented either with a decimal point or using scientific notation. Examples are 
8903.123421 and 
1.89e-7. Floating point numbers MUST fit within the range of a 64-bit floating point value as defined by IEEE 754, but MAY require so many bits in the mantissa that results in lost precision. This MAY be used to encode nanosecond resolution timestamps.
Arbitrary integer and floating point rendering of numbers MUST NOT be used for "quantile" and "le" label values as in section "Canonical Numbers". They MAY be used anywhere else numbers are used.
Considerations: Canonical Numbers
Numbers in the "le" label values of histograms and "quantile" label values of summary metrics are special in that they're label values, and label values are intended to be opaque. As end users will likely directly interact with these string values, and as many monitoring systems lack the ability to deal with them as first-class numbers, it would be beneficial if a given number had the exact same text representation.
Consistency is highly desirable, but real world implementations of languages and their runtimes make mandating this impractical. The most important common quantiles are 0.5, 0.95, 0.9, 0.99, 0.999 and bucket values representing values from a millisecond up to 10.0 seconds, because those cover cases like latency SLAs and Apdex for typical web services. Powers of ten are covered to try to ensure that the switch between fixed point and exponential rendering is consistent as this varies across runtimes. The target rendering is equivalent to the default Go rendering of float64 values (i.e. %g), with a .0 appended in case there is no decimal point or exponent to make clear that they are floats.
Exposers MUST produce output for positive infinity as +Inf.
Exposers SHOULD produce output for the values 0.0 up to 10.0 in 0.001 increments in line with the following examples:
0.0 0.001 0.002 0.01 0.1 0.9 0.95 0.99 0.999 1.0 1.7 10.0
Exposers SHOULD produce output for the values 1e-10 up to 1e+10 in powers of ten in line with the following examples:
1e-10 1e-09 1e-05 0.0001 0.1 1.0 100000.0 1e+06 1e+10
Parsers MUST NOT reject inputs which are outside of the canonical values merely because they are not consistent with the canonical values. For example 1.1e-4 must not be rejected, even though it is not the consistent rendering of 0.00011.
Exposers SHOULD follow these patterns for non-canonical numbers, and the intention is by adjusting the rendering algorithm to be consistent for these values that the vast majority of other values will also have consistent rendering. Exposers using only a few particular le/quantile values could also hardcode. In languages such as C where a minimal floating point rendering algorithm such as Grisu3 is not readily available, exposers MAY use a different rendering.
A warning to implementers in C and other languages that share its printf implementation: The standard precision of %f, %e and %g is only six significant digits. 17 significant digits are required for full precision, e.g. 
printf("%.17g", d).
Timestamps
Timestamps SHOULD NOT use exponential float rendering for timestamps if nanosecond precision is needed as rendering of a float64 does not have sufficient precision, e.g. 
1604676851.123456789.
MetricFamily
There MUST NOT be an explicit separator between MetricFamilies. The next MetricFamily MUST be signalled with either metadata or a new sample metric name which cannot be part of the previous MetricFamily.
MetricFamilies MUST NOT be interleaved.
MetricFamily metadata
There are four pieces of metadata: The MetricFamily name, TYPE, UNIT and HELP.  An example of the metadata for a counter Metric called foo is:
# TYPE foo counter
If no TYPE is exposed, the MetricFamily MUST be of type Unknown.
If a unit is specified it MUST be provided in a UNIT metadata line. In addition, an underscore and the unit MUST be the suffix of the MetricFamily name.
A valid example for a foo_seconds metric with a unit of "seconds":
# TYPE foo_seconds counter
# UNIT foo_seconds seconds
An invalid example, where the unit is not a suffix on the name:
# TYPE foo counter
# UNIT foo seconds
It is also valid to have:
# TYPE foo_seconds counter
If the unit is known it SHOULD be provided.
The value of a UNIT or HELP line MAY be empty. This MUST be treated as if no metadata line for the MetricFamily existed.
# TYPE foo_seconds counter
# UNIT foo_seconds seconds
# HELP foo_seconds Some text and \n some \" escaping
There MUST NOT be more than one of each type of metadata line for a MetricFamily. The ordering SHOULD be TYPE, UNIT, HELP.
Aside from this metadata and the EOF line at the end of the message, you MUST NOT expose lines beginning with a #.
Metric
Metrics MUST NOT be interleaved.
See the example in "Text format -> MetricPoint".
Labels
A sample without labels or a timestamp and the value 0 MUST be rendered either like:
bar_seconds_count 0
or like
bar_seconds_count{} 0
Label values MAY be any valid UTF-8 value, so escaping MUST be applied as per the ABNF. A valid example with two labels:
bar_seconds_count{a="x",b="escaping\" example \n "} 0
The rendering of values for a MetricPoint can include additional labels (e.g. the "le" label for a Histogram type), which MUST be rendered in the same way as a Metric's own LabelSet.
MetricPoint
MetricPoints MUST NOT be interleaved.
A correct example where there were multiple MetricPoints and Samples within a MetricFamily would be:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_sum{a="bb"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_sum{a="bb"} 0 456
foo_seconds_count{a="ccc"} 0 123
foo_seconds_sum{a="ccc"} 0 123
foo_seconds_count{a="ccc"} 0 456
foo_seconds_sum{a="ccc"} 0 456
An incorrect example where Metrics are interleaved:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_count{a="ccc"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_count{a="ccc"} 0 456
An incorrect example where MetricPoints are interleaved:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
foo_seconds_count{a="bb"} 0 123
foo_seconds_count{a="bb"} 0 456
foo_seconds_sum{a="bb"} 0 123
foo_seconds_sum{a="bb"} 0 456
Metric types
Gauge
The Sample MetricName for the value of a MetricPoint for a MetricFamily of type Gauge MUST NOT have a suffix.
An example MetricFamily with a Metric with no labels and a MetricPoint with no timestamp:
# TYPE foo gauge
foo 17.0
An example of a MetricFamily with two Metrics with a label and MetricPoints with no timestamp:
# TYPE foo gauge
foo{a="bb"} 17.0
foo{a="ccc"} 17.0
An example of a MetricFamily with no Metrics:
# TYPE foo gauge
An example with a Metric with a  label and a MetricPoint with a timestamp:
# TYPE foo gauge
foo{a="b"} 17.0 1520879607.789
An example with a Metric with no labels and MetricPoint with a timestamp:
# TYPE foo gauge
foo 17.0 1520879607.789
An example with a Metric with no labels and two MetricPoints with timestamps:
# TYPE foo gauge
foo 17.0 123
foo 18.0 456
Counter
The MetricPoint's Total Value Sample MetricName MUST have the suffix 
_total. If present the MetricPoint's Created Value Sample MetricName MUST have the suffix 
_created.
An example with a Metric with no labels, and a MetricPoint with no timestamp and no created:
# TYPE foo counter
foo_total 17.0
An example with a Metric with no labels, and a MetricPoint with a timestamp and no created:
# TYPE foo counter
foo_total 17.0 1520879607.789
An example with a Metric with no labels, and a MetricPoint with no timestamp and a created:
# TYPE foo counter
foo_total 17.0
foo_created 1520430000.123
An example with a Metric with no labels, and a MetricPoint with a timestamp and a created:
# TYPE foo counter
foo_total 17.0 1520879607.789
foo_created 1520430000.123 1520879607.789
Exemplars MAY be attached to the MetricPoint's Total sample.
StateSet
The Sample MetricName for the value of a MetricPoint for a MetricFamily of type StateSet MUST NOT have a suffix.
StateSets MUST have one sample per State in the MetricPoint. Each State's sample MUST have a label with the MetricFamily name as the label name and the State name as the label value. The State sample's value MUST be 1 if the State is true and MUST be 0 if the State is false.
An example with the states "a", "bb", and "ccc" in which only the value bb is enabled and the metric name is foo:
# TYPE foo stateset
foo{foo="a"} 0
foo{foo="bb"} 1
foo{foo="ccc"} 0
An example of an "entity" label on the Metric:
# TYPE foo stateset
foo{entity="controller",foo="a"} 1.0
foo{entity="controller",foo="bb"} 0.0
foo{entity="controller",foo="ccc"} 0.0
foo{entity="replica",foo="a"} 1.0
foo{entity="replica",foo="bb"} 0.0
foo{entity="replica",foo="ccc"} 1.0
Info
The Sample MetricName for the value of a MetricPoint for a MetricFamily of type Info MUST have the suffix 
_info. The Sample value MUST always be 1.
An example of a Metric with no labels, and one MetricPoint value with "name" and "version" labels:
# TYPE foo info
foo_info{name="pretty name",version="8.2.7"} 1
An example of a Metric with label "entity" and one MetricPoint value with “name” and “version” labels:
# TYPE foo info
foo_info{entity="controller",name="pretty name",version="8.2.7"} 1.0
foo_info{entity="replica",name="prettier name",version="8.1.9"} 1.0
Metric labels and MetricPoint value labels MAY be in any order.
Summary
If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix 
_sum. If present, the MetricPoint's Count Value Sample MetricName MUST have the suffix 
_count. If present, the MetricPoint's Created Value Sample MetricName MUST have the suffix 
_created. If present, the MetricPoint's Quantile Values MUST specify the quantile measured using a label with a label name of "quantile" and with a label value of the quantile measured.
An example of a Metric with no labels and a MetricPoint with Sum, Count and Created values:
# TYPE foo summary
foo_count 17.0
foo_sum 324789.3
foo_created 1520430000.123
An example of a Metric with no labels and a MetricPoint with two quantiles:
# TYPE foo summary
foo{quantile="0.95"} 123.7
foo{quantile="0.99"} 150.0
Quantiles MAY be in any order.
Histogram
The MetricPoint's Bucket Values Sample MetricNames MUST have the suffix 
_bucket. If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix 
_sum. If present, the MetricPoint's Created Value Sample MetricName MUST have the suffix 
_created.
If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix "_count".
Buckets MUST be sorted in number increasing order of "le", and the value of the "le" label MUST follow the rules for Canonical Numbers.
An example of a Metric with no labels and a MetricPoint with Sum, Count, and Created values, and with 12 buckets. A wide and atypical but valid variety of “le” values is shown on purpose:
# TYPE foo histogram
foo_bucket{le="0.0"} 0
foo_bucket{le="1e-05"} 0
foo_bucket{le="0.0001"} 5
foo_bucket{le="0.1"} 8
foo_bucket{le="1.0"} 10
foo_bucket{le="10.0"} 11
foo_bucket{le="100000.0"} 11
foo_bucket{le="1e+06"} 15
foo_bucket{le="1e+23"} 16
foo_bucket{le="1.1e+23"} 17
foo_bucket{le="+Inf"} 17
foo_count 17
foo_sum 324789.3
foo_created 1520430000.123
Exemplars
Exemplars without Labels MUST represent an empty LabelSet as {}.
An example of Exemplars showcasing several valid cases:
The "0.01" bucket has no Exemplar. The 0.1 bucket has an Exemplar with no Labels. The 1 bucket has an Exemplar with one Label. The 10 bucket has an Exemplar with a Label and a timestamp. In practice all buckets SHOULD have the same style of Exemplars.
# TYPE foo histogram
foo_bucket{le="0.01"} 0
foo_bucket{le="0.1"} 8 # {} 0.054
foo_bucket{le="1"} 11 # {trace_id="KOO5S4vxi0o"} 0.67
foo_bucket{le="10"} 17 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
foo_bucket{le="+Inf"} 17
foo_count 17
foo_sum 324789.3
foo_created  1520430000.123
GaugeHistogram
The MetricPoint's Bucket Values Sample MetricNames MUST have the suffix 
_bucket. If present, the MetricPoint's Sum Value Sample MetricName MUST have the suffix 
_gsum.
If and only if a Sum Value is present in a MetricPoint, then the MetricPoint's +Inf Bucket value MUST also appear in a Sample with a MetricName with the suffix 
_gcount.
Buckets MUST be sorted in number increasing order of "le", and the value of the "le" label MUST follow the rules for Canonical Numbers.
An example of a Metric with no labels, and one MetricPoint value with no Exemplar with no Exemplars in the buckets:
# TYPE foo gaugehistogram
foo_bucket{le="0.01"} 20.0
foo_bucket{le="0.1"} 25.0
foo_bucket{le="1"} 34.0
foo_bucket{le="10"} 34.0
foo_bucket{le="+Inf"} 42.0
foo_gcount 42.0
foo_gsum 3289.3
Unknown
The sample metric name for the value of the MetricPoint for a MetricFamily of type Unknown MUST NOT have a suffix.
An example with a Metric with no labels and a MetricPoint with no timestamp:
# TYPE foo unknown
foo 42.23
Protobuf format
Overall Structure
Protobuf messages MUST be encoded in binary and MUST have 
application/openmetrics-protobuf; version=1.0.0 as their content type.
All payloads MUST be a single binary encoded MetricSet message, as defined by the OpenMetrics protobuf schema.
Version
The protobuf format MUST follow the proto3 version of the protocol buffer language.
Strings
All string fields MUST be UTF-8 encoded.
Timestamps
Timestamp representations in the OpenMetrics protobuf schema MUST follow the published google.protobuf.Timestamp [timestamp] message. The timestamp message MUST be in Unix epoch seconds as an int64 and a non-negative fraction of a second at nanosecond resolution as an int32 that counts forward from the seconds timestamp component. It MUST be within 0 to 999,999,999 inclusive.
Protobuf schema
Protobuf schema is currently available here.
NOTE: Prometheus and ecosystem does not support OpenMetrics protobuf schema, instead it uses similar 
io.prometheus.client format. Discussions about the future of the protobuf schema in OpenMetrics 2.0 are in progress.
Design Considerations
Scope
OpenMetrics is intended to provide telemetry for online systems. It runs over protocols which do not provide hard or soft real time guarantees, so it can not make any real time guarantees itself. Latency and jitter properties of OpenMetrics are as imprecise as the underlying network, operating systems, CPUs, and the like. It is sufficiently accurate for aggregations to be used as a basis for decision-making, but not to reflect individual events.
Systems of all sizes should be supported, from applications that receive a few requests an hour up to monitoring bandwidth usage on a 400Gb network port. Aggregation and analysis of transmitted telemetry should be possible over arbitrary time periods.
It is intended to transport snapshots of state at the time of data transmission at a regular cadence.
Out of scope
How ingestors discover which exposers exist, and vice-versa, is out of scope for and thus not defined in this standard.
Extensions and Improvements
This first version of OpenMetrics is based upon well established and de facto standard Prometheus text format 0.0.4, deliberately without adding major syntactic or semantic extensions, or optimisations on top of it. For example no attempt has been made to make the text representation of Histogram buckets more compact, relying on compression in the underlying stack to deal with their repetitive nature.
This is a deliberate choice, so that the standard can take advantage of the adoption and momentum of the existing user base. This ensures a relatively easy transition from the Prometheus text format 0.0.4.
It also ensures that there is a basic standard which is easy to implement. This can be built upon in future versions of the standard. The intention is that future versions of the standard will always require support for this 1.0 version, both syntactically and semantically.
We want to allow monitoring systems to get usable information from an OpenMetrics exposition without undue burden. If one were to strip away all metadata and structure and just look at an OpenMetrics exposition as an unordered set of samples that should be usable on its own. As such, there are also no opaque binary types, such as sketches or t-digests which could not be expressed as a mix of gauges and counters as they would require custom parsing and handling.
This principle is applied consistently throughout the standard. For example a MetricFamily's unit is duplicated in the name so that the unit is available for systems that don't understand the unit metadata. The "le" label is a normal label value, rather than getting its own special syntax, so that ingestors don't have to add special histogram handling code to ingest them. As a further example, there are no composite data types. For example, there is no geolocation type for latitude/longitude as this can be done with separate gauge metrics.
Units and Base Units
For consistency across systems and to avoid confusion, units are largely based on SI base units. Base units include seconds, bytes, joules, grams, meters, ratios, volts, amperes, and celsius. Units should be provided where they are applicable.
For example, having all duration metrics in seconds, there is no risk of having to guess whether a given metric is nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days or weeks nor having to deal with mixed units. By choosing unprefixed units, we avoid situations like ones in which kilomilliseconds were the result of emergent behaviour of complex systems.
As values can be floating point, sub-base-unit precision is built into the standard.
Similarly, mixing bits and bytes is confusing, so bytes are chosen as the base. While Kelvin is a better base unit in theory, in practice most existing hardware exposes Celsius. Kilograms are the SI base unit, however the kilo prefix is problematic so grams are chosen as the base unit.
While base units SHOULD be used in all possible cases, Kelvin is a well-established unit which MAY be used instead of Celsius for use cases such as color or black body temperatures where a comparison between a Celsius and Kelvin metric are unlikely.
Ratios are the base unit, not percentages. Where possible, raw data in the form of gauges or counters for the given numerator and denominator should be exposed. This has better mathematical properties for analysis and aggregation in the ingestors.
Decibels are not a base unit as firstly, deci is a SI prefix and secondly, bels are logarithmic. To expose signal/energy/power ratios exposing the ratio directly would be better, or better still the raw power/energy if possible. Floating point exponents are more than sufficient to cover even extreme scientific uses. An electron volt (~1e-19 J) all the way up to the energy emitted by a supernova (~1e44 J) is 63 orders of magnitude, and a 64-bit floating point number can cover over 2000 orders of magnitude.
If non-base units can not be avoided and conversion is not feasible, the actual unit should still be included in the metric name for clarity. For example, joule is the base unit for both energy and power, as watts can be expressed as a counter with a joule unit. In practice a given 3rd party system may only expose watts, so a gauge expressed in watts would be the only realistic choice in that case.
Not all MetricFamilies have units. For example a count of HTTP requests wouldn't have a unit. Technically the unit would be HTTP requests, but in that sense the entire MetricFamily name is the unit. Going to that extreme would not be useful. The possibility of having good axes on graphs in downstream systems for human consumption should always be kept in mind.
Statelessness
The wire format defined by OpenMetrics is stateless across expositions. What information has been exposed before MUST have no impact on future expositions. Each exposition is a self-contained snapshot of the current state of the exposer.
The same self-contained exposition MUST be provided to existing and new ingestors.
A core design choice is that exposers MUST NOT exclude a metric merely because it has had no recent changes, or observations. An exposer must not make any assumptions about how often ingestors are consuming expositions.
Exposition Across Time and Metric Evolution
Metrics are most useful when their evolution over time can be analysed, so accordingly expositions must make sense over time. Thus, it is not sufficient for one single exposition on its own to be useful and valid. Some changes to metric semantics can also break downstream users.
Parsers commonly optimize by caching previous results. Thus, changing the order in which labels are exposed across expositions SHOULD be avoided even though it is technically not breaking This also tends to make writing unit tests for exposition easier.
Metrics and samples SHOULD NOT appear and disappear from exposition to exposition, for example a counter is only useful if it has history. In principle, a given Metric should be present in exposition from when the process starts until the process terminates. It is often not possible to know in advance what Metrics a MetricFamily will have over the lifetime of a given process (e.g. a label value of a latency histogram is a HTTP path, which is provided by an end user at runtime), but once a counter-like Metric is exposed it should continue to be exposed until the process terminates. That a counter is not getting increments doesn't invalidate that it still has its current value. There are cases where it may make sense to stop exposing a given Metric; see the section on Missing Data.
In general changing a MetricFamily's type, or adding or removing a label from its Metrics will be breaking to ingestors.
A notable exception is that adding a label to the value of an Info MetricPoints is not breaking. This is so that you can add additional information to an existing Info MetricFamily where it makes sense to be, rather than being forced to create a brand new info metric with an additional label value. ingestor systems should ensure that they are resilient to such additions.
Changing a MetricFamily's Help is not breaking. For values where it is possible, switching between floats and ints is not breaking. Adding a new state to a stateset is not breaking. Adding unit metadata where it doesn't change the metric name is not breaking.
Histogram buckets SHOULD NOT change from exposition to exposition, as this is likely to both cause performance issues and break ingestors and cause. Similarly all expositions from any consistent binary and environment of an application SHOULD have the same buckets for a given Histogram MetricFamily, so that they can be aggregated by all ingestors without ingestors having to implement histogram merging logic for heterogeneous buckets.
An exception might be occasional manual changes to buckets which are considered breaking, but may be a valid tradeoff when performance characteristics change due to a new software release.
Even if changes are not technically breaking, they still carry a cost. For example frequent changes may cause performance issues for ingestors. A Help string that varies from exposition to exposition may cause each Help value to be stored. Frequently switching between int and float values could prevent efficient compression.
NaN
NaN is a number like any other in OpenMetrics, usually resulting from a division by zero such as for a summary quantile if there have been no observations recently. NaN does not have any special meaning in OpenMetrics, and in particular MUST NOT be used as a marker for missing or otherwise bad data.
Missing Data
There are valid cases when data stops being present. For example a filesystem can be unmounted and thus its Gauge Metric for free disk space no longer exists. There is no special marker or signal for this situation. Subsequent expositions simply do not include this Metric.
Exposition Performance
Metrics are only useful if they can be collected in reasonable time frames. Metrics that take minutes to expose are not considered useful.
As a rule of thumb, exposition SHOULD take no more than a second.
Metrics from legacy systems serialized through OpenMetrics may take longer. For this reason, no hard performance assumptions can be made.
Exposition SHOULD be of the most recent state. For example, a thread serving the exposition request SHOULD NOT rely on cached values, to the extent it is able to bypass any such caching
Concurrency
For high availability and ad-hoc access a common approach is to have multiple ingestors. To support this, concurrent expositions MUST be supported. All BCPs for concurrent systems SHOULD be followed, common pitfalls include deadlocks, race conditions, and overly-coarse grained locking preventing expositions progressing concurrently.
Metric Naming and Namespaces
We aim for a balance between understandability, avoiding clashes, and succinctness in the naming of metrics and label names. Names are separated through underscores, so metric names end up being in “snake_case”.
To take an example "http_request_seconds" is succinct but would clash between large numbers of applications, and it's also unclear exactly what this metric is measuring. For example, it might be before or after auth middleware in a complex system.
Metric names should indicate what piece of code they come from. So a company called A Company Manufacturing Everything might prefix all metrics in their code with "acme_", and if they had a HTTP router library measuring latency it might have a metric such as "acme_http_router_request_seconds" with a Help string indicating that it is the overall latency.
It is not the aim to prevent all potential clashes across all applications, as that would require heavy handed solutions such as a global registry of metric namespaces or very long namespaces based on DNS. Rather the aim is to keep to a lightweight informal approach, so that for a given application that it is very unlikely that there is clash across its constituent libraries.
Across a given deployment of a monitoring system as a whole the aim is that clashes where the same metric name means different things are uncommon. For example acme_http_router_request_seconds might end up in hundreds of different applications developed by A Company Manufacturing Everything, which is normal. If Another Corporation Making Entities also used the metric name acme_http_router_request_seconds in their HTTP router that's also fine. If applications from both companies were being monitored by the same monitoring system the clash is undesirable, but acceptable as no application is trying to expose both names and no one target is trying to (incorrectly) expose the same metric name twice. If an application wished to contain both My Example Company's and Mega Exciting Company's HTTP router libraries that would be a problem, and one of the metric names would need to be changed somehow.
As a corollary, the more public a library is the better namespaced its metric names should be to reduce the risk of such scenarios arising. acme_ is not a bad choice for internal use within a company, but these companies might for example choose the prefixes acmeverything_ or acorpme_ for code shared outside their company.
After namespacing by company or organisation, namespacing and naming should continue by library/subsystem/application fractally as needed such as the http_router library above. The goal is that if you are familiar with the overall structure of a codebase, you could make a good guess at where the instrumentation for a given metric is given its metric name.
For a common very well known existing piece of software, the name of the software itself may be sufficiently distinguishing. For example bind_ is probably sufficient for the DNS software, even though isc_bind_ would be the more usual naming.
Metric names prefixed by scrape_ are used by ingestors to attach information related to individual expositions, so should not be exposed by applications directly. Metrics that have already been consumed and passed through a general purpose monitoring system may include such metric names on subsequent expositions.
If an exposer wishes to provide information about an individual exposition, a metric prefix such as myexposer_scrape_ may be used. A common example is a gauge myexposer_scrape_duration_seconds for how long that exposition took from the exposer's standpoint.
Within the Prometheus ecosystem a set of per-process metrics has emerged that are consistent across all implementations, prefixed with process_. For example for open file ulimits the MetricFamiles process_open_fds and process_max_fds gauges provide both the current and maximum value. (These names are legacy, if such metrics were defined today they would be more likely called process_fds_open and process_fds_limit). In general it is very challengings to get names with identical semantics like this, which is why different instrumentation should use different names.
Avoid redundancy in metric names. Avoid substrings like "metric", "timer", "stats", "counter", "total", "float64" and so on - by virtue of being a metric with a given type (and possibly unit) exposed via OpenMetrics information like this is already implied so should not be included explicitly. You should not include label names of a metric in the metric name for the same reasons, and in addition subsequent aggregation of the metric by a monitoring system could make such information incorrect.
Avoid including implementation details from other layers of your monitoring system in the metric names contained in your instrumentation. For example a MetricFamily name should not contain the string "openmetrics" merely because it happens to be currently exposed via OpenMetrics somewhere, or "prometheus" merely because your current monitoring system is Prometheus.
Label Namespacing
For label names no explicit namespacing by company or library is recommended, namespacing from the metric name is sufficient for this when considered against the length increase of the label name. However some minimal care to avoid common clashes is recommended.
There are label names such as region, zone, cluster, availability_zone, az, datacenter, dc, owner, customer, stage, service, team, job, instance, environment, and env which are highly likely to clash with labels used to identify targets which a general purpose monitoring system may add. Try to avoid them, adding minimal namespacing may be appropriate in these cases.
The label name "type" is highly generic and should be avoided. For example for HTTP-related metrics "method" would be a better label name if you were distinguishing between GET, POST, and PUT requests.
While there is metadata about metric names such as HELP, TYPE and UNIT there is no metadata for label names. This is as it would be bloating the format for little gain. Out-of-band documentation is one way for exposers could present this their ingestors.
Metric Names versus Labels
There are situations in which both using multiple Metrics within a MetricFamily or multiple MetricFamilies seem to make sense. Summing or averaging aMetricFamily should be meaningful even if it's not always useful. For example, mixing voltage and fan speed is not meaningful.
As a reminder, OpenMetrics is built with the assumption that ingestors can process and perform aggregations on data.
Exposing a total sum alongside other metrics is wrong, as this would result in double-counting upon aggregation in downstream ingestors.
wrong_metric{label="a"} 1
wrong_metric{label="b"} 6
wrong_metric{label="total"} 7
Labels of a Metric should be to the minimum needed to ensure uniqueness as every extra label is one more that users need to consider when determining what Labels to work with downstream. Labels which could be applied many MetricFamilies are candidates for being moved into _info metrics similar to database {{normalization}}. If virtually all users of a Metric could be expected to want the additional label, it may be a better trade-off to add it to all MetricFamilies. For example if you had a MetricFamily relating to different SQL statements where uniqueness was provided by a label containing a hash of the full SQL statements, it would be okay to have another label with the first 500 characters of the SQL statement for human readability.
Experience has shown that downstream ingestors find it easier to work with separate total and failure MetricFamiles rather than using {result="success"} and {result="failure"} Labels within one MetricFamily. Also it is usually better to expose separate read & write and send & receive MetricFamiles as full duplex systems are common and downstream ingestors are more likely to care about those values separately than in aggregate.
All of this is not as easy as it may sound. It's an area where experience and engineering trade-offs by domain-specific experts in both exposition and the exposed system are required to find a good balance.
Metric and Label Name Characters
OpenMetrics builds on the existing widely adopted Prometheus text exposition format and the ecosystem which formed around it. Backwards compatibility is a core design goal. Expanding or contracting the set of characters that are supported by the Prometheus text format would work against that goal. Breaking backwards compatibility would have wider implications than just the wire format. In particular, the query languages created or adopted to work with data transmitted within the Prometheus ecosystem rely on these precise character sets. Label values support full UTF-8, so the format can represent multi-lingual metrics.
Types of Metadata
Metadata can come from different sources. Over the years, two main sources have emerged. While they are often functionally the same, it helps in understanding to talk about their conceptual differences.
"Target metadata" is metadata commonly external to an exposer.  Common examples would be data coming from service discovery, a CMDB, or similar, like information about a datacenter region, if a service is part of a particular deployment, or production or testing. This can be  achieved by either the exposer or the ingestor adding labels to all Metrics that capture this metadata. Doing this through the ingestor is preferred as it is more flexible and carries less overhead. On flexibility, the hardware maintenance team might care about which server rack a machine is located in, whereas the database team using that same machine might care that it contains replica number 2 of the production database. On overhead, hardcoding or configuring this information needs an additional distribution path.
"Exposer metadata" is coming from within an exposer. Common examples would be software version, compiler version, or Git commit SHA.
Supporting Target Metadata in both Push-based and Pull-based Systems
In push-based consumption, it is typical for the exposer to provide the relevant target metadata to the ingestor. In pull-based consumption the push-based approach could be taken, but more typically the ingestor already knows the metadata of the target a-priori such as from a machine database or service discovery system, and associates it with the metrics as it consumes the exposition.
OpenMetrics is stateless and provides the same exposition to all ingestors, which is in conflict with the push-style approach. In addition the push-style approach would break pull-style ingestors, as unwanted metadata would be exposed.
One approach would be for push-style ingestors to provide target metadata based on operator configuration out-of-band, for example as a HTTP header. While this would transport target metadata for push-style ingestors, and is not precluded by this standard, it has the disadvantage that even though pull-style ingestors should use their own target metadata, it is still often useful to have access to the metadata the exposer itself is aware of.
The preferred solution is to provide this target metadata as part of the exposition, but in a way that does not impact on the exposition as a whole. Info MetricFamilies are designed for this. An exposer may include an Info MetricFamily called "target" with a single Metric with no labels with the metadata. An example in the text format might be:
# TYPE target info
# HELP target Target metadata
target_info{env="prod",hostname="myhost",datacenter="sdc",region="europe",owner="frontend"} 1
When an exposer is providing this metric for this purpose it SHOULD be first in the exposition. This is for efficiency, so that ingestors relying on it for target metadata don't have to buffer up the rest of the exposition before applying business logic based on its content.
Exposers MUST NOT add target metadata labels to all Metrics from an exposition, unless explicitly configured for a specific ingestor. Exposers MUST NOT prefix MetricFamily names or otherwise vary MetricFamily names based on target metadata.
Generally, the same Label should not appear on every Metric of an exposition, but there are rare cases where this can be the result of emergent behaviour. Similarly all MetricFamily names from an exposer may happen to share a prefix in very small expositions. For example an application written in the Go language by A Company Manufacturing Everything would likely include metrics with prefixes of acme_, go_, process_, and metric prefixes from any 3rd party libraries in use.
Exposers can expose exposer metadata as Info MetricFamilies.
The above discussion is in the context of individual exposers. An exposition from a general purpose monitoring system may contain metrics from many individual targets, and thus may expose multiple target info Metrics. The metrics may already have had target metadata added to them as labels as part of ingestion. The metric names MUST NOT be varied based on target metadata. For example it would be incorrect for all metrics to end up being prefixed with staging_ even if they all originated from targets in a staging environment).
Client Calculations and Derived Metrics
Exposers should leave any math or calculation up to ingestors. A notable exception is the  Summary quantile which is unfortunately required for backwards compatibility. Exposition should be of raw values which are useful over arbitrary time periods.
As an example, you should not expose a gauge with the average rate of increase of a counter over the last 5 minutes. Letting the ingestor calculate the increase over the data points they have consumed across expositions has better mathematical properties and is more resilient to scrape failures.
Another example is the average event size of a histogram/summary. Exposing the average rate of increase of a counter since an application started or since a Metric was created has the problems from the earlier example and it also prevents aggregation.
Standard deviation also falls into this category. Exposing a sum of squares as a counter would be the correct approach. It was not included in this standard as a Histogram value because 64bit floating point precision is not sufficient for this to work in practice. Due to the squaring only half the 53bit mantissa would be available in terms of precision. As an example a histogram observing 10k events per second would lose precision within 2 hours. Using 64bit integers would be no better due to the loss of the floating decimal point because a nanosecond resolution integer typically tracking events of a second in length would overflow after 19 observations.
This design decision can be revisited when 128bit floating point numbers become common.
Another example is to avoid exposing a request failure ratio, exposing separate counters for failed requests and total requests instead.
Number Types
For a counter that was incremented a million times per second it would take over a century to begin to lose precision with a float64 as it has a 53 bit mantissa. Yet a 100 Gbps network interface's octet throughput precision could begin to be lost with a float64 within around 20 hours. While losing 1KB of precision over the course of years for a 100Gbps network interface is unlikely to be a problem in practice, int64s are an option for integral data with such a high throughput.
Summary quantiles must be float64, as they are estimates and thus fundamentally inaccurate.
Exposing Timestamps
One of the core assumptions of OpenMetrics is that exposers expose the most up to date snapshot of what they're exposing.
While there are limited use cases for attaching timestamps to exposed data, these are very uncommon. Data which had timestamps previously attached, in particular data which has been ingested into a general purpose monitoring system may carry timestamps. Live or raw data should not carry timestamps. It is valid to expose the same metric MetricPoint value with the same timestamp across expositions, however it is invalid to do so if the underlying metric is now missing.
Time synchronization is a hard problem and data should be internally consistent in each system. As such, ingestors should be able to attach the current timestamp from their perspective to data rather than based on the system time of the exposer device.
With timestamped metrics it is not generally possible to detect the time when a Metric went missing across expositions. However with non-timestamped metrics the ingestor can use its own timestamp from the exposition where the Metric is no longer present.
All of this is to say that, in general, MetricPoint timestamps should not be exposed, as it should be up to the ingestor to apply their own timestamps to samples they ingest.
Tracking When Metrics Last Changed
Presume you had a counter my_counter which was initialized, and then later incremented by 1 at time 123. This would be a correct way to expose it in the text format:
# HELP my_counter Good increment example
# TYPE my_counter counter
my_counter_total 1
As per the parent section, ingestors should be free to attach their own timestamps, so this would be incorrect:
# HELP my_counter Bad increment example
# TYPE my_counter counter
my_counter_total 1 123
In case the specific time of the last change of a counter matters, this would be the correct way:
# HELP my_counter Good increment example
# TYPE my_counter counter
my_counter_total 1
# HELP my_counter_last_increment_timestamp_seconds When my_counter was last incremented
# TYPE my_counter_last_increment_timestamp_seconds gauge
# UNIT my_counter_last_increment_timestamp_seconds seconds
my_counter_last_increment_timestamp_seconds 123
By putting the timestamp of last change into its own Gauge as a value, ingestors are free to attach their own timestamp to both Metrics.
Experience has shown that exposing absolute timestamps (epoch is considered absolute here) is more robust than time elapsed, seconds since, or similar. In either case, they would be gauges. For example:
# TYPE my_boot_time_seconds gauge
# HELP my_boot_time_seconds Boot time of the machine
# UNIT my_boot_time_seconds seconds
my_boot_time_seconds 1256060124
Is better than:
# TYPE my_time_since_boot_seconds gauge
# HELP my_time_since_boot_seconds Time elapsed since machine booted
# UNIT my_time_since_boot_seconds seconds
my_time_since_boot_seconds 123
Conversely, there are no best practice restrictions on exemplars timestamps.
Keep in mind that due to race conditions or time not being perfectly synced across devices, that an exemplar timestamp may appear to be slightly in the future relative to a ingestor's system clock or other metrics from the same exposition. Similarly it is possible that a "_created" for a MetricPoint could appear to be slightly after an exemplar or sample timestamp for that same MetricPoint.
Keep in mind that there are monitoring systems in common use which support everything from nanosecond to second resolution, so having two MetricPoints that have the same timestamp when truncated to second resolution may cause an apparent duplicate in the ingestor. In this case the MetricPoint with the earliest timestamp MUST be used.
Thresholds
Exposing desired bounds for a system can make sense, but proper care needs to be taken. For values which are universally true, it can make sense to emit Gauge metrics for such thresholds. For example, a data center HVAC system knows the current measurements, the setpoints, and the alert setpoints. It has a globally valid and correct view of the desired system state.
As a counter example, some thresholds can change with scale, deployment model, or over time. A certain amount of CPU usage may be acceptable in one setting and undesirable in another. Aggregation of values can further change acceptable values. In such a system, exposing bounds could be counter-productive.
For example the maximum size of a queue may be exposed alongside the number of items currently in the queue like:
# HELP acme_notifications_queue_capacity The capacity of the notifications queue.
# TYPE acme_notifications_queue_capacity gauge
acme_notifications_queue_capacity 10000
# HELP acme_notifications_queue_length The number of notifications in the queue.
# TYPE acme_notifications_queue_length gauge
acme_notifications_queue_length 42
Size Limits
This standard does not prescribe any particular limits on the number of samples exposed by a single exposition, the number of labels that may be present, the number of states a stateset may have, the number of labels in an info value, or metric name/label name/label value/help character limits.
Specific limits run the risk of preventing reasonable use cases, for example while a given exposition may have an appropriate number of labels after passing through a general purpose monitoring system a few target labels may have been added that would push it over the limit. Specific limits on numbers such as these would also not capture where the real costs are for general purpose monitoring systems. These guidelines are thus both to aid exposers and ingestors in understanding what is reasonable.
On the other hand, an exposition which is too large in some dimension could cause significant performance problems compared to the benefit of the metrics exposed. Thus some guidelines on the size of any single exposition would be useful.
ingestors may choose to impose limits themselves, for in particular to prevent attacks or outages. Still, ingestors need to consider reasonable use cases and try not to  disproportionately impact them. If any single value/metric/exposition exceeds such limits then the whole exposition must be rejected.
In general there are three things which impact the performance of a general purpose monitoring system ingestion time series data: the number of unique time series, the number of samples over time in those series, and the number of unique strings such as metric names, label names, label values, and HELP. ingestors can control how often they ingest, so that aspect does not need further consideration.
The number of unique time series is roughly equivalent to the number of non-comment lines in the text format. As of 2020, 10 million time series in total is considered a large amount and is commonly the order of magnitude of the upper bound of any single-instance ingestor. Any single exposition should not go above 10k time series without due diligence. One common consideration is horizontal scaling: What happens if you scale your instance count by 1-2 orders of magnitude? Having a thousand top-of-rack switches in a single deployment would have been hard to imagine 30 years ago. If a target was a singleton (e.g. exposing metrics relating to an entire cluster) then several hundred thousand time series may be reasonable. It is not the number of unique MetricFamilies or the cardinality of individual labels/buckets/statesets that matters, it is the total order of magnitude of the time series. 1,000 gauges with one Metric each are as costly as a single gauge with 1,000 Metrics.
If all targets of a particular type are exposing the same set of time series, then each additional targets' strings poses no incremental cost to most reasonably modern monitoring systems. If however each target has unique strings, there is such a cost. As an extreme example, a single 10k character metric name used by many targets is on its own very unlikely to be a problem in practice. To the contrary, a thousand targets each exposing a unique 36 character UUID is over three times as expensive as that single 10k character metric name in terms of strings to be stored assuming modern approaches. In addition, if these strings change over time older strings will still need to be stored for at least some time, incurring extra cost. Assuming the 10 million times series from the last paragraph, 100MB of unique strings per hour might indicate a use case for then the use case may be more like event logging, not metric time series.
There is a hard 128 UTF-8 character limit on exemplar length, to prevent misuse of the feature for tracing span data and other event logging.
Security
Implementors MAY choose to offer authentication, authorization, and accounting; if they so choose, this SHOULD be handled outside of OpenMetrics.
All exposer implementations SHOULD be able to secure their HTTP traffic with TLS 1.2 or later.
If an exposer implementation does not support encryption, operators SHOULD use reverse proxies, firewalling, and/or ACLs where feasible.
Metric exposition should be independent of production services exposed to end users; as such, having a /metrics endpoint on ports like TCP/80, TCP/443, TCP/8080, and TCP/8443 is generally discouraged for publicly exposed services using OpenMetrics.
IANA
While currently most implementations of the Prometheus exposition format are using non-IANA-registered ports from an informal registry at {{PrometheusPorts}}, OpenMetrics can be found on a well-defined port.
The port assigned by IANA for clients exposing data is <9099 requested for historical consistency>.
If more than one metric endpoint needs to be reachable at a common IP address and port, operators might consider using a reverse proxy that communicates with exposers over localhost addresses. To ease multiplexing, endpoints SHOULD carry their own name in their path, i.e. 
/node_exporter/metrics. Expositions SHOULD NOT be combined into one exposition, for the reasons covered under "Supporting target metadata in both push-based and pull-based systems" and to allow for independent ingestion without a single point of failure.
OpenMetrics would like to register two MIME types, 
application/openmetrics-text and 
application/openmetrics-proto.

OpenMetrics 2.0 [EXPERIMENTAL]
Version: 2.0.0-rc1
Status: Experimental
Date: October 2026
Authors: Arthur Silva Sens, Bartłomiej Płotka, David Ashpole, György Krajcsovits, Owen Williams, Richard Hartmann
Emeritus: Ben Kochie, Brian Brazil, Rob Skillington
Created in 2012, Prometheus has been the default for cloud-native observability since 2015. A central part of Prometheus' design is its text metric exposition format, called the Prometheus exposition format 0.0.4, stable since 2014. In this format, special care has been taken to make it easy to generate, to ingest, and to understand by humans. As of 2020, there are more than 700 publicly listed exporters, an unknown number of unlisted exporters, and thousands of native library integrations using this format. Dozens of ingestors from various projects and companies support consuming it.
In 2020, OpenMetrics 1.0 was released to clean up and tighten the specification with the additional purpose of bringing it into IETF. OpenMetrics 1.0 text exposition documented a working standard with wide and organic adoption among dozens of exporters, integrations, and ingestors.
Around 2024, the OpenMetrics project was incorporated under the CNCF Prometheus project umbrella. Together with production learnings from deploying OpenMetrics 1.0 on wide scale and a backlog of new Prometheus innovations missing from the text formats, Prometheus community decided to pursue a second version of the OpenMetrics standard.
The intention of OpenMetrics 2.0 is to use OpenMetrics 1.0 as a foundation and enhance it to achieve even greater reliability, usability and consistency with the modern Prometheus data model, without sacrificing the ease of use and readability. OpenMetrics 2.0 also improves compatibility with OpenTelemetry data model and naming conventions.
This document is meant to be used as a standalone specification.
NOTE: This is a release candidate (RC) version of the OpenMetrics 2.0 specification. This means that this specification is currently in an experimental state--no major changes are expected, but we reserve the right to break the compatibility if it's necessary, based on the early adopters' feedback. The potential feedback, questions and suggestions should be added as an issue on the 
prometheus/openmetrics repository.
Overview
Metrics are a specific kind of telemetry data. They represent a snapshot of the current state for a set of data. They are distinct from logs or events, which focus on records or information about individual events.
OpenMetrics is primarily a wire format, independent of any particular transport for that format. The format is expected to be consumed on a regular basis and to be meaningful over successive expositions.
Implementers SHOULD expose metrics in the OpenMetrics text format in response to an HTTP GET request to a documented URL for a given process or device. This endpoint SHOULD be called "/metrics". Implementers MAY also expose OpenMetrics formatted metrics in other ways, for example, by regularly pushing metric sets to an operator-configured endpoint over HTTP.
Metrics and Time Series
This standard expresses all system states as numerical values; counts, current values, distributions, enumerations, and boolean states being common examples. Contrary to metrics, singular events occur at a specific time. Metrics tend to aggregate data temporally and provide a sample of the system state. While this can lose information, the reduction in overhead is an engineering trade-off commonly chosen in many modern monitoring systems.
Time series are a record of changing information over time. Common examples of metric time series would be network interface counters, device temperatures, BGP connection states, latency distributions, and alert states.
Normative Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174 when, and only when, they appear in all capitals, as shown here.
The word "RESERVED" is used in this document to designate values, names, or fields that are set aside for future use or for use by this standard itself. Values, names, or fields described as "RESERVED" MUST NOT be used unless explicitly permitted by this standard or a future version thereof.
Data Model
This section MUST be read together with the ABNF section. In case of disagreements between the two, the ABNF's restrictions MUST take precedence.
Data Types
Sample Values
Metric values in OpenMetrics MUST be either Number or CompositeValue.
Number
Number value MUST be either floating point or integer.
Note that ingestors of the format MAY only support float64, for example Go's 
float64 which is an IEEE 754-2008 double-precision (binary64) floating-point number with approximately 15–17 significant decimal digits of precision. The non-real values NaN, +Inf and -Inf MUST be supported. NaN value MUST NOT be considered a missing value, but it MAY be used to signal a division by zero or any other mathematical operation that yields an undefined or indeterminate result.
Booleans MUST be represented as a Number value where 
1 is true and 
0 is false.
CompositeValue
CompositeValue MUST contain all information necessary to recreate a sample value for Metric within the MetricFamily.
The following MetricFamily Types MUST use CompositeValue for Metric Values:
Histogram MetricFamily Type.
GaugeHistogram MetricFamily Type.
Summary MetricFamily Type.
Other MetricFamily Types MUST use Numbers.
Timestamps
Timestamps MUST be Unix Epoch in seconds. Timestamps SHOULD be floating point to represent sub-second precision, for example milliseconds or microseconds. Negative timestamps MAY be used.
There are few places in this standard that use Timestamps:
Exemplar's Timestamp
Sample's Timestamp
Sample's Start Timestamp
Strings
Strings MUST only consist of valid UTF-8 characters and MAY be zero length. NULL (ASCII 0x0) MUST be supported.
Label
Labels are key-value pairs consisting of strings.
Label names beginning with one or more underscores are RESERVED and MUST NOT be used unless specified by this standard. Such Label names MAY be used in place of TYPE and UNIT metadata in cases where MetricFamilies' metadata might otherwise be conflicting, such as metric federation cases.
Label names SHOULD follow the restrictions in the ABNF section under the 
label-name section. Label names MAY be any quoted escaped UTF-8 string as described in the ABNF section. Be aware that exposing UTF-8 metrics may reduce usability.
Empty label values SHOULD be treated as if the label was not present.
LabelSet
A LabelSet MUST consist of Labels and MAY be empty. Label names MUST be unique within a LabelSet.
Exemplars
Exemplars are references to data outside of the MetricSet. A common use case are IDs of program traces.
Exemplars MUST consist of a LabelSet and a Number value, and MUST have a timestamp. The LabelSet SHOULD NOT contain any Label names included in the Metric's LabelSet. The timestamp SHOULD be before or equal to the Sample's timestamp, if present. The timestamp SHOULD be after or equal to the Sample's start timestamp, if present. Exemplars of a Sample SHOULD have the same Label names to have a consistent style.
The Exemplar's timestamp SHOULD be close to the point when it was observed, but doesn't have to be exact. For example, if getting an exact timestamp is costly, it is acceptable to use some external source or an estimate.
When an exemplar references a Trace Context, it SHOULD use the 
trace_id key for the trace-id field, and the 
span_id key for the 
parent-id field.
While there's no hard limit specified, Exemplar's LabelSet SHOULD NOT be used to transport large data like tracing span details or other event logging.
Ingestors MAY truncate the Exemplar's LabelSet or discard Exemplars. When truncating the Exemplar's LabelSet the 
trace_id and 
span_id SHOULD be preserved even after truncation.
Sample
A Sample is a single data point within a Metric. It MUST have a Value, MAY have a Timestamp. It MAY include Exemplars and MAY have a Start Timestamp, depending on the MetricFamily Type.
Samples SHOULD NOT have Timestamps. See Exposing Timestamps for why this is not recommended. If present, a Sample's Timestamp specifies when the value was observed.
If present, a Sample's Start Timestamp SHOULD specify when the measurement period started. This can help ingestors discern between new metrics and long-running ones it did not see before and detect counter resets even when the counter value has not decreased between ingestions.
Metric
Metrics are defined by a unique LabelSet within a MetricFamily. Metrics MUST contain a list of one or more Samples. If more than one Sample is exposed for a Metric, then its Samples MUST have monotonically increasing Timestamps.
Metrics with the same name for a given MetricFamily SHOULD have the same set of label names in their LabelSet.
MetricFamily
A MetricFamily MAY have zero or more Metrics. Every Metric within a MetricFamily MUST have a unique LabelSet. A MetricFamily MUST have a name and SHOULD have Help, Type, and Unit metadata.
Name
MetricFamily name:
MUST be string.
MUST be unique within a MetricSet.
MUST be the same as every Metric's Name in the family.
NOTE: OpenMetrics 1.0 required mandatory suffixes for MetricName and a matching MetricFamily name without such suffixes. To improve parser reliability (i.e. matching MetricFamily metadata) and future compatibility, this specification requires Metric name to strictly match its MetricFamily name.
Names SHOULD be in snake_case. Names SHOULD follow the restrictions in the ABNF section under 
metricname. MetricFamily names MAY be any quoted and escaped UTF-8 string as described in the ABNF section. Be aware that exposing UTF-8 metrics may reduce usability, especially when 
_total or unit suffixes are not included in the names.
Colons in MetricFamily names are RESERVED to signal that the MetricFamily is the result of a calculation or aggregation of a general purpose monitoring system.
MetricFamily names beginning with one or more underscores are RESERVED and MUST NOT be used unless specified by this standard.
Discouraged Suffixes
MetricFamily name SHOULD NOT end with 
_count, 
_sum, 
_gcount, 
_gsum, 
_bucket. Specifically, a name SHOULD NOT create a MetricName collision when converted to the OpenMetrics 1.0 Text. Ingestors MAY reject the MetricSet with such MetricFamily.
A non-compliant example would be a gauge called 
foo_bucket and a histogram called 
foo. Exposers negotiating the older OpenMetrics or Text formats, or ingestors which support only the older data model could end up storing the 
foo histogram in the classic representation (
foo_bucket, 
foo_count, 
foo_sum), which would clash with the gauge and cause a scrape rejection or dropped data.
This rule exists because this specification is following a shift in Prometheus ecosystem towards composite values instead of the "classic" representation. However, this transformation will take time. Avoiding such suffixes improves compatibility with older ingestors and the eventual migration process.
Type
Type specifies the MetricFamily type. Valid values are "unknown", "gauge", "counter", "stateset", "info", "histogram", "gaugehistogram", and "summary".
Unit
Unit specifies MetricFamily units. If non-empty, it SHOULD be a suffix of the MetricFamily name separated by an underscore. Further type specific suffixes come after the unit suffix. Exposing metrics without the unit being a suffix of the MetricFamily name directly to end-users may reduce the usability due to confusion about what the metric's unit is. For recommended unit values see Units and Base Units.
Help
Help is a string and SHOULD be non-empty. It is used to give a brief description of the MetricFamily for human consumption and SHOULD be short enough to be used as a tooltip.
MetricSet
A MetricSet is the top level object exposed by OpenMetrics. It MUST consist of MetricFamilies and MAY be empty.
Each MetricFamily name MUST be unique. The same label name and value SHOULD NOT appear on every Metric within a MetricSet.
There is no specific ordering of MetricFamilies required within a MetricSet. An exposer MAY make an exposition easier to read for humans, for example sort alphabetically if the performance tradeoff makes sense.
If present, an Info MetricFamily called "target_info" per the Supporting target metadata in both push-based and pull-based systems section below SHOULD be first.
MetricFamily Types
Gauge
Gauges are current measurements, such as bytes of memory currently used or the number of items in a queue. For gauges the absolute value is what is of interest to a user.
A Sample in a Metric with the Type gauge MUST have a Number value.
Gauges MAY increase, decrease, or stay constant over time. Even if they only ever go in one direction, they might still be gauges and not counters. The size of a log file would usually only increase, a resource might decrease, and the limit of a queue size may be constant.
Counter
Counters measure discrete events. Common examples are the number of HTTP requests received, CPU seconds spent, or bytes sent. For counters how quickly they are increasing over time is what is of interest to a user.
The MetricFamily name for Counters SHOULD end in 
_total. Exposing metrics without a 
_total suffix may reduce the usability due to confusion about what the metric's Type is.
A Sample in a Metric with the Type Counter SHOULD have a Start Timestamp.
A Sample in a Metric with the Type Counter MUST have a Number value which is non-NaN. The value MUST be monotonically non-decreasing over time, unless it is reset to 0, and start from 0. The value MAY reset its value to 0. If present, the corresponding Start Timestamp MUST also be set to the approximate reset time.
A Sample in a Metric with the type Counter MAY have exemplars.
StateSet
StateSets represent a series of related boolean values, also called a bitset. If ENUMs need to be encoded this MAY be done via StateSet.
A StateSet is structured as a set of Metrics, one for each state, called a StateSet MetricGroup.
NOTE: In OpenMetrics 1.0, Metrics are composed of MetricPoints (e.g. a Histogram metric has a MetricPoint representing each Bucket with a special "le" label), which is no longer the case in OpenMetrics 2.0. An OpenMetrics 1.0 StateSet Metric is equivalent to an OpenMetrics 2.0 StateSet MetricGroup, and an OpenMetrics 1.0 StateSet MetricPoint is equivalent to an OpenMetrics 2.0 StateSet Metric.
A StateSet MetricGroup contains one or more states and MUST contain one Metric with a boolean value per state. States have a name which is a String.
If encoded as a StateSet, ENUMs MUST have exactly one Sample which is 
1 (true) within a MetricGroup, for a single Timestamp.
This is suitable where the enum value changes over time, and the number of States isn't much more than a handful.
MetricFamilies of Type StateSets MUST have an empty Unit string.
Info
Info metrics are used to expose textual information which SHOULD NOT change during process lifetime. Common examples are an application's version, revision control commit, and the version of a compiler.
The MetricFamily name for Info metrics MUST end in 
_info.
MetricFamilies of Type Info MUST have an empty Unit string.
Histogram
Histograms measure distributions of discrete events. Common examples are the latency of HTTP requests, function runtimes, or I/O request sizes.
A Histogram Sample MUST contain Count and Sum.
The Count value MUST be equal to the number of measurements taken by the Histogram. The Count is a counter semantically. The Count SHOULD be an integer. The Count MUST NOT be negative. The Count SHOULD NOT be +Inf, NaN.
Float Count is allowed to make it possible to expose results of arithmetic operations on histograms, such as addition that may result in values beyond the range of integers.
The Sum value MUST be equal to the sum of all the measured event values. The Sum is only a counter semantically as long as there are no negative event values measured by the Histogram.
A Histogram MUST measure values that are not NaN in either Classic Buckets or Native Buckets or both. Measuring NaN is different for Classic and Native Buckets, see in their respective sections.
Every Bucket MUST have well-defined boundaries and a value. The bucket value is called the bucket count colloquially. Boundaries of a Bucket MUST NOT be NaN. Bucket values are counters semantically. Bucket values SHOULD be integers. Bucket values MUST NOT be negative. Bucket values SHOULD NOT be +Inf, NaN.
Float bucket values are allowed to make it possible to expose results of arithmetic operations on histograms, such as addition that may result in values beyond the range of integers.
A Histogram SHOULD NOT include NaN measurements as including NaN in the Sum will make the Sum equal to NaN and mask the sum of the real measurements for the lifetime of the time series. If a Histogram includes NaN measurements, then NaN measurements MUST be counted in the Count and the Sum MUST be NaN.
If a Histogram includes +Inf or -Inf measurement, then +Inf or -Inf MUST be counted in Count and MUST be added to the Sum, potentially resulting in +Inf, -Inf or NaN in the Sum, the latter for example in case of adding +Inf to -Inf. Note that in this case the Sum of finite measurements is masked until the next reset of the Histogram.
A Histogram Sample SHOULD have a Start Timestamp.
If the Histogram Metric has Samples with Classic Buckets, the Histogram's Metric's LabelSet MUST NOT have a "le" label name, because in case the Samples are stored as classic histogram series with the 
_bucket suffix, then the "le" label in the Histogram will conflict with the "le" label generated from the bucket thresholds.
The Histogram Type is cumulative over time, but MAY be reset. When a Histogram is reset, the Sum, Count, Classic Buckets and Native Buckets MUST be reset to their zero state, and if the Start Timestamp is present then it MUST be set to the approximate reset time. Histogram resets can be useful for limiting the number of Native Buckets used by Histograms.
A Histogram Sample MAY have exemplars. The values of exemplars in a Histogram Sample SHOULD be evenly distributed, such as by keeping one exemplar for each Classic Bucket if Classic Buckets are included.
Classic Buckets
Every Classic Bucket MUST have a threshold. Classic Bucket thresholds within a Sample MUST be unique. Classic Bucket thresholds MAY be negative.
A Classic Bucket MUST count the number of measured values less than or equal to its threshold, including measured values that are also counted in lower buckets. This allows monitoring systems to drop any bucket other than the +Inf bucket for performance or anti-denial-of-service reasons in a way that loses granularity but is still a valid Histogram.
As an example, for a metric representing request latency in seconds with Classic Buckets and thresholds 1, 2, 3, and +Inf, it follows that value_1 <= value_2 <= value_3 <= value_+Inf. If ten requests took one second each, the values of the 1, 2, 3, and +Inf buckets will be all equal to 10.
Histogram Samples with Classic Buckets MUST have one Classic Bucket with a +Inf threshold. The +Inf bucket counts all measurements. The Count value MUST be equal to the value of the +Inf bucket.
Exposed Classic Bucket thresholds SHOULD stay constant over time and between targets whose metrics are intended to be aggregated. A change of thresholds may prevent the affected histograms to be part of the same operation (e.g. an aggregation of different metrics or a rate calculation over time).
If the NaN value is allowed, it MUST be counted in the +Inf bucket, and MUST NOT be counted in any other bucket. The rationale is that NaN does not belong to any bucket mathematically, however instrumentation libraries traditionally put it into the +Inf bucket.
Native Buckets
Histogram Samples with Native Buckets MUST have a Schema value. The Schema MUST be an 8-bit signed integer between -4 and 8 (inclusive), these are called Standard (exponential) schemas.
Schema values outside the -4 to 8 range are reserved for future use and MUST NOT be used.
For any Standard Schema 
n, the Histogram Sample MAY contain positive and/or negative Native Buckets and MUST contain a zero Native Bucket. Empty positive or negative Native Buckets SHOULD NOT be present.
In case of Standard Schemas, the boundaries of a positive or negative Native Bucket with index 
i MUST be calculated as follows (using Python syntax):
The upper inclusive limit of a positive Native Bucket: 
(2**2**-n)**i
The lower exclusive limit of a positive Native Bucket: 
(2**2**-n)**(i-1)
The lower inclusive limit of a negative Native Bucket: 
-((2**2**-n)**i)
The upper exclusive limit of a negative Native Bucket: 
-((2**2**-n)**(i-1))
i is an integer number that MAY be negative.
There are exceptions to the rules above concerning the largest and smallest finite values representable as a float64 (called MaxFloat64 and MinFloat64) and the positive and negative infinity values (+Inf and -Inf):
The positive Native Bucket that contains MaxFloat64 (according to the boundary formulas above) has an upper inclusive limit of MaxFloat64 (rather than the limit calculated by the formulas above, which would overflow float64).
The next positive Native Bucket (index 
i+1 relative to the bucket from the previous item) has a lower exclusive limit of MaxFloat64 and an upper inclusive limit of +Inf. (It could be called a positive Native overflow Bucket.)
The negative Native Bucket that contains MinFloat64 (according to the boundary formulas above) has a lower inclusive limit of MinFloat64 (rather than the limit calculated by the formulas above, which would underflow float64).
The next negative Native Bucket (index 
i+1 relative to the bucket from the previous item) has an upper exclusive limit of MinFloat64 and a lower inclusive limit of -Inf. (It could be called a negative Native overflow Bucket.)
Native Buckets beyond the +Inf and -Inf buckets described above MUST NOT be used.
The boundaries of the zero Native Bucket are 
[-threshold, threshold] inclusively. The Zero threshold MUST be a non-negative float64 value (threshold >= 0.0).
If the Zero threshold is positive (threshold > 0), then any measured value that falls into the zero Native Bucket MUST be counted towards the zero Native Bucket and MUST NOT be counted in any other native bucket. The Zero threshold SHOULD be equal to a lower limit of an arbitrary Native Bucket.
If the NaN value is not allowed, then the Count value MUST be equal to the sum of the negative, positive and zero Native Buckets.
If the NaN value is allowed, it MUST NOT be counted in any Native Bucket, and MUST be counted towards the Count. The difference between the Count and the sum of the negative, positive and zero Native Buckets MUST BE the number of NaN observations. The rationale is that NaN does not belong to any bucket mathematically.
GaugeHistogram
GaugeHistograms measure current distributions. Common examples are how long items have been waiting in a queue, or size of the requests in a queue.
A GaugeHistogram Sample MUST contain Gcount, Gsum values.
The Gcount value MUST be equal to the number of measurements currently in the GaugeHistogram. The Gcount is a gauge semantically. The Gcount SHOULD be an integer. The Gcount MUST NOT be negative. The Gcount SHOULD NOT be +Inf, NaN.
Float Gcount is allowed to make it possible to expose results of arithmetic operations on GaugeHistograms, such as the rate of change of a Histogram over time.
The Gsum value MUST be equal to the sum of all the measured values currently in the GaugeHistogram. The Gsum is a gauge semantically.
A GaugeHistogram MUST measure values that are not NaN in either Classic Buckets or Native Buckets or both. Measuring NaN is different for Classic and Native Buckets, see in their respective sections.
If a GaugeHistogram stops measuring values in either Classic or Native Buckets and keeps measuring values in the other, it MUST clear and not expose the buckets it stopped measuring into. This avoids exposing different distribution from the two kind of buckets at the same time.
Every Bucket MUST have well-defined boundaries and a value. Boundaries of a Bucket MUST NOT be NaN. Bucket values are gauges semantically. Bucket values SHOULD be integers. Bucket values MUST NOT be negative. Bucket values SHOULD NOT be +Inf, NaN.
Float bucket values are allowed to make it possible to expose results of arithmetic operations on GaugeHistograms, such as the rate of change of a Histogram over time.
A GaugeHistogram SHOULD NOT include NaN measurements. If a GaugeHistogram includes NaN measurements, then NaN measurements MUST be counted in the Gcount and the Gsum MUST be NaN.
If a GaugeHistogram includes +Inf or -Inf measurement, then +Inf or -Inf MUST be counted in Gcount and MUST be added to the Gsum, potentially resulting in +Inf, -Inf or NaN in the Gsum, the latter for example in case of adding +Inf to -Inf.
If the GaugeHistogram Metric has Samples with Classic Buckets, the GaugeHistogram's Metric's LabelSet MUST NOT have a "le" label name, because in case the Samples are stored as classic histogram series with the 
_bucket suffix, then the "le" label in the GaugeHistogram will conflict with the "le" label generated from the bucket thresholds.
The Classic and Native buckets for a GaugeHistogram follow all the same rules as for a Histogram, with Gcount and Gsum playing the same role as Count and Sum.
The exemplars for a GaugeHistogram follow all the same rules as for a Histogram.
Summary
Summaries also measure distributions of discrete events and MAY be used when Histograms are too expensive and a small number of precomputed quantiles is sufficient.
Summaries SHOULD NOT be used, because quantiles are not aggregatable and the user often can not deduce what timeframe they cover. They MAY be used for backwards compatibility, because some existing instrumentation libraries expose precomputed quantiles and do not support Histograms.
A Summary Sample MUST contain a Count, Sum and a set of quantiles.
Semantically, Count is a counter so MUST NOT be NaN or negative. Count MUST be an integer. Sum MUST NOT be NaN. Sum is only a counter semantically as long as there are no negative event values measured by the Summary.
A Summary SHOULD have a Start Timestamp.
Start Timestamp MUST NOT be based on the collection period of quantile values.
Quantiles are a map from a quantile to a value. An example is a quantile 0.95 with value 0.2 in a metric called 
myapp_http_request_duration_seconds which means that the 95th percentile latency is 200ms over an unknown timeframe. If there are no events in the relevant timeframe, the value for a quantile MUST be NaN. A Quantile's Metric's LabelSet MUST NOT have "quantile" label name. Quantiles MUST be between 0 and 1 inclusive. Quantile values SHOULD represent the recent values. Commonly this would be over the last 5-10 minutes.
Unknown
Unknown SHOULD NOT be used. Unknown MAY be used when it is impossible to determine the types of individual metrics from 3rd party systems.
A Sample in a metric with the Unknown Type MUST have a Number or CompositeValue value.
Text Format
The OpenMetrics formats are Regular Chomsky Grammars, making writing quick and small parsers possible.
Partial or invalid expositions MUST be considered erroneous in their entirety.
NOTE: Previous versions of OpenMetrics used to specify a OpenMetric protobuf format. OpenMetrics 2.0 does not include the protobuf representation. For available formats, including the official Prometheus protobuf wire format, see exposition formats documentation.
Protocol Negotiation
All ingestor implementations MUST be able to ingest data secured with TLS 1.2 or later, and SHOULD support TLS 1.3 or later. All exposers SHOULD be able to emit data secured with TLS 1.3 or later. Ingestor implementations SHOULD be able to ingest data from HTTP without TLS. All implementations SHOULD use TLS to transmit data.
Negotiation of what version of the OpenMetrics format to use is out-of-band. For example for pull-based exposition over HTTP standard HTTP content type negotiation is used, and MUST default to the oldest version of the standard (i.e. 1.0.0) if no newer version is requested.
Push-based negotiation is inherently more complex, as the exposer typically initiates the connection. Producers MUST use the oldest version of the standard (i.e. 1.0.0) unless requested otherwise by the ingestor.
ABNF
ABNF as per RFC 7405.
RFC 7405 is built on RFC 5234, but adds explicit case-sensitivity notation for string literals. The literal 
%s"text" means 
text is case-sensitive, and 
%i"text" means case-insensitive.
"exposition" is the top level token of the ABNF.
exposition = metricset HASH SP %s"EOF" [ LF ]
metricset = *metricfamily
metricfamily = *metric-descriptor *sample
metric-descriptor = HASH SP %s"TYPE" SP (metricname / metricname-utf8) SP metric-type LF
metric-descriptor =/ HASH SP %s"HELP" SP (metricname / metricname-utf8) SP escaped-string LF
metric-descriptor =/ HASH SP %s"UNIT" SP (metricname / metricname-utf8) SP *metricname-char LF
metric-type = %s"counter" / %s"gauge" / %s"histogram" / %s"gaugehistogram" / %s"stateset"
metric-type =/ %s"info" / %s"summary" / %s"unknown"
sample = metricname-and-labels SP value [SP timestamp] [SP start-timestamp] *exemplar LF
value = number / "{" composite-value "}"
timestamp = realnumber
; Lowercase st @ timestamp
start-timestamp = %s"st" "@" timestamp
exemplar = SP HASH SP labels-in-braces SP number SP timestamp
metricname-and-labels = metricname [labels-in-braces] / name-and-labels-in-braces
labels-in-braces = "{" [label *(COMMA label)] "}"
name-and-labels-in-braces = "{" metricname-utf8 *(COMMA label) "}"
label = label-key "=" DQUOTE escaped-string DQUOTE
; Number value
number = realnumber
; Case insensitive
number =/ [SIGN] (%i"inf" / %i"infinity")
number =/ %i"nan"
; Real floats
; Leading 0s explicitly okay
realnumber = [SIGN] 1*DIGIT ["." *DIGIT] [ "e" [SIGN] 1*DIGIT ]
realnumber =/ [SIGN] *DIGIT "." 1*DIGIT [ "e" [SIGN] 1*DIGIT ]
; Integers
; Leading 0s explicitly okay
integer = [SIGN] 1*"0" / [SIGN] positive-integer
non-negative-integer = ["+"] 1*"0" / ["+"] positive-integer
positive-integer = *"0" positive-digit *DIGIT
positive-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9"
BS = "\"
COMMA = ","
HASH = "#"
SIGN = "-" / "+"
metricname = metricname-initial-char 0*metricname-char
metricname-char = metricname-initial-char / DIGIT
metricname-initial-char = ALPHA / "_" / ":"
metricname-utf8 = DQUOTE escaped-string-non-empty DQUOTE
label-key = label-name / DQUOTE escaped-string-non-empty DQUOTE
label-name = label-name-initial-char *label-name-char
label-name-char = label-name-initial-char / DIGIT
label-name-initial-char = ALPHA / "_"
escaped-string = *escaped-char
escaped-string-non-empty = 1*escaped-char
escaped-char = normal-char
escaped-char =/ BS ("n" / DQUOTE / BS)
escaped-char =/ BS normal-char
; Any unicode character, except newline, double quote, and backslash
normal-char = %x00-09 / %x0B-21 / %x23-5B / %x5D-D7FF / %xE000-10FFFF
; Composite values
composite-value = histogram-value / gauge-histogram-value / summary-value
; Histograms
histogram-value = h-count "," h-sum "," histogram-buckets
gauge-histogram-value = %s"g" h-count "," %s"g" h-sum "," histogram-buckets
; count:x
h-count = %s"count" ":" number
; sum:f allows real numbers and +-Inf and NaN
h-sum = %s"sum" ":" number
histogram-buckets = classic-buckets / native-buckets [ "," classic-buckets ]
; bucket:[...,+Inf:v] The +Inf bucket is required.
classic-buckets = %s"bucket" ":" "[" [ ch-le-counts "," ] ch-pos-inf-bucket "]"
ch-le-counts = (ch-neg-inf-bucket / ch-le-bucket) *("," ch-le-bucket)
ch-pos-inf-bucket = "+" %s"Inf" ":" number
ch-neg-inf-bucket = "-" %s"Inf" ":" number
ch-le-bucket = realnumber ":" number
; schema:3,zero_threshold:1e-128,zero_count:2,negative_spans:[1:1],negative_buckets:[2],positive_spanes:[-3:1,2:2],positive_buckets:[3,1,0]
native-buckets = nh-schema "," nh-zero-threshold "," nh-zero-count [ "," nh-negative-spans "," nh-negative-buckets ] [ "," nh-positive-spans "," nh-positive-buckets ]
; schema:i
nh-schema = %s"schema" ":" integer
; zero_threshold:f
nh-zero-threshold = %s"zero_threshold" ":" realnumber
; zero_count:x
nh-zero-count = %s"zero_count" ":" number
; negative_spans:[1:2,3:4] and positive_spans:[-3:1,2:2]
nh-negative-spans = %s"negative_spans" ":" "[" [nh-spans] "]"
nh-positive-spans = %s"positive_spans" ":" "[" [nh-spans] "]"
; Spans hold offset and length. The offset can start from any index, even
; negative, however subsequent spans can only advance the index, not decrease it.
nh-spans = nh-start-span *("," nh-span)
nh-start-span = integer ":" non-negative-integer
nh-span = non-negative-integer ":" non-negative-integer
; negative_buckets:[1,2,3] and positive_buckets:[1,2,3]
nh-negative-buckets = %s"negative_buckets" ":" "[" [nh-buckets] "]"
nh-positive-buckets = %s"positive_buckets" ":" "[" [nh-buckets] "]"
nh-buckets = number *("," number)
; Summary
; count:12.0,sum:100.0,quantile:[0.9:2.0,0.95:3.0,0.99:20.0]
summary-value = cs-count "," cs-sum "," cs-quantile
; count:x where x is a number
cs-count = %s"count" ":" number
; sum:x where x is a real number or +-Inf or NaN
cs-sum = %s"sum" ":" number
; quantile:[...]
cs-quantile = %s"quantile" ":" "[" [ cs-q-counts ] "]"
cs-q-counts = cs-q-count *("," cs-q-count)
cs-q-count = realnumber ":" number
Overall Structure
UTF-8 MUST be used. Byte order markers (BOMs) MUST NOT be used. Note that NULL (byte 0x00) is a valid UTF-8 byte, while byte 0xFF, for example, is not.
The content type MUST be:
application/openmetrics-text; version=2.0.0; charset=utf-8
Line endings MUST be signalled with line feed (\n) and MUST NOT contain carriage returns (\r). Expositions MUST end with 
EOF and SHOULD end with 
EOF\n.
An example of a complete exposition:
# TYPE acme_http_router_request_seconds summary
# UNIT acme_http_router_request_seconds seconds
# HELP acme_http_router_request_seconds Latency though all of ACME's HTTP request router.
acme_http_router_request_seconds{path="/api/v1",method="GET"} {count:807283,sum:9036.32,quantile:[0.95:2,0.99:20]} st@1605281325.0
acme_http_router_request_seconds{path="/api/v2",method="GET"} {count:34,sum:479.3,quantile:[0.95:2.5,0.99:2.9]} st@1605301325.0
# TYPE go_goroutines gauge
# HELP go_goroutines Number of goroutines that currently exist.
go_goroutines 69
# TYPE process_cpu_seconds_total counter
# UNIT process_cpu_seconds_total seconds
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
process_cpu_seconds_total 4.20072246e+06
# TYPE acme_http_request_seconds histogram
# UNIT acme_http_request_seconds seconds
# HELP acme_http_request_seconds Latency histogram of all of ACME's HTTP requests.
acme_http_request_seconds{path="/api/v1",method="GET"} {count:2,sum:1.2e2,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[1:2],positive_buckets:[1,1],bucket:[0.5:1,1:2,+Inf:2]} st@1605301325.0
# TYPE acme_http_request_seconds:rate5m gaugehistogram
acme_http_request_seconds:rate5m{path="/api/v1",method="GET"} {gcount:0.01,gsum:2.0,schema:0,zero_threshold:1e-4,zero_count:0.0,positive_spans:[1:2],positive_buckets:[0.005,0.005]}
# TYPE "foodb.read.errors" counter
# HELP "foodb.read.errors" The number of errors in the read path for fooDb.
{"foodb.read.errors","service.name"="my_service"} 3482
# EOF
UTF-8 Quoting
Metric names not conforming to the ABNF definition of 
metricname MUST be enclosed in double quotes and the alternative UTF-8 syntax MUST be used. In these Metrics, the quoted metric name MUST be moved inside the brackets as the first item without a label name and equal sign, in accordance with the ABNF. The metric names MUST be enclosed in double quotes in TYPE, UNIT, and HELP lines. Quoting and the alternative metric syntax MAY be used for any metric name, regardless of whether the name requires quoting or not.
Label names not conforming to the 
label-name ABNF definition MUST be enclosed in double quotes. Any label name MAY be enclosed in double quotes.
Expressed as regular expressions, metric names that don't need to be enclosed in quotes match: 
^[a-zA-Z_:][a-zA-Z0-9_:]*$. For label names, the string matches: 
^[a-zA-Z_][a-zA-Z0-9_]*$.
Complete example:
# TYPE "process.cpu.seconds" counter
# UNIT "process.cpu.seconds" seconds
# HELP "process.cpu.seconds" Total user and system CPU time spent in seconds.
{"process.cpu.seconds","node.name"="my_node"} 4.20072246e+06
# TYPE "quoting_example" gauge
# HELP "quoting_example" Number of goroutines that currently exist.
{"quoting_example","foo"="bar"} 4.5
# EOF
Escaping
Where the ABNF notes escaping, the following escaping MUST be applied:
Line feed, 
\n (0x0A) -> literally 
\n (Bytecode 0x5c 0x6e)
Double quotes -> 
\" (Bytecode 0x5c 0x22)
Backslash -> 
\\ (Bytecode 0x5c 0x5c)
A double backslash SHOULD be used to represent a backslash character. A single backslash SHOULD NOT be used for undefined escape sequences. As an example, 
\\a is equivalent and preferable to 
\a.
Escaping MUST also be applied to quoted UTF-8 strings.
Numbers
Integer numbers MUST NOT have a decimal point. Examples are 
23, 
0042, and 
1341298465647914.
Floating point numbers MUST be represented either with a decimal point or using scientific notation. Examples are 
8903.123421 and 
1.89e-7. Floating point numbers MUST fit within the range of a 64-bit floating point value as defined by IEEE 754, but MAY require so many bits in the mantissa that results in lost precision. This MAY be used to encode nanosecond resolution timestamps.
CompositeValues
CompositeValue is represented as structured data with fields. There MUST NOT be any whitespace around fields. See the ABNF for exact details about the format and possible values.
Timestamps
Timestamps SHOULD NOT use exponential float rendering for timestamps if nanosecond precision is needed as rendering of a float64 does not have sufficient precision, e.g. 
1604676851.123456789.
Exemplars
Exemplars without Labels MUST represent an empty LabelSet as {}.
MetricFamily
There MUST NOT be an explicit separator between MetricFamilies. The next MetricFamily MUST be signalled with either metadata or a new Metric name for a new MetricFamily.
MetricFamilies MUST NOT be interleaved.
The same MetricFamily's Name and Metric's Name SHOULD have the same quoting.
An example that would violate this:
# TYPE "read_errors" counter
# HELP read_errors The number of errors in the read path for fooDb.
{"read_errors","service.name"="my_service"} 3482
read_errors{"service.name"="my_service2"} 123
MetricFamily metadata
There are four pieces of metadata: The MetricFamily name, TYPE, UNIT and HELP. An example of the metadata for a counter Metric called 
foo_total is:
# TYPE foo_total counter
If no TYPE is exposed, the MetricFamily MUST be interpreted as Type Unknown.
If a unit is specified it MUST be provided in a UNIT metadata line. In addition, an underscore and the unit SHOULD be the suffix (or infix before 
_total for Counters) of the MetricFamily name.
Be aware that exposing metrics without the unit being a suffix (or infix) of the MetricFamily name directly to end-users may reduce the usability due to confusion about what the metric's unit is.
A valid example for a 
foo_seconds_total metric with a unit of "seconds":
# TYPE foo_seconds_total counter
# UNIT foo_seconds_total seconds
A valid, but discouraged example, where the unit is not a suffix nor infix on the name:
# TYPE foo_total counter
# UNIT foo_total seconds
It is also valid to have:
# TYPE foo_seconds_total counter
If the unit is known it SHOULD be provided.
A UNIT or HELP metadata line MAY have an empty value string before the newline. This MUST be treated as if the metadata line were not present.
Full example:
# TYPE foo_seconds_total counter
# UNIT foo_seconds_total seconds
# HELP foo_seconds_total Some text and \n some \" escaping
See the UTF-8 Quoting section for circumstances where the metric name MUST be enclosed in double quotes.
There MUST NOT be more than one of each type of metadata line for a MetricFamily. The ordering SHOULD be TYPE, UNIT, HELP.
Aside from this metadata and the EOF line at the end of the message, you MUST NOT expose lines beginning with a #.
Unknown metadata
Ingestors MUST support Metric Families without metadata lines.
A valid, but discouraged example, for 
foo_seconds_total counter and a set of unrelated, unknown type metrics without metadata lines:
# TYPE foo_seconds_total counter
foo_seconds_total 1
foo_milliseconds_total 2
foo_count 3
Metric
Metrics MUST NOT be interleaved. See the StateSet example below.
A Sample without labels or a timestamp and the value 0 MUST be rendered either like:
bar_seconds_count 0
or like
bar_seconds_count{} 0
Label values MAY be any valid UTF-8 value, so escaping MUST be applied as per the ABNF. A valid example with two labels:
bar_seconds_count{a="x",b="escaping\" example \n "} 0
Metric names and label names MAY also be any valid UTF-8 value, and under certain circumstances they MUST be quoted and escaped per the ABNF. See the UTF-8 Quoting section for specifics.
{"\"bar\".seconds.count","b\\"="escaping\" example \n "} 0
Metric types
Gauge
The Sample's value MUST be a Number.
There are no recommended suffixes for the MetricFamily name for a MetricFamily of Type Gauge.
An example MetricFamily with a Metric with no labels and a Sample with no timestamp:
# TYPE foo gauge
foo 17.0
An example of a MetricFamily with two Metrics with a label and Samples with no timestamp:
# TYPE foo gauge
foo{a="bb"} 17.0
foo{a="ccc"} 17.0
An example of a MetricFamily with no Metrics:
# TYPE foo gauge
An example with a Metric with a label and a Sample with a timestamp:
# TYPE foo gauge
foo{a="b"} 17.0 1520879607.789
An example with a Metric with no labels and Sample with a timestamp:
# TYPE foo gauge
foo 17.0 1520879607.789
An example with a Metric with no labels and two Samples with timestamps:
# TYPE foo gauge
foo 17.0 123
foo 18.0 456
Counter
The Sample's value MUST be a Number.
If present, the Sample's Start Timestamp MUST be inlined with the Sample with a 
st@ prefix. If the value's timestamp is present, the Start Timestamp MUST be added right after it. If exemplar is present, the Start Timestamp MUST be added before it.
An example with a Metric with no labels, and a Sample with no timestamp and no Start Timestamp:
# TYPE foo_total counter
foo_total 17.0
An example with a Metric with no labels, and a Sample with a timestamp and no Start Timestamp:
# TYPE foo_total counter
foo_total 17.0 1520879607.789
An example with a Metric with no labels, and a Sample with no timestamp and a Start Timestamp:
# TYPE foo_total counter
foo_total 17.0 st@1520430000.123
An example with a Metric with no labels, and a Sample with a timestamp and a Start Timestamp:
# TYPE foo_total counter
foo_total 17.0 1520879607.789 st@1520430000.123
An example with a Metric with no labels, and without the 
_total suffix and a Sample with a Timestamp and a Start Timestamp:
# TYPE foo counter
foo 17.0 1520879607.789 st@1520879607.789
Be aware that exposing metrics without 
_total being a suffix of the MetricFamily name directly to end-users may reduce the usability due to confusion about what the metric's type is.
The Sample MAY have Exemplars.
An example with a Metric with no labels, and a Sample with a timestamp and a Start Timestamp and an exemplar:
# TYPE foo_total counter
foo_total 17.0 1520879607.789 st@1520430000.123 # {trace_id="KOO5S4vxi0o"} 0.67 1520879606.1
StateSet
There are no recommended suffixes for the MetricFamily name for a MetricFamily of Type StateSet.
StateSets MUST have one Metric per state in the StateSet MetricGroup. Each state's Metric MUST have a label with the MetricFamily name as the label name and the state name as the label value. The Metric Sample's Value MUST be 1 if the state is true and MUST be 0 if the state is false.
An example with the states "a", "bb", and "ccc" in which only the value bb is enabled and the metric name is foo:
# TYPE foo stateset
foo{foo="a"} 0
foo{foo="bb"} 1
foo{foo="ccc"} 0
An example of an "entity" label on the Metric:
# TYPE foo stateset
foo{entity="controller",foo="a"} 1.0
foo{entity="controller",foo="bb"} 0.0
foo{entity="controller",foo="ccc"} 0.0
foo{entity="replica",foo="a"} 1.0
foo{entity="replica",foo="bb"} 0.0
foo{entity="replica",foo="ccc"} 1.0
StateSet MetricGroups MUST NOT be interleaved.
A correct example where there are multiple MetricGroups within a MetricFamily, and multiple Metrics within each MetricGroup, and multiple Samples within each Metric:
# TYPE foo stateset
foo{entity="controller",foo="a"} 1.0 1000000000.000
foo{entity="controller",foo="a"} 0.0 1000000001.000
foo{entity="controller",foo="bb"} 0.0 1000000000.000
foo{entity="controller",foo="bb"} 1.0 1000000001.000
foo{entity="controller",foo="ccc"} 0.0 1000000000.000
foo{entity="controller",foo="ccc"} 0.0 1000000001.000
foo{entity="replica",foo="a"} 1.0 1000000000.000
foo{entity="replica",foo="a"} 1.0 1000000001.000
foo{entity="replica",foo="bb"} 0.0 1000000000.000
foo{entity="replica",foo="bb"} 1.0 1000000001.000
foo{entity="replica",foo="ccc"} 0.0 1000000000.000
foo{entity="replica",foo="ccc"} 0.0 1000000001.000
An incorrect example where MetricGroups are interleaved:
# TYPE foo stateset
foo{entity="controller",env="dev",foo="a"} 1.0
foo{entity="controller",env="dev",foo="bb"} 0.0
foo{entity="controller",env="dev",foo="ccc"} 0.0
foo{entity="replica",env="dev",foo="a"} 1.0
foo{entity="replica",env="dev",foo="bb"} 0.0
foo{entity="replica",env="dev",foo="ccc"} 1.0
foo{entity="controller",env="prod",foo="a"} 1.0
foo{entity="controller",env="prod",foo="bb"} 0.0
foo{entity="controller",env="prod",foo="ccc"} 0.0
An incorrect example where Metrics are interleaved:
# TYPE foo_seconds summary
# UNIT foo_seconds seconds
# TYPE foo stateset
foo{entity="controller",env="dev",foo="a"} 1.0
foo{entity="controller",env="dev",foo="bb"} 0.0
foo{entity="controller",env="prod",foo="a"} 1.0
foo{entity="controller",env="dev",foo="ccc"} 0.0
foo{entity="controller",env="prod",foo="bb"} 0.0
foo{entity="controller",env="prod",foo="ccc"} 0.0
Info
The Sample value MUST always be 1.
An example of a Metric with no labels, and one Sample value with "name" and "version" labels:
# TYPE foo_info info
foo_info{name="pretty name",version="8.2.7"} 1
An example of a Metric with label "entity" and one Sample value with “name” and “version” labels:
# TYPE foo_info info
foo_info{entity="controller",name="pretty name",version="8.2.7"} 1.0
foo_info{entity="replica",name="prettier name",version="8.1.9"} 1.0
Metric labels and Sample value labels MAY be in any order.
Summary
The Sample's value MUST be a CompositeValue.
The CompositeValue MUST include the Count, Sum and quantile values as the fields 
count, 
sum, 
quantile, in this order.
If present the Sample's Start Timestamp MUST be inlined with the Sample with a 
st@ prefix. If the value's timestamp is present, the Start Timestamp MUST be added right after it. If exemplars are present, the Start Timestamp MUST be added before it.
The quantiles MUST be sorted in increasing order of the quantile.
An example of a Metric with no labels and a Sample with Sum, Count and Start Timestamp:
# TYPE foo summary
foo {count:17,sum:324789.3,quantile:[]} st@1520430000.123
An example of a Metric with no labels and a Sample with two quantiles and Start Timestamp:
# TYPE foo summary
foo {count:0,sum:0.0,quantile:[0.95:123.7,0.99:150]} st@1520430000.123
Histogram with Classic Buckets
The Sample's value MUST be a CompositeValue.
The CompositeValue MUST include the Count, Sum and Classic Bucket values as the fields 
count, 
sum, 
bucket, in this order.
If present the Sample's Start Timestamp MUST be inlined with the Sample with a 
st@ prefix. If the value's timestamp is present, the Start Timestamp MUST be added right after it. If exemplars are present, the Start Timestamp MUST be added before it.
Classic Buckets MUST be sorted in number increasing order of their threshold.
All Classic Buckets MUST be present, even ones with the value 0.
An example of a Metric with no labels and a Sample with Sum, Count, and Start Timestamp values, and with 12 Classic Buckets. A wide and atypical but valid variety of bucket threshold values is shown on purpose:
# TYPE foo histogram
foo {count:17,sum:324789.3,bucket:[0.0:0,1e-05:0,0.0001:5,0.1:8,1.0:10,10.0:11,100000.0:11,1e+06:15,1e+23:16,1.1e+23:17,+Inf:17]} st@1520430000.123
Histogram with Native Buckets
The Sample's value MUST be a CompositeValue.
The CompositeValue MUST include the Count, Sum, Schema, Zero Threshold, Zero Native Bucket value as the fields 
count, 
sum, 
schema, 
zero_threshold, 
zero_count, in this order.
If there are no negative Native Buckets, then the fields 
negative_spans and 
negative_buckets SHOULD be omitted. If there are no positive Native Buckets, then the fields 
positive_spans and 
positive_buckets SHOULD be omitted.
If there are negative (and/or positive) Native Buckets, then the fields 
negative_spans, 
negative_buckets (and/or 
positive_spans, 
positive_buckets) MUST be present in this order after the 
zero_count field.
Native Bucket values MUST be ordered by their index, and their values MUST be placed in the 
negative_buckets (and/or 
positive_buckets) fields.
NOTE: Bucket values are absolute counts, as opposed to some implementations that store bucket values as deltas relative to the preceding bucket.
Native Buckets that have a value of 0 SHOULD NOT be present.
To map the 
negative_buckets (and/or 
positive_buckets) back to their indices, the 
negative_spans (and/or 
positive_spans) field MUST be constructed in the following way: Each span consists of a pair of numbers, an integer called offset and an non-negative integer called length. Only the first span in each list can have a negative offset. It defines the index of the first bucket in its corresponding 
negative_buckets (and/or 
positive_buckets). The length defines the number of consecutive buckets the bucket list starts with. The offsets of the following spans define the number of excluded (and thus unpopulated buckets). The lengths define the number of consecutive buckets in the list following the excluded buckets.
An example of when to keep empty positive or negative Native Buckets is to reduce the number of spans needed to represent the case where the offset between two spans is just 1, meaning that with the inclusion of one empty bucket, the number of spans is reduced by one.
The sum of all length values in each span list MUST be equal to the length of the corresponding bucket list.
An example with all fields:
# TYPE acme_http_request_seconds histogram
acme_http_request_seconds{path="/api/v1",method="GET"} {count:59,sum:1.2e2,schema:7,zero_threshold:1e-4,zero_count:0,negative_spans:[1:2],negative_buckets:[5,7],positive_spans:[-1:2,3:4],positive_buckets:[5,7,10,9,8,8]} st@1520430000.123
An example without any buckets in use:
# TYPE acme_http_request_seconds histogram
acme_http_request_seconds{path="/api/v1",method="GET"} {count:0,sum:0,schema:3,zero_threshold:1e-4,zero_count:0} st@1520430000.123
Histogram with both Classic and Native Buckets
The Histogram Sample's value MUST be a CompositeValue.
The CompositeValue MUST include the Count and Sum as the fields 
count, 
sum, in this order.
After the 
count and 
sum, the remaining fields of the Native Buckets MUST be included, then the remaining fields of the Classic Buckets (i.e. the 
bucket field) MUST be included.
The order ensures that implementations can easily skip the Classic Buckets if the Native Buckets are preferred.
# TYPE acme_http_request_seconds histogram
# UNIT acme_http_request_seconds seconds
# HELP acme_http_request_seconds Latency histogram of all of ACME's HTTP requests.
acme_http_request_seconds{path="/api/v1",method="GET"} {count:2,sum:1.2e2,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[1:2],positive_buckets:[1,1],bucket:[0.5:1,1:2,+Inf:2]}
Exemplars and Start Timestamp
Exemplars MAY be attached to the Histogram Sample.
If the exposer is keeping a separate set of exemplars for Classic and Native Buckets, then the exposer MAY attach only one set for performance and backwards compatibility reasons, and that set SHOULD be the exemplars associated with Classic Buckets.
If present, the Sample's Start Timestamp MUST be inlined with the Sample with a 
st@ prefix. If the value's timestamp is present, the Start Timestamp MUST be added right after it. If exemplars are present, the Start Timestamp MUST be added before it.
An example of a Histogram with Native Buckets and Start Timestamp that has multiple Exemplars:
# TYPE foo histogram
foo {count:17,sum:324789.3,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[0:2],positive_buckets:[5,12]} st@1520430000.123 # {trace_id="shaZ8oxi"} 0.67 1520879607.789 # {trace_id="ookahn0M"} 1.2 1520879608.589
An example of a Histogram with Classic Buckets, and Start Timestamp where no exemplar falls within the "0.01" bucket and the "+Inf" bucket. An exemplar without Labels falls within the "0.1" bucket. An exemplar with one Label falls within the "1" bucket and another in the "10" bucket.
# TYPE foo histogram
foo {count:17,sum:324789.3,bucket:[0.01:0,0.1:8,1.0:11,10.0:17,+Inf:17]} st@1520430000.123 # {} 0.054 1520879607.7 # {trace_id="KOO5S4vxi0o"} 1.67 1520879602.890 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
An example of a Histogram with both Classic and Native Buckets and Start Timestamp.
# TYPE foo histogram
foo {count:17,sum:324789.3,schema:0,zero_threshold:1e-4,zero_count:0,positive_spans:[0:2],positive_buckets:[5,12],bucket:[0.01:0,0.1:8,1.0:11,10.0:17,+Inf:17]} st@1520430000.123 # {} 0.054 1520879607.7 # {trace_id="KOO5S4vxi0o"} 1.67 1520879602.890 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789
GaugeHistogram with Classic Buckets
GaugeHistogram Samples with Classic Buckets follow the same syntax as Histogram Samples with Classic Buckets, except that the Count and Sum are exposed as the fields 
gcount and 
gsum and GaugeHistograms do not have Start Timestamp.
An example of a Metric with no labels, and one Sample value with no Timestamp, and no Exemplars:
# TYPE foo gaugehistogram
foo {gcount:42,gsum:3289.3,bucket:[0.01:20,0.1:25,1:34,+Inf:42]}
GaugeHistogram with Native Buckets
GaugeHistogram Samples with Native Buckets follow the same syntax as Histogram Samples with Native Buckets, except that the Count and Sum are exposed as the fields 
gcount and 
gsum and GaugeHistograms do not have Start Timestamp.
An example of a Metric with no labels, and one Sample value with no Timestamp, and no Exemplars:
# TYPE acme_http_request_seconds gaugehistogram
acme_http_request_seconds{path="/api/v1",method="GET"} {gcount:59,gsum:1.2e2,schema:7,zero_threshold:1e-4,zero_count:0,negative_spans:[1:2],negative_buckets:[5,7],positive_spans:[-1:2,3:4],positive_buckets:[5,7,10,9,8,8]}
GaugeHistogram with both Classic and Native buckets
GaugeHistogram Samples with both Classic and Native Buckets follow the same syntax as Histogram Samples with both Classic and Native Buckets, except that the Count and Sum are exposed as the fields 
gcount and 
gsum and GaugeHistograms do not have Start Timestamp.
Unknown
The Sample's value MUST be a Number or a CompositeValue.
There are no recommended suffixes for the MetricFamily name for a MetricFamily of Type Unknown.
An example with a Metric with no labels and a Sample with no Timestamp:
# TYPE foo unknown
foo 42.23
An example with a Metric without MetricFamily metadata and a Sample with no Timestamp:
foo 42.23
Design Considerations
Scope
OpenMetrics is intended to provide telemetry for online systems. It runs over protocols which do not provide hard or soft real time guarantees, so it can not make any real time guarantees itself. Latency and jitter properties of OpenMetrics are as imprecise as the underlying network, operating systems, CPUs, and the like. It is sufficiently accurate for aggregations to be used as a basis for decision-making, but not to reflect individual events.
Systems of all sizes should be supported, from applications that receive a few requests an hour up to monitoring bandwidth usage on a 400Gb network port. Aggregation and analysis of transmitted telemetry should be possible over arbitrary time periods.
It is intended to transport snapshots of state at the time of data transmission at a regular cadence.
Out of scope
How ingestors discover which exposers exist, and vice-versa, is out of scope for and thus not defined in this standard.
Failure Modes
This specification advocates for transactional processing: any encoding, decoding, or validation errors must reject the whole MetricSet ingestion. A failed scrape is better than an inaccurate scrape or a partial metric view that breaks transactionality (e.g., scraping a portion of a StateSet MetricGroup, or scraping only one Counter out of two that are aggregated in a single alert expression).
There's one exception to this rule: failures specific to exemplars should not cause the entire exposition to fail. If an exemplar is malformed or invalid, it should be dropped or ignored, allowing the valid metric data to be ingested.
Extensions and Improvements
This second version of OpenMetrics is based upon the well-established de facto standard Prometheus exposition formats such as the Prometheus text format 0.0.4, Prometheus Protobuf format, and OpenMetrics 1.0.
This version introduces major changes to the first version to improve reliability, performance, compatibility with the Prometheus Protobuf format and the OpenTelemetry data model and naming conventions. At the same time, the format retains the ability to expose telemetry in a simple way and to be human-readable. This format is close enough to the previous version, the Prometheus query language, and the data model so as to ease the transition.
It also ensures that there is a basic standard which is easy to implement. This can be built upon in future versions of the standard. The intention is that future minor versions of the standard will always require support for this 2.0 version, both syntactically and semantically.
We want to allow monitoring systems to get usable information from an OpenMetrics exposition without undue burden. If one were to strip away all metadata and structure and just look at an OpenMetrics exposition as an unordered set of samples, it should be usable on its own.
This principle is applied consistently throughout the standard. For example, it is encouraged that a MetricFamily's unit is duplicated in the name so that the unit is available for systems that don't understand the unit metadata. However, as opposed to the previous version, duplicating the unit name and adding the 
_total suffix for counters is not enforced anymore to foster compatibility with OpenTelemetry.
Another change from the previous version is that the Metric name is now required to strictly match its MetricFamily name. In OpenMetrics 1.0, the MetricFamily name was the Metric name without type-specific suffixes such as 
_total or 
_bucket. This change improves parser reliability by making the association between a Metric and its MetricFamily unambiguous.
Each line exposed via this format is self-contained in the sense that the information derived from it is complete and can be put into storage in a meaningful way. This is achieved by the introduction of composite types and moving the Start Timestamp (formerly Created value) in-line. These are major changes from the first version, made necessary by the introduction of native histograms in Prometheus and the performance of parsing the 
_created lines in the previous version.
A summary of notable changes from OpenMetrics 1.0:
Start Timestamp moved inline with 
st@ notation, replacing separate 
_created samples.
Metric name must exactly match its MetricFamily name; implicit suffix stripping is removed.
_total suffix for Counters and unit suffixes changed from MUST to SHOULD.
Histogram, GaugeHistogram, and Summary values consolidated into a single CompositeValue line; Count and Sum are now required.
Native Histograms with exponential bucket schemas introduced.
UTF-8 metric and label names allowed with quoting.
Exemplar timestamps are now mandatory; multiple exemplars per sample are allowed.
Protobuf format specification removed.
Units and Base Units
For consistency across systems and to avoid confusion, units are largely based on SI base units. Base units include seconds, bytes, joules, grams, meters, ratios, volts, amperes, and celsius. Units should be provided where they are applicable.
For example, having all duration metrics in seconds, there is no risk of having to guess whether a given metric is nanoseconds, microseconds, milliseconds, seconds, minutes, hours, days or weeks nor having to deal with mixed units. By choosing unprefixed units, we avoid situations like ones in which kilomilliseconds were the result of emergent behaviour of complex systems.
As values can be floating point, sub-base-unit precision is built into the standard.
Similarly, mixing bits and bytes is confusing, so bytes are chosen as the base. While Kelvin is a better base unit in theory, in practice most existing hardware exposes Celsius. Kilograms are the SI base unit, however the kilo prefix is problematic so grams are chosen as the base unit.
While base units SHOULD be used in all possible cases, Kelvin is a well-established unit which MAY be used instead of Celsius for use cases such as color or black body temperatures where a comparison between a Celsius and Kelvin metric are unlikely.
Ratios are the base unit, not percentages. Where possible, raw data in the form of gauges or counters for the given numerator and denominator should be exposed. This has better mathematical properties for analysis and aggregation in the ingestors.
Decibels are not a base unit as firstly, deci is a SI prefix and secondly, bels are logarithmic. To expose signal/energy/power ratios exposing the ratio directly would be better, or better still the raw power/energy if possible. Floating point exponents are more than sufficient to cover even extreme scientific uses. An electron volt (~1e-19 J) all the way up to the energy emitted by a supernova (~1e44 J) is 63 orders of magnitude, and a 64-bit floating point number can cover over 2000 orders of magnitude.
If non-base units can not be avoided and conversion is not feasible, the actual unit should still be included in the metric name for clarity. For example, joule is the base unit for both energy and power, as watts can be expressed as a counter with a joule unit. In practice a given 3rd party system may only expose watts, so a gauge expressed in watts would be the only realistic choice in that case.
Not all MetricFamilies have units. For example a count of HTTP requests wouldn't have a unit. Technically the unit would be HTTP requests, but in that sense the entire MetricFamily name is the unit. Going to that extreme would not be useful. The possibility of having good axes on graphs in downstream systems for human consumption should always be kept in mind.
Statelessness
The wire format defined by OpenMetrics is stateless across expositions. What information has been exposed before MUST have no impact on future expositions. Each exposition is a self-contained snapshot of the current state of the exposer.
The same self-contained exposition MUST be provided to existing and new ingestors.
A core design choice is that exposers MUST NOT exclude a metric merely because it has had no recent changes, or observations. An exposer must not make any assumptions about how often ingestors are consuming expositions.
Exposition Across Time and Metric Evolution
Metrics are most useful when their evolution over time can be analysed, so accordingly expositions must make sense over time. Thus, it is not sufficient for one single exposition on its own to be useful and valid. Some changes to metric semantics can also break downstream users.
Parsers commonly optimize by caching previous results. Thus, changing the order in which labels are exposed across expositions SHOULD be avoided even though it is technically not breaking This also tends to make writing unit tests for exposition easier.
Metrics and samples SHOULD NOT appear and disappear from exposition to exposition, for example a counter is only useful if it has history. In principle, a given Metric should be present in exposition from when the process starts until the process terminates. It is often not possible to know in advance what Metrics a MetricFamily will have over the lifetime of a given process (e.g. a label value of a latency histogram is a HTTP path, which is provided by an end user at runtime), but once a counter-like Metric is exposed it should continue to be exposed until the process terminates. That a counter is not getting increments doesn't invalidate that it still has its current value. There are cases where it may make sense to stop exposing a given Metric; see the section on Missing Data.
In general changing a MetricFamily's Type, or adding or removing a label from its Metrics will be breaking to ingestors.
A notable exception is that adding a label to an Info Metric is not breaking. This is so that you can add additional information to an existing Info MetricFamily where it makes sense to be, rather than being forced to create a brand new info metric with an additional label value. Ingestor systems should ensure that they are resilient to such additions.
Changing a MetricFamily's Help is not breaking. For values where it is possible, switching between floats and ints is not breaking. Adding a new state to a stateset is not breaking. Adding unit metadata where it doesn't change the metric name is not breaking.
Histogram buckets SHOULD NOT change from exposition to exposition, as this is likely to both cause performance issues and break ingestors and cause. Similarly all expositions from any consistent binary and environment of an application SHOULD have the same buckets for a given Histogram MetricFamily, so that they can be aggregated by all ingestors without ingestors having to implement histogram merging logic for heterogeneous buckets. An exception might be occasional manual changes to buckets which are considered breaking, but may be a valid tradeoff when performance characteristics change due to a new software release.
Even if changes are not technically breaking, they still carry a cost. For example frequent changes may cause performance issues for ingestors. A Help string that varies from exposition to exposition may cause each Help value to be stored. Frequently switching between int and float values could prevent efficient compression.
NaN
NaN is a number like any other in OpenMetrics, usually resulting from a division by zero such as for a summary quantile if there have been no observations recently. NaN does not have any special meaning in OpenMetrics, and in particular MUST NOT be used as a marker for missing or otherwise bad data.
Missing Data
There are valid cases when data stops being present. For example a filesystem can be unmounted and thus its Gauge Metric for free disk space no longer exists. There is no special marker or signal for this situation. Subsequent expositions simply do not include this Metric.
Exposition Performance
Metrics are only useful if they can be collected in reasonable time frames. Metrics that take minutes to expose are not considered useful.
As a rule of thumb, exposition SHOULD take no more than a second.
Metrics from legacy systems serialized through OpenMetrics may take longer. For this reason, no hard performance assumptions can be made.
Exposition SHOULD be of the most recent state. For example, a thread serving the exposition request SHOULD NOT rely on cached values, to the extent it is able to bypass any such caching
Concurrency
For high availability and ad-hoc access a common approach is to have multiple ingestors. To support this, concurrent expositions MUST be supported. All BCPs for concurrent systems SHOULD be followed, common pitfalls include deadlocks, race conditions, and overly-coarse grained locking preventing expositions progressing concurrently.
Metric Naming and Namespaces
We aim for a balance between understandability, avoiding clashes, and succinctness in the naming of metrics and label names. Names are separated through underscores, so metric names end up being in “snake_case”. While we strongly recommend the practices recommended in this document, other metric systems have different philosophies regarding naming conventions. OpenMetrics allows these metrics to be exposed, but without the conventions and suffixes recommended here there is an increased risk of collisions and incompatibilities along the chain of services in a metrics system. Users wishing to use alternative conventions will need to take special care and expend additional effort to ensure that the entire system is consistent.
To take an example "http_request_seconds" is succinct but would clash between large numbers of applications, and it's also unclear exactly what this metric is measuring. For example, it might be before or after auth middleware in a complex system.
Metric names should indicate what piece of code they come from. So a company called A Company Manufacturing Everything might prefix all metrics in their code with "acme_", and if they had a HTTP router library measuring latency it might have a metric such as "acme_http_router_request_seconds" with a Help string indicating that it is the overall latency.
It is not the aim to prevent all potential clashes across all applications, as that would require heavy handed solutions such as a global registry of metric namespaces or very long namespaces based on DNS. Rather the aim is to keep to a lightweight informal approach, so that for a given application that it is very unlikely that there is clash across its constituent libraries.
Across a given deployment of a monitoring system as a whole the aim is that clashes where the same metric name means different things are uncommon. For example acme_http_router_request_seconds might end up in hundreds of different applications developed by A Company Manufacturing Everything, which is normal. If Another Corporation Making Entities also used the metric name acme_http_router_request_seconds in their HTTP router that's also fine. If applications from both companies were being monitored by the same monitoring system the clash is undesirable, but acceptable as no application is trying to expose both names and no one target is trying to (incorrectly) expose the same metric name twice. If an application wished to contain both My Example Company's and Mega Exciting Company's HTTP router libraries that would be a problem, and one of the metric names would need to be changed somehow.
As a corollary, the more public a library is the better namespaced its metric names should be to reduce the risk of such scenarios arising. acme_ is not a bad choice for internal use within a company, but these companies might for example choose the prefixes acmeverything_ or acorpme_ for code shared outside their company.
After namespacing by company or organisation, namespacing and naming should continue by library/subsystem/application fractally as needed such as the http_router library above. The goal is that if you are familiar with the overall structure of a codebase, you could make a good guess at where the instrumentation for a given metric is given its metric name.
For a common very well known existing piece of software, the name of the software itself may be sufficiently distinguishing. For example bind_ is probably sufficient for the DNS software, even though isc_bind_ would be the more usual naming.
Metric names prefixed by scrape_ are used by ingestors to attach information related to individual expositions, so should not be exposed by applications directly. Metrics that have already been consumed and passed through a general purpose monitoring system may include such metric names on subsequent expositions. If an exposer wishes to provide information about an individual exposition, a metric prefix such as myexposer_scrape_ may be used. A common example is a gauge myexposer_scrape_duration_seconds for how long that exposition took from the exposer's standpoint.
Within the Prometheus ecosystem a set of per-process metrics has emerged that are consistent across all implementations, prefixed with process_. For example for open file ulimits the MetricFamiles process_open_fds and process_max_fds gauges provide both the current and maximum value. (These names are legacy, if such metrics were defined today they would be more likely called process_fds_open and process_fds_limit). In general it is very challengings to get names with identical semantics like this, which is why different instrumentation should use different names.
Avoid redundancy in metric names. Avoid substrings like "metric", "timer", "stats", "counter", "total", "float64" and so on - by virtue of being a metric with a given type (and possibly unit) exposed via OpenMetrics information like this is already implied so should not be included explicitly. You should not include label names of a metric in the metric name for the same reasons, and in addition subsequent aggregation of the metric by a monitoring system could make such information incorrect.
Avoid including implementation details from other layers of your monitoring system in the metric names contained in your instrumentation. For example a MetricFamily name should not contain the string "openmetrics" merely because it happens to be currently exposed via OpenMetrics somewhere, or "prometheus" merely because your current monitoring system is Prometheus.
Label Namespacing
For label names no explicit namespacing by company or library is recommended, namespacing from the metric name is sufficient for this when considered against the length increase of the label name. However some minimal care to avoid common clashes is recommended.
There are label names such as region, zone, cluster, availability_zone, az, datacenter, dc, owner, customer, stage, service, team, job, instance, environment, and env which are highly likely to clash with labels used to identify targets which a general purpose monitoring system may add. Try to avoid them, adding minimal namespacing may be appropriate in these cases.
The label name "type" is highly generic and should be avoided. For example for HTTP-related metrics "method" would be a better label name if you were distinguishing between GET, POST, and PUT requests.
While there is metadata about metric names such as HELP, TYPE and UNIT there is no metadata for label names. This is as it would be bloating the format for little gain. Out-of-band documentation is one way for exposers could present this their ingestors.
Metric Names versus Labels
There are situations in which both using multiple Metrics within a MetricFamily or multiple MetricFamilies seem to make sense. Summing or averaging a MetricFamily should be meaningful even if it's not always useful. For example, mixing voltage and fan speed is not meaningful.
As a reminder, OpenMetrics is built with the assumption that ingestors can process and perform aggregations on data.
Exposing a total sum alongside other metrics is wrong, as this would result in double-counting upon aggregation in downstream ingestors.
wrong_metric{label="a"} 1
wrong_metric{label="b"} 6
wrong_metric{label="total"} 7
Labels of a Metric should be to the minimum needed to ensure uniqueness as every extra label is one more that users need to consider when determining what Labels to work with downstream. Labels which could be applied many MetricFamilies are candidates for being moved into _info metrics similar to database {{normalization}}. If virtually all users of a Metric could be expected to want the additional label, it may be a better trade-off to add it to all MetricFamilies. For example if you had a MetricFamily relating to different SQL statements where uniqueness was provided by a label containing a hash of the full SQL statements, it would be okay to have another label with the first 500 characters of the SQL statement for human readability.
Experience has shown that downstream ingestors find it easier to work with separate total and failure MetricFamiles rather than using {result="success"} and {result="failure"} Labels within one MetricFamily. Also it is usually better to expose separate read & write and send & receive MetricFamiles as full duplex systems are common and downstream ingestors are more likely to care about those values separately than in aggregate.
All of this is not as easy as it may sound. It's an area where experience and engineering trade-offs by domain-specific experts in both exposition and the exposed system are required to find a good balance.
Metric and Label Name Characters
OpenMetrics builds on the existing widely adopted Prometheus text exposition format and the ecosystem which formed around it. Backwards compatibility is a core design goal. Expanding or contracting the set of characters that are supported by the Prometheus text format would work against that goal. Breaking backwards compatibility would have wider implications than just the wire format. In particular, the query languages created or adopted to work with data transmitted within the Prometheus ecosystem rely on these precise character sets. Label values support full UTF-8, so the format can represent multi-lingual metrics.
Types of Metadata
Metadata can come from different sources. Over the years, two main sources have emerged. While they are often functionally the same, it helps in understanding to talk about their conceptual differences.
"Target metadata" is metadata commonly external to an exposer. Common examples would be data coming from service discovery, a CMDB, or similar, like information about a datacenter region, if a service is part of a particular deployment, or production or testing. This can be achieved by either the exposer or the ingestor adding labels to all Metrics that capture this metadata. Doing this through the ingestor is preferred as it is more flexible and carries less overhead. On flexibility, the hardware maintenance team might care about which server rack a machine is located in, whereas the database team using that same machine might care that it contains replica number 2 of the production database. On overhead, hardcoding or configuring this information needs an additional distribution path.
"Exposer metadata" is coming from within an exposer. Common examples would be software version, compiler version, or Git commit SHA.
Supporting Target Metadata in both Push-based and Pull-based Systems
In push-based consumption, it is typical for the exposer to provide the relevant target metadata to the ingestor. In pull-based consumption the push-based approach could be taken, but more typically the ingestor already knows the metadata of the target a-priori such as from a machine database or service discovery system, and associates it with the metrics as it consumes the exposition.
OpenMetrics is stateless and provides the same exposition to all ingestors, which is in conflict with the push-style approach. In addition the push-style approach would break pull-style ingestors, as unwanted metadata would be exposed.
One approach would be for push-style ingestors to provide target metadata based on operator configuration out-of-band, for example as a HTTP header. While this would transport target metadata for push-style ingestors, and is not precluded by this standard, it has the disadvantage that even though pull-style ingestors should use their own target metadata, it is still often useful to have access to the metadata the exposer itself is aware of.
The preferred solution is to provide this target metadata as part of the exposition, but in a way that does not impact on the exposition as a whole. Info MetricFamilies are designed for this. An exposer may include an Info MetricFamily called "target_info" with a single Metric with no labels with the metadata. An example in the text format might be:
# TYPE target_info info
# HELP target_info Target metadata
target_info{env="prod",hostname="myhost",datacenter="sdc",region="europe",owner="frontend"} 1
When an exposer is providing this metric for this purpose it SHOULD be first in the exposition. This is for efficiency, so that ingestors relying on it for target metadata don't have to buffer up the rest of the exposition before applying business logic based on its content.
Exposers MUST NOT add target metadata labels to all Metrics from an exposition, unless explicitly configured for a specific ingestor. Exposers MUST NOT prefix MetricFamily names or otherwise vary MetricFamily names based on target metadata. Generally, the same Label should not appear on every Metric of an exposition, but there are rare cases where this can be the result of emergent behaviour. Similarly all MetricFamily names from an exposer may happen to share a prefix in very small expositions. For example an application written in the Go language by A Company Manufacturing Everything would likely include metrics with prefixes of acme_, go_, process_, and metric prefixes from any 3rd party libraries in use.
Exposers can expose exposer metadata as Info MetricFamilies.
The above discussion is in the context of individual exposers. An exposition from a general purpose monitoring system may contain metrics from many individual targets, and thus may expose multiple target_info Metrics. The metrics may already have had target metadata added to them as labels as part of ingestion. The metric names MUST NOT be varied based on target metadata. For example it would be incorrect for all metrics to end up being prefixed with staging_ even if they all originated from targets in a staging environment).
Client Calculations and Derived Metrics
Exposers should leave any math or calculation up to ingestors. A notable exception is the Summary quantile which is unfortunately required for backwards compatibility. Exposition should be of raw values which are useful over arbitrary time periods.
As an example, you should not expose a gauge with the average rate of increase of a counter over the last 5 minutes. Letting the ingestor calculate the increase over the data points they have consumed across expositions has better mathematical properties and is more resilient to scrape failures.
Another example is the average event size of a histogram/summary. Exposing the average rate of increase of a counter since an application started or since a Metric was created has the problems from the earlier example and it also prevents aggregation.
Standard deviation also falls into this category. Exposing a sum of squares as a counter would be the correct approach. It was not included in this standard as a Histogram value because 64bit floating point precision is not sufficient for this to work in practice. Due to the squaring only half the 53bit mantissa would be available in terms of precision. As an example a histogram observing 10k events per second would lose precision within 2 hours. Using 64bit integers would be no better due to the loss of the floating decimal point because a nanosecond resolution integer typically tracking events of a second in length would overflow after 19 observations. This design decision can be revisited when 128bit floating point numbers become common.
Another example is to avoid exposing a request failure ratio, exposing separate counters for failed requests and total requests instead.
Number Types
For a counter that was incremented a million times per second it would take over a century to begin to lose precision with a float64 as it has a 53 bit mantissa. Yet a 100 Gbps network interface's octet throughput precision could begin to be lost with a float64 within around 20 hours. While losing 1KB of precision over the course of years for a 100Gbps network interface is unlikely to be a problem in practice, int64s are an option for integral data with such a high throughput.
Summary quantiles must be float64, as they are estimates and thus fundamentally inaccurate.
Exposing Timestamps
One of the core assumptions of OpenMetrics is that exposers expose the most up to date snapshot of what they're exposing.
While there are limited use cases for attaching timestamps to exposed data, these are very uncommon. Data which had timestamps previously attached, in particular data which has been ingested into a general purpose monitoring system may carry timestamps. Live or raw data should not carry timestamps. It is valid to expose the same metric Sample value with the same timestamp across expositions, however it is invalid to do so if the underlying metric is now missing.
Time synchronization is a hard problem and data should be internally consistent in each system. As such, ingestors should be able to attach the current timestamp from their perspective to data rather than based on the system time of the exposer device.
With timestamped metrics it is not generally possible to detect the time when a Metric went missing across expositions. However with non-timestamped metrics the ingestor can use its own timestamp from the exposition where the Metric is no longer present.
All of this is to say that, in general, Sample timestamps should not be exposed, as it should be up to the ingestor to apply their own timestamps to samples they ingest.
Tracking When Metrics Last Changed
Presume you had a counter my_counter which was initialized, and then later incremented by 1 at time 123. This would be a correct way to expose it in the text format:
# HELP my_counter Good increment example
# TYPE my_counter counter
my_counter_total 1
As per the parent section, ingestors should be free to attach their own timestamps, so this would be incorrect:
# HELP my_counter Bad increment example
# TYPE my_counter counter
my_counter_total 1 123
In case the specific time of the last change of a counter matters, this would be the correct way:
# HELP my_counter Good increment example
# TYPE my_counter counter
my_counter_total 1
# HELP my_counter_last_increment_timestamp_seconds When my_counter was last incremented
# TYPE my_counter_last_increment_timestamp_seconds gauge
# UNIT my_counter_last_increment_timestamp_seconds seconds
my_counter_last_increment_timestamp_seconds 123
By putting the timestamp of last change into its own Gauge as a value, ingestors are free to attach their own timestamp to both Metrics.
Experience has shown that exposing absolute timestamps (epoch is considered absolute here) is more robust than time elapsed, seconds since, or similar. In either case, they would be gauges. For example:
# TYPE my_boot_time_seconds gauge
# HELP my_boot_time_seconds Boot time of the machine
# UNIT my_boot_time_seconds seconds
my_boot_time_seconds 1256060124
Is better than:
# TYPE my_time_since_boot_seconds gauge
# HELP my_time_since_boot_seconds Time elapsed since machine booted
# UNIT my_time_since_boot_seconds seconds
my_time_since_boot_seconds 123
Conversely, there are no best practice restrictions on exemplars timestamps. Keep in mind that due to race conditions or time not being perfectly synced across devices, that an exemplar timestamp may appear to be slightly in the future relative to a ingestor's system clock or other metrics from the same exposition. Similarly it is possible that a "st@" for a Sample could appear to be slightly after an exemplar or sample timestamp for that same Sample.
Keep in mind that there are monitoring systems in common use which support everything from nanosecond to second resolution, so having two Samples with the same timestamp when truncated to second resolution may cause an apparent duplicate in the ingestor. In this case the Sample with the earliest timestamp MUST be used.
Thresholds
Exposing desired bounds for a system can make sense, but proper care needs to be taken. For values which are universally true, it can make sense to emit Gauge metrics for such thresholds. For example, a data center HVAC system knows the current measurements, the setpoints, and the alert setpoints. It has a globally valid and correct view of the desired system state. As a counter example, some thresholds can change with scale, deployment model, or over time. A certain amount of CPU usage may be acceptable in one setting and undesirable in another. Aggregation of values can further change acceptable values. In such a system, exposing bounds could be counter-productive.
For example the maximum size of a queue may be exposed alongside the number of items currently in the queue like:
# HELP acme_notifications_queue_capacity The capacity of the notifications queue.
# TYPE acme_notifications_queue_capacity gauge
acme_notifications_queue_capacity 10000
# HELP acme_notifications_queue_length The number of notifications in the queue.
# TYPE acme_notifications_queue_length gauge
acme_notifications_queue_length 42
Size Limits
This standard does not prescribe any particular limits on the number of samples exposed by a single exposition, the number of labels that may be present, the number of states a stateset may have, the number of labels in an info value, or metric name/label name/label value/help character limits.
Specific limits run the risk of preventing reasonable use cases, for example while a given exposition may have an appropriate number of labels after passing through a general purpose monitoring system a few target labels may have been added that would push it over the limit. Specific limits on numbers such as these would also not capture where the real costs are for general purpose monitoring systems. These guidelines are thus both to aid exposers and ingestors in understanding what is reasonable.
On the other hand, an exposition which is too large in some dimension could cause significant performance problems compared to the benefit of the metrics exposed. Thus some guidelines on the size of any single exposition would be useful.
Ingestors may choose to impose limits themselves, for in particular to prevent attacks or outages. Still, ingestors need to consider reasonable use cases and try not to disproportionately impact them. If any single value/metric/exposition exceeds such limits then the whole exposition must be rejected.
In general there are three things which impact the performance of a general purpose monitoring system ingestion time series data: the number of unique time series, the number of samples over time in those series, and the number of unique strings such as metric names, label names, label values, and HELP. ingestors can control how often they ingest, so that aspect does not need further consideration.
The number of unique time series is roughly equivalent to the number of non-comment lines in the text format. As of 2020, 10 million time series in total is considered a large amount and is commonly the order of magnitude of the upper bound of any single-instance ingestor. Any single exposition should not go above 10k time series without due diligence. One common consideration is horizontal scaling: What happens if you scale your instance count by 1-2 orders of magnitude? Having a thousand top-of-rack switches in a single deployment would have been hard to imagine 30 years ago. If a target was a singleton (e.g. exposing metrics relating to an entire cluster) then several hundred thousand time series may be reasonable. It is not the number of unique MetricFamilies or the cardinality of individual labels/buckets/statesets that matters, it is the total order of magnitude of the time series. 1,000 gauges with one Metric each are as costly as a single gauge with 1,000 Metrics.
If all targets of a particular type are exposing the same set of time series, then each additional targets' strings poses no incremental cost to most reasonably modern monitoring systems. If however each target has unique strings, there is such a cost. As an extreme example, a single 10k character metric name used by many targets is on its own very unlikely to be a problem in practice. To the contrary, a thousand targets each exposing a unique 36 character UUID is over three times as expensive as that single 10k character metric name in terms of strings to be stored assuming modern approaches. In addition, if these strings change over time older strings will still need to be stored for at least some time, incurring extra cost. Assuming the 10 million times series from the last paragraph, 100MB of unique strings per hour might indicate a use case for then the use case may be more like event logging, not metric time series.
Security
Implementors MAY choose to offer authentication, authorization, and accounting; if they so choose, this SHOULD be handled outside of OpenMetrics.
All exposer implementations SHOULD be able to secure their HTTP traffic with TLS 1.3 or later. If an exposer implementation does not support encryption, operators SHOULD use reverse proxies, firewalling, and/or ACLs where feasible.
Metric exposition should be independent of production services exposed to end users; as such, having a /metrics endpoint on ports like TCP/80, TCP/443, TCP/8080, and TCP/8443 is generally discouraged for publicly exposed services using OpenMetrics.
IANA
While currently most implementations of the Prometheus exposition format are using non-IANA-registered ports from an informal registry at {{PrometheusPorts}}, OpenMetrics can be found on a well-defined port.
The port assigned by IANA for clients exposing data is <9099 requested for historical consistency>.
If more than one metric endpoint needs to be reachable at a common IP address and port, operators might consider using a reverse proxy that communicates with exposers over localhost addresses. To ease multiplexing, endpoints SHOULD carry their own name in their path, i.e. 
/node_exporter/metrics. Expositions SHOULD NOT be combined into one exposition, for the reasons covered under "Supporting target metadata in both push-based and pull-based systems" and to allow for independent ingestion without a single point of failure.
OpenMetrics would like to register the MIME type 
application/openmetrics-text

Prometheus Remote-Write 1.0 specification
Version: 1.0
Status: Published
Date: April 2023
This document is intended to define and standardise the API, wire format, protocol and semantics of the existing, widely and organically adopted protocol, and not to propose anything new.
The remote write specification is intended to document the standard for how Prometheus and Prometheus remote-write-compatible agents send data to a Prometheus or Prometheus remote-write compatible receiver.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
NOTE: This specification has a 2.0 version available, see here.
Introduction
Background
The remote write protocol is designed to make it possible to reliably propagate samples in real-time from a sender to a receiver, without loss.
The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB.
The remote write protocol contains opportunities for batching, e.g. sending multiple samples for different series in a single request.  It is not expected that multiple samples for the same series will be commonly sent in the same request, although there is support for this in the protocol.
The remote write protocol is not intended for use by applications to push metrics to Prometheus remote-write-compatible receivers.  It is intended that a Prometheus remote-write-compatible sender scrapes instrumented applications or exporters and sends remote write messages to a server.
A test suite can be found at https://github.com/prometheus/compliance/tree/main/remotewrite/sender.
Glossary
For the purposes of this document the following definitions MUST be followed:
a "Sender" is something that sends Prometheus Remote Write data.
a "Receiver" is something that receives Prometheus Remote Write data.
a "Sample" is a pair of (timestamp, value).
a "Label" is a pair of (key, value).
a "Series" is a list of samples, identified by a unique set of labels.
Definitions
Protocol
The Remote Write Protocol MUST consist of RPCs with the following signature:
func Send(WriteRequest)
message WriteRequest {
  repeated TimeSeries timeseries = 1;
  // Cortex uses this field to determine the source of the write request.
  // We reserve it to avoid any compatibility issues.
  reserved  2;
  // Prometheus uses this field to send metadata, but this is
  // omitted from v1 of the spec as it is experimental.
  reserved  3;
}
message TimeSeries {
  repeated Label labels   = 1;
  repeated Sample samples = 2;
}
message Label {
  string name  = 1;
  string value = 2;
}
message Sample {
  double value    = 1;
  int64 timestamp = 2;
}
Remote write Senders MUST encode the Write Request in the body of a HTTP POST request and send it to the Receivers via HTTP at a provided URL path.  The Receiver MAY specify any HTTP URL path to receive metrics.
Timestamps MUST be int64 counted as milliseconds since the Unix epoch.  Values MUST be float64.
The following headers MUST be sent with the HTTP request:
Content-Encoding: snappy
Content-Type: application/x-protobuf
User-Agent: <name & version of the sender>
X-Prometheus-Remote-Write-Version: 0.1.0
Clients MAY allow users to send custom HTTP headers; they MUST NOT allow users to configure them in such a way as to send reserved headers.  For more info see https://github.com/prometheus/prometheus/pull/8416.
The remote write request in the body of the HTTP POST MUST be compressed with Google’s Snappy.  The block format MUST be used - the framed format MUST NOT be used.
The remote write request MUST be encoded using Google Protobuf 3, and MUST use the schema defined above.  Note the Prometheus implementation uses gogoproto optimisations - for receivers written in languages other than Golang the gogoproto types MAY be substituted for line-level equivalents.
The response body from the remote write receiver SHOULD be empty; clients MUST ignore the response body. The response body is RESERVED for future use.
Backward and forward compatibility
The protocol follows semantic versioning 2.0: any 1.x compatible receivers MUST be able to read any 1.x compatible sender and so on.  Breaking/backwards incompatible changes will result in a 2.x version of the spec.
The proto format itself is forward / backward compatible, in some respects:
Removing fields from the proto will mean a major version bump.
Adding (optional) fields will be a minor version bump.
Negotiation:
Senders MUST send the version number in a headers.
Receivers MAY return the highest version number they support in a response header ("X-Prometheus-Remote-Write-Version").
Senders who wish to send in a format >1.x MUST start by sending an empty 1.x, and see if the response says the receiver supports something else.  The Sender MAY use any supported version .  If there is no version header in the response, senders MUST assume 1.x compatibility only.
Labels
The complete set of labels MUST be sent with each sample. Whatsmore, the label set associated with samples:
SHOULD contain a 
__name__ label.
MUST NOT contain repeated label names.
MUST have label names sorted in lexicographical order.
MUST NOT contain any empty label names or values.
Senders MUST only send valid metric names, label names, and label values:
Metric names MUST adhere to the regex 
[a-zA-Z_:]([a-zA-Z0-9_:])*.
Label names MUST adhere to the regex 
[a-zA-Z_]([a-zA-Z0-9_])*.
Label values MAY be any sequence of UTF-8 characters .
Receivers MAY impose limits on the number and length of labels, but this will be receiver-specific and is out of scope for this document.
Label names beginning with "__" are RESERVED for system usage and SHOULD NOT be used, see Prometheus Data Model.
Remote write Receivers MAY ingest valid samples within a write request that otherwise contains invalid samples. Receivers MUST return a HTTP 400 status code ("Bad Request") for write requests that contain any invalid samples. Receivers SHOULD provide a human readable error message in the response body. Senders MUST NOT try and interpret the error message, and SHOULD log it as is.
Ordering
Prometheus Remote Write compatible senders MUST send samples for any given series in timestamp order. Prometheus Remote Write compatible Senders MAY send multiple requests for different series in parallel.
Retries & Backoff
Prometheus Remote Write compatible senders MUST retry write requests on HTTP 5xx responses and MUST use a backoff algorithm to prevent overwhelming the server. They MUST NOT retry write requests on HTTP 2xx and 4xx responses other than 429. They MAY retry on HTTP 429 responses, which could result in senders "falling behind" if the server cannot keep up. This is done to ensure data is not lost when there are server side errors, and progress is made when there are client side errors.
Prometheus remote Write compatible receivers MUST respond with a HTTP 2xx status code when the write is successful. They MUST respond with HTTP status code 5xx when the write fails and SHOULD be retried. They MUST respond with HTTP status code 4xx when the request is invalid, will never be able to succeed and should not be retried.
Stale Markers
Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to.
Stale markers MUST be signalled by  the special NaN value 0x7ff0000000000002. This value MUST NOT be used otherwise.
Typically the sender can detect when a time series will no longer be appended to using the following techniques:
Detecting, using service discovery, that the target exposing the series has gone away
Noticing the target is no longer exposing the time series between successive scrapes
Failing to scrape the target that originally exposed a time series
Tracking configuration and evaluation for recording and alerting rules
Out of Scope
This document does not intend to explain all the features required for a fully Prometheus-compatible monitoring system.  In particular, the following areas are out of scope for the first version of the spec:
The "up" metric The definition and semantics of the "up" metric are beyond the scope of the remote write protocol and should be documented separately.
HTTP Path The path for HTTP handler can be anything - and MUST be provided by the sender.  Generally we expect the whole URL to be specified in config.
Persistence It is recommended that Prometheus Remote Write compatible senders should persistently buffer sample data in the event of outages in the receiver.
Authentication & Encryption as remote write uses HTTP, we consider authentication & encryption to be a transport-layer problem.  Senders and receivers should support all the usual suspects (Basic auth, TLS etc) and are free to add potentially custom authentication options.  Support for custom authentication in the Prometheus remote write sender and eventual agent should not be assumed, but we will endeavour to support common and widely used auth protocols, where feasible.
Remote Read this is a separate interface that has already seen some iteration, and is less widely used.
Sharding the current sharding scheme in Prometheus for remote write parallelisation is very much an implementation detail, and isn’t part of the spec.  When senders do implement parallelisation they MUST preserve per-series sample ordering.
Backfill The specification does not place a limit on how old series can be pushed, however server/implementation specific constraints may exist.
Limits Limits on the number and length of labels, batch sizes etc are beyond the scope of this document, however it is expected that implementation will impose reasonable limits.
Push-based Prometheus Applications pushing metrics to Prometheus Remote Write compatible receivers was not a design goal of this system, and should be explored in a separate doc.
Labels Every series MAY include a "job" and/or "instance" label, as these are typically added by service discovery in the Sender. These are not mandatory.
Future Plans
This section contains speculative plans that are not considered part of protocol specification, but are mentioned here for completeness.
Transactionality Prometheus aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with remote write -  for instance, in the future we would like to "align" remote write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single remote write request.  This is yet to be designed.
Metadata and Exemplars In line with above, we also send metadata (type information, help text) and exemplars along with the scraped samples. We plan to package this up in a single remote write request - future versions of the spec may insist on this.  Prometheus currently has experimental support for sending metadata and exemplars.
Optimizations We would like to investigate various optimizations to reduce message size by eliminating repetition of label names and values.
Related
Compatible Senders and Receivers
The spec is intended to describe how the following components interact (as of April 2023):
Prometheus (as both a "sender" and a "receiver")
Avalanche (as a "sender") - A Load Testing Tool Prometheus Metrics.
Cortex (as a "receiver")
Elastic Agent (as a "receiver")
Grafana Agent (as both a "sender" and a "receiver")
GreptimeDB (as a "receiver")
InfluxData’s Telegraf agent. (as a sender, and as a receiver)
M3 (as a "receiver")
Mimir (as a "receiver")
Oodle (as a "receiver")
OpenTelemetry Collector (as a "sender" and eventually as a "receiver")
Thanos (as a "receiver")
Vector (as a "sender" and a "receiver")
VictoriaMetrics (as a "receiver")
FAQ
Why did you not use gRPC?
Funnily enough we initially used gRPC, but switched to Protos atop HTTP as in 2016 it was hard to get them past ELBs: https://github.com/prometheus/prometheus/issues/1982
Why not streaming protobuf messages?
If you use persistent HTTP/1.1 connections, they are pretty close to streaming…  Of course headers have to be re-sent, but yes that is less expensive than a new TCP set up.
Why do we send samples in order?
The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is append only. It is possible to remove this constraint, for instance by buffering samples and reordering them before encoding.  We can investigate this in future versions of the protocol.
How can we parallelise requests with the in-order constraint?
Samples must be in-order for a given series.  Remote write requests can be sent in parallel as long as they are for different series. In Prometheus, we shard the samples by their labels into separate queues, and then writes happen sequentially in each queue.  This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.
We believe this is necessary as, even if the receiver could support out-of-order samples, we can't have agents sending out of order as they would never be able to send to Prometheus, Cortex and Thanos.  We’re doing this to ensure the integrity of the ecosystem and to prevent confusing/forking the community into "prometheus-agents-that-can-write-to-prometheus" and those that can’t.

Prometheus Remote-Write 2.0 specification [EXPERIMENTAL]
Version: 2.0-rc.6
Status: Experimental
Date: October 2026
The Remote-Write specification, in general, is intended to document the standard for how Prometheus and Prometheus Remote-Write compatible senders send data to Prometheus or Prometheus Remote-Write compatible receivers.
This document is intended to define a second version of the Prometheus Remote-Write API with minor changes to protocol and semantics. This second version adds a new Protobuf Message with new features enabling more use cases and wider adoption on top of performance and cost savings. The second version also deprecates the previous Protobuf Message from a 1.0 Remote-Write specification and adds mandatory 
X-Prometheus-Remote-Write-*-Written HTTP response headers for reliability purposes. Finally, this spec outlines how to implement backwards-compatible senders and receivers (even under a single endpoint) using existing basic content negotiation request headers. More advanced, automatic content negotiation mechanisms might come in a future minor version if needed. For the rationales behind the 2.0 specification, see the formal proposal.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
NOTE: This is a release candidate for Remote-Write 2.0 specification. This means that this specification is currently in an experimental state--no major changes are expected, but we reserve the right to break the compatibility if it's necessary, based on the early adopters' feedback. The potential feedback, questions and suggestions should be added as comments to the PR with the open proposal.
Introduction
Background
The Remote-Write protocol is designed to make it possible to reliably propagate samples in real-time from a sender to a receiver, without loss.
The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB.
The Remote-Write protocol contains opportunities for batching, e.g. sending multiple samples for different series in a single request. It is not expected that multiple samples for the same series will be commonly sent in the same request, although there is support for this in the Protobuf Message.
Compliance tests can be found at:
sender: https://github.com/prometheus/compliance/tree/main/remotewrite/sender
receiver: https://github.com/prometheus/compliance/tree/main/remotewrite/receiver
Glossary
In this document, the following definitions are followed:
Remote-Write is the name of this Prometheus protocol.
a 
Protocol is a communication specification that enables the client and server to transfer metrics.
a 
Protobuf Message (or Proto Message) refers to the content type definition of the data structure for this Protocol. Since the specification uses Google Protocol Buffers ("protobuf") exclusively, the schema is defined in a "proto" file and represented by a single Protobuf "message".
a 
Wire Format is the format of the data as it travels on the wire (i.e. in a network). In the case of Remote-Write, this is always the compressed binary protobuf format.
a 
Sender is something that sends Remote-Write data.
a 
Receiver is something that receives (writes) Remote-Write data. The meaning of 
Written is up to the Receiver e.g. usually it means storing received data in a database, but also just validating, splitting or enhancing it.
a 
Mode is a set of additional rules on top of 
Protocol that tightens optional sender and receiver semantics into a deterministic contract.
Written refers to data the 
Receiver has received and is accepting. Whether or not it has ingested this data to persistent storage, written it to a WAL, etc. is up to the 
Receiver. The only distinction is that the 
Receiver has accepted this data rather than explicitly rejecting it with an error response.
a 
Sample is a triplet of (start timestamp, timestamp, value).
a 
Histogram is a triplet of (start timestamp, timestamp, histogram value).
a 
Label is a pair of (key, value).
a 
Series is a list of samples (or histograms), identified by a unique set of labels.
Definitions
Protocol
The Remote-Write Protocol MUST consist of RPCs with the request body serialized using a Google Protocol Buffers and then compressed.
The protobuf serialization MUST use either of the following Protobuf Messages:
The 
prometheus.WriteRequest introduced in the Remote-Write 1.0 specification. As of 2.0, this message is deprecated. It SHOULD be used only for compatibility reasons. Senders and Receivers MAY NOT support the 
prometheus.WriteRequest.
The 
io.prometheus.write.v2.Request introduced in this specification and defined below. Senders and Receivers SHOULD use this message when possible. Senders and Receivers MUST support the 
io.prometheus.write.v2.Request.
Protobuf Message MUST use binary Wire Format. Then, MUST be compressed with Google’s Snappy. Snappy's block format MUST be used -- the framed format MUST NOT be used.
Senders MUST send a serialized and compressed Protobuf Message in the body of an HTTP POST request and send it to the Receiver via HTTP at the provided URL path. Receivers MAY specify any HTTP URL path to receive metrics.
Senders MUST send the following reserved headers with the HTTP request:
Content-Encoding
Content-Type
X-Prometheus-Remote-Write-Version
User-Agent
Senders MAY allow users to add custom HTTP headers; they MUST NOT allow users to configure them in such a way as to send reserved headers.
Content-Encoding
Content-Encoding: <compression>
Content encoding request header MUST follow the RFC 9110. Senders MUST use the 
snappy value. Receivers MUST support 
snappy compression. New, optional compression algorithms might come in 2.x or beyond.
Content-Type
Content-Type: application/x-protobuf
Content-Type: application/x-protobuf;proto=<fully qualified name>
Content-Type: application/x-protobuf;proto=<fully qualified name>;mode=<mode>
Content type request header MUST follow the RFC 9110. Senders MUST use 
application/x-protobuf as the only media type. Senders MAY add 
;proto= parameter to the header's value to indicate the fully qualified name of the Protobuf Message that was used, from the two mentioned above. Senders MAY also add 
;mode= parameter to indicate the mode (additional semantic rules) for the Protobuf Message, if any is defined for it. As a result, Senders MUST send one of the following supported header values:
For the deprecated message introduced in PRW 1.0, identified by 
prometheus.WriteRequest:
Content-Type: application/x-protobuf
Content-Type: application/x-protobuf;proto=prometheus.WriteRequest
For the message introduced in PRW 2.0, identified by 
io.prometheus.write.v2.Request:
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=none (equivalent to omitting 
;mode=; see Modes)
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless (see Modes)
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-st (see Modes)
When talking to 1.x Receivers, Senders SHOULD use 
Content-Type: application/x-protobuf for backward compatibility. Otherwise, Senders SHOULD use 
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request (optionally with 
;mode=none, 
;mode=stateless, or 
;mode=stateless-st). More Protobuf Messages or modes might come in 2.x or beyond.
Receivers MUST use the content type header to identify the Protobuf Message schema to use, and MAY use it to identify the mode. Accidental wrong schema choices may result in non-deterministic behaviour (e.g. corruptions).
NOTE: Thanks to reserved fields in 
io.prometheus.write.v2.Request, Receiver accidental use of wrong schema with 
prometheus.WriteRequest will result in empty message. This is generally for convenience to avoid surprising errors, but don't rely on it -- future Protobuf Messages might not have this feature.
X-Prometheus-Remote-Write-Version
X-Prometheus-Remote-Write-Version: <Remote-Write spec major and minor version>
When talking to 1.x Receivers, Senders MUST use 
X-Prometheus-Remote-Write-Version: 0.1.0 for backward compatibility. Otherwise, Senders SHOULD use the newest Remote-Write version it is compatible with e.g. 
X-Prometheus-Remote-Write-Version: 2.0.0.
User-Agent
User-Agent: <name & version of the Sender>
Senders MUST include a user agent header that SHOULD follow the RFC 9110 User-Agent header format.
Response
Receivers that written all data successfully MUST return a success 2xx HTTP status code. In such a successful case, the response body from the Receiver SHOULD be empty and the status code SHOULD be 204 HTTP No Content; Senders MUST ignore the response body. The response body is RESERVED for future use.
Receivers that operate asynchronously (e.g. ingest payloads into a message queue or pipeline for deferred processing without synchronous parsing) MAY return 202 HTTP Accepted status code. This indicates that the request has been accepted for processing, but processing has not been completed.
Receivers MUST NOT return a 2xx HTTP status code, if any of the pieces of sent data known to the Receiver (e.g. Samples, Histograms, Exemplars) were NOT synchronously written successfully (both partial write or full write rejection) or wasn't sent for asynchronous processing (queue full for example). In such a case, the Receiver MUST provide a human-readable error message in the response body. The Receiver's error SHOULD contain information about the amount of the samples being rejected and for what reasons. Senders MUST NOT try and interpret the error message and SHOULD log it as is.
The following subsections specify Sender and Receiver semantics around headers and different write error cases.
Required 
Written Response Headers
Upon a successful content negotiation, Receivers process (write) the received batch of data. Once completed (with success or failure) for each important piece of data (currently Samples, Histograms and Exemplars) Receivers MUST send a dedicated HTTP 
X-Prometheus-Remote-Write-*-Written response header with the precise number of successfully written elements.
Asynchronous Receivers returning 
202 Accepted MAY omit these headers if counts cannot be calculated synchronously (validation is deferred). Asynchronous Receivers SHOULD propagate errors to clients via different channels.
Each header value MUST be a single 64-bit integer. The header names MUST be as follows:
X-Prometheus-Remote-Write-Samples-Written <count of all successfully written Samples>
X-Prometheus-Remote-Write-Histograms-Written <count of all successfully written Histogram samples>
X-Prometheus-Remote-Write-Exemplars-Written <count of all successfully written Exemplars>
Upon receiving a 2xx status code or upon receiving a 4xx status code, Senders CAN assume that any missing 
X-Prometheus-Remote-Write-*-Written response header means no element from this category (e.g. Sample) was written by the Receiver (count of 
0).
Senders MUST NOT assume the same when:
Receiving 
202 Accepted status code as asynchronous Receivers may defer validation.
When using the deprecated 
prometheus.WriteRequest Protobuf Message due to the risk of hitting 1.0 Receiver without this feature.
Senders MAY use those headers to confirm which parts of data were successfully written by the Receiver. Common use cases:
Better handling of the Partial Write failure situations: Senders MAY use those headers for more accurate client instrumentation and error handling.
Detecting broken 1.0 Receiver implementations: Senders SHOULD assume 415 HTTP Unsupported Media Type status code when sending the data using 
io.prometheus.write.v2.Request request and receiving a 2xx HTTP status code other than 
202 Accepted, but none of the 
X-Prometheus-Remote-Write-*-Written response headers from the Receiver. This is a common issue for the 1.0 Receivers that do not check the 
Content-Type request header; accidental decoding of the 
io.prometheus.write.v2.Request payload with 
prometheus.WriteRequest schema results in empty result and no decoding errors.
Detecting other broken implementations or issues: Senders MAY use those headers to detect broken Sender and Receiver implementations or other problems.
Senders MUST NOT assume what Remote Write specification version the Receiver implements from the remote write response headers.
More (optional) headers might come in the future, e.g. when more entities or fields are added and worth confirming.
Partial Write
Senders SHOULD use Remote-Write to send samples for multiple series in a single request. As a result, Receivers MAY write valid samples within a write request that also contains some invalid or otherwise unwritten samples, which represents a partial write case. In such a case, the Receiver MUST return non-2xx status code following the Invalid Samples and Retry on Partial Writes sections.
Unsupported Request Content
Receivers MUST return 415 HTTP Unsupported Media Type status code if they don't support a given content type or encoding provided by Senders. When returning 
415 HTTP Unsupported Media Type, Receivers SHOULD include the 
Accept (listing supported 
Content-Type values, including supported 
;proto= and 
;mode= parameters) and 
Accept-Encoding (listing supported 
Content-Encoding values) response headers as defined in RFC 7694 so Senders can determine what the Receiver supports without guessing.
Senders SHOULD expect 400 HTTP Bad Request for the above reasons from 1.x Receivers, for backwards compatibility.
Invalid Samples
Receivers SHOULD reject data (such as samples, histograms, exemplars, or entire time series) that violates this specification or the negotiated mode. Receivers MAY reject data that violates receiver-specific limits and gaps (such as not supporting native histograms, or label count or length limits). Receivers MUST return a 400 HTTP Bad Request status code for write requests that contain any invalid data (whether rejecting the entire request or performing a partial write) unless the partial retriable write occurs.
Senders MUST NOT retry on a 4xx HTTP status codes (other than 429), which MUST be used by Receivers to indicate that the write operation will never be able to succeed and should not be retried. Senders MAY retry on the 415 HTTP status code with a different content type or encoding (using the 
Accept and 
Accept-Encoding response headers if present; see Unsupported Request Content) to see if the Receiver supports it.
Retries & Backoff
Receivers MAY return a 429 HTTP Too Many Requests status code to indicate the overloaded server situation. Receivers MAY return the Retry-After header to indicate the time for the next write attempt. Receivers MAY return a 5xx HTTP status code to represent internal server errors.
Senders MAY retry on a 429 HTTP status code. Senders MUST retry write requests on 5xx HTTP. Senders MUST use a backoff algorithm to prevent overwhelming the server. Senders MAY handle the Retry-After response header to estimate the next retry time.
The difference between 429 vs 5xx handling is due to the potential situation of a Sender “falling behind” when the Receiver cannot keep up with the request volume, or the Receiver choosing to rate limit the Sender to protect its availability. As a result, Senders has the option to NOT retry on 429, which allows progress to be made when there are Sender side errors (e.g. too much traffic), while the data is not lost when there are Receiver side errors (5xx).
Retries on Partial Writes
Receivers MAY return a 5xx HTTP or 429 HTTP status code on partial write or partial invalid sample cases when it expects Senders to retry the whole request. In that case, the Receiver MUST support idempotency as Senders MAY retry with the same request.
Backward and Forward Compatibility
The protocol follows semantic versioning 2.0: any 2.x compatible Receiver MUST be able to read any 2.x compatible Senders and vice versa. Breaking or backwards incompatible changes will result in a 3.x version of the spec.
The Protobuf Messages (in Wire Format) themselves are forward / backward compatible, in some respects:
Removing fields from the Protobuf Message requires a major version bump.
Adding (optional) fields can be done in a minor version bump.
In other words, this means that future minor versions of 2.x MAY add new optional fields to 
io.prometheus.write.v2.Request, new compressions, Protobuf Messages and negotiation mechanisms, as long as they are backwards compatible (e.g. optional to both Receiver and Sender).
2.x vs 1.x Compatibility
The 2.x protocol is breaking compatibility with 1.x by introducing a new, mandatory 
io.prometheus.write.v2.Request Protobuf Message and deprecating the 
prometheus.WriteRequest.
2.x Senders MAY support 1.x Receivers by allowing users to configure what content type Senders should use. 2.x Senders also MAY automatically fall back to different content types, if the Receiver returns 415 HTTP status code.
Protobuf Message
io.prometheus.write.v2.Request
The 
io.prometheus.write.v2.Request references the new Protobuf Message that's meant to replace and deprecate the Remote-Write 1.0's 
prometheus.WriteRequest message.
The full schema and source of the truth is in Prometheus repository in 
prompb/io/prometheus/write/v2/types.proto. The 
gogo dependency and options CAN be ignored (will be removed eventually). They are not part of the specification as they don't impact the serialized format.
The simplified version of the new 
io.prometheus.write.v2.Request is presented below.
message Request {
  reserved 1 to 3;
  // symbols contains a de-duplicated array of string elements used for various
  // items in a Request message, like labels and metadata items. For the sender's convenience
  // around empty values for optional fields like unit_ref, symbols array MUST start with
  // empty string.
  //
  // To decode each of the symbolized strings, referenced, by "ref(s)" suffix, you
  // need to lookup the actual string by index from symbols array. The order of
  // strings is up to the sender. The receiver should not assume any particular encoding.
  repeated string symbols = 4;
  // timeseries represents an array of distinct series with 0 or more samples.
  repeated TimeSeries timeseries = 5;
}
// TimeSeries represents a single series.
message TimeSeries {
  reserved 6;
    
  // labels_refs is a list of label name-value pair references, encoded
  // as indices to the Request.symbols array. This list's length is always
  // a multiple of two, and the underlying labels should be sorted lexicographically.
  //
  // Note that there might be multiple TimeSeries objects in the same
  // Requests with the same labels e.g. for different exemplars, metadata
  // or start timestamp.
  repeated uint32 labels_refs = 1;
  // TimeSeries messages must specify at least one float sample, native histogram sample
  // or exemplar. Samples and histograms must not be specified at the same time.
  // Exemplars may be specified without samples or histograms. For a typical sender
  // (real-time metric streaming), in healthy cases, there will be only one sample or histogram.
  //
  // Samples and histograms are sorted by timestamp (older first).
  repeated Sample samples = 2;
  repeated Histogram histograms = 3;
  // exemplars represents an optional set of exemplars for this series or series's samples/histograms if present.
  repeated Exemplar exemplars = 4;
  // metadata represents the metadata associated with the given series' samples.
  Metadata metadata = 5;
}
// Exemplar contains additional information associated with a series or series's samples/histograms if present.
// It is typically used to attach an example trace or request ID associated with
// the metric changes.
message Exemplar {
  // labels_refs is an optional list of label name-value pair references, encoded
  // as indices to the Request.symbols array. This list's len is always
  // a multiple of 2, and the underlying labels should be sorted lexicographically.
  // If the exemplar references a trace it should use the `trace_id` label name, as a best practice.
  repeated uint32 labels_refs = 1;
  // value represents an exact example value. This can be useful when the exemplar
  // is attached to a histogram, which only gives an estimated value through buckets.
  double value = 2;
  // timestamp represents the timestamp of the exemplar in ms.
  //
  // For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go
  // for conversion from/to time.Time to Prometheus timestamp.
  int64 timestamp = 3;
}
// Sample represents series sample.
message Sample {
  // value of the sample.
  double value = 1;
  // timestamp represents timestamp of the sample in ms.
  //
  // For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go
  // for conversion from/to time.Time to Prometheus timestamp.
  int64 timestamp = 2;
  // start_timestamp represents an optional start timestamp for the sample,
  // in ms format. This information is typically used for counter, histogram (cumulative)
  // or delta type metrics.
  //
  // For cumulative metrics, the start timestamp represents the time when the
  // counter started counting (sometimes referred to as created timestamp), which
  // can increase the accuracy of certain processing and query semantics (e.g. rates).
  //
  // Note:
  // * That some receivers might require start timestamps for certain metric
  // types; rejecting such samples within the Request as a result.
  // * start timestamp is the same as "created timestamp" name Prometheus used in the past.
  //
  // For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go
  // for conversion from/to time.Time to Prometheus timestamp.
  //
  // Note that the "optional" keyword is omitted due to efficiency and consistency.
  // Zero value means value not set. If you need to use exactly zero value for
  // the timestamp, use 1 millisecond before or after.
  int64 start_timestamp = 3;
}
// Metadata represents the metadata associated with the given series' samples.
message Metadata {
  enum MetricType {
    METRIC_TYPE_UNSPECIFIED    = 0;
    METRIC_TYPE_COUNTER        = 1;
    METRIC_TYPE_GAUGE          = 2;
    METRIC_TYPE_HISTOGRAM      = 3;
    METRIC_TYPE_GAUGEHISTOGRAM = 4;
    METRIC_TYPE_SUMMARY        = 5;
    METRIC_TYPE_INFO           = 6;
    METRIC_TYPE_STATESET       = 7;
  }
  MetricType type = 1;
  // help_ref is a reference to the Request.symbols array representing help
  // text for the metric. Help is optional, reference should point to an empty string in
  // such a case.
  uint32 help_ref = 3;
  // unit_ref is a reference to the Request.symbols array representing a unit
  // for the metric. Unit is optional, reference should point to an empty string in
  // such a case.
  uint32 unit_ref = 4;
}
// A native histogram message, supporting
// * sparse exponential bucketing, custom bucketing.
// * float or integer histograms.
//
// See the full spec: https://prometheus.io/docs/specs/native_histograms/
message Histogram { ... }
All timestamps MUST be int64 counted as milliseconds since the Unix epoch. Sample's values MUST be float64.
For every 
TimeSeries message:
labels_refs MUST be provided.
At least one element in 
samples, in 
histograms, or in 
exemplars MUST be provided (in 
stateless and 
stateless-st modes, at least one element in 
samples or in 
histograms MUST be provided). For example, a 
TimeSeries with only 
metadata and 
labels_refs is not valid.
A 
TimeSeries MUST NOT include both 
samples and 
histograms. For series which (rarely) would mix float and histogram samples, a separate 
TimeSeries message MUST be used.
metadata sub-fields SHOULD be provided (in 
stateless and 
stateless-st modes, 
Metadata.type MUST be specified). Receivers MAY reject series with unspecified 
Metadata.type.
Exemplars SHOULD be provided if they exist for a series.
The following subsections define some schema elements in detail.
Symbols
The 
io.prometheus.write.v2.Request Protobuf Message is designed to intern all strings for the proven additional compression and memory efficiency gains on top of the standard compressions.
The 
symbols table MUST be provided, and it MUST contain deduplicated strings used in series, exemplar labels, and metadata strings. The first element of the 
symbols table MUST be an empty string, which is used to represent empty or unspecified values such as when 
Metadata.unit_ref or 
Metadata.help_ref are not provided. References MUST point to the existing index in the 
symbols string array.
Series Labels
The complete set of labels MUST be sent with each 
Sample or 
Histogram sample. Additionally, the label set associated with samples:
SHOULD contain a 
__name__ label.
MUST NOT contain repeated label names.
MUST have label names sorted in lexicographical order.
MUST NOT contain any empty label names or values.
Metric names, label names, and label values MUST be any sequence of UTF-8 characters.
Metric names SHOULD adhere to the regex 
[a-zA-Z_:]([a-zA-Z0-9_:])*.
Label names SHOULD adhere to the regex 
[a-zA-Z_]([a-zA-Z0-9_])*.
Names that do not adhere to the above, might be harder to use for PromQL users (see the UTF-8 proposal for more details).
Label names beginning with "__" are RESERVED for system usage and SHOULD NOT be used, see Prometheus Data Model.
Receivers also MAY impose limits on the number and length of labels, but this is receiver-specific and is out of the scope of this document.
Samples and Histogram Samples
Senders MUST send 
samples (or 
histograms) for any given 
TimeSeries in timestamp order. Senders MAY send multiple requests for different series in parallel. In 
stateless and 
stateless-st modes, histogram metrics MUST be sent using 
histograms (e.g. Native Histograms with Custom Buckets) rather than classic histogram float 
samples.
Sample's or Histogram's 
start_timestamp SHOULD be provided for types that follow counter semantics (e.g. counters and counter histograms; in 
stateless-st mode, 
start_timestamp MUST be set for these types, except on stale markers). Receivers MAY reject those series without 
start_timestamp being set. Given optionality, the 0 value MUST be treated by receivers as unset value. To represent the unlikely 0 unix timestamp in milliseconds, "1" or "-1" value MUST be used.
Senders SHOULD send stale markers when a time series will no longer be appended to.
Senders MUST send stale markers if the discontinuation of time series is possible to detect, for example:
For series that were pulled (scraped), unless explicit timestamp was used.
For series that is resulted by a recording rule evaluation.
Generally, not sending stale markers for series that are discontinued can lead to the Receiver non-trivial query time alignment issues.
Stale markers MUST be signalled by the special NaN value 
0x7ff0000000000002. This value MUST NOT be used otherwise.
Typically, Senders can detect when a time series will no longer be appended using the following techniques:
Detecting, using service discovery, that the target exposing the series has gone away.
Noticing the target is no longer exposing the time series between successive scrapes.
Failing to scrape the target that originally exposed a time series.
Tracking configuration and evaluation for recording and alerting rules.
Tracking discontinuation of metrics for non-scrape source of metric (e.g. in k6 when the benchmark has finished for series per benchmark, it could emit a stale marker).
Metadata
Metadata SHOULD follow the official Prometheus guidelines for Type and Help.
Metadata MAY follow the official OpenMetrics guidelines for Unit.
Exemplars
Each exemplar, if attached to a 
TimeSeries:
MUST contain a value.
MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the 
trace_id label name, as a best practice.
MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars.
MAY be sent in a 
TimeSeries that carries neither samples nor histograms (except in 
stateless and 
stateless-st modes, where exemplars MUST only be sent on a 
TimeSeries that also contains at least one element in 
samples or in 
histograms). For cases where there are multiple 
samples or 
histograms on a 
TimeSeries, all exemplars MUST be associated with all of those 
samples or 
histograms.
Modes
By default (
none mode, when no 
;mode= parameter is set or when 
;mode=none is specified), 
io.prometheus.write.v2.Request keeps several structural and metadata rules optional (
SHOULD or 
MAY) so Senders can forward any scraped metric without restrictions or conversion overhead. Senders MAY omit 
;mode= or explicitly send 
;mode=none to indicate this baseline mode. However, some common stateless or strongly typed Receivers cannot ingest all the data e.g. detached exemplars, classic histogram series, untyped metrics, or counters without start timestamps.
To address this, 
io.prometheus.write.v2.Request defines two progressive modes, 
stateless and 
stateless-st, that tighten optional rules into 
MUST requirements on Senders while keeping every payload a valid 
io.prometheus.write.v2.Request message. See the FAQ for the rationale.
Senders SHOULD support 
stateless and 
stateless-st modes. Senders MAY operate in either mode behind configuration or by default.
Receivers MAY accept any valid 
io.prometheus.write.v2.Request regardless of mode, require at least 
stateless mode, or require 
stateless-st mode only. If rejected, 415 HTTP Unsupported Media Type MUST be used (see Unsupported Request Content).
stateless Mode
When operating in 
stateless mode, Senders MUST:
Use content type of 
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless.
Ensure all of the following for every 
TimeSeries in a 
Request:
Exemplars MUST only be sent on a 
TimeSeries that also contains at least one element in 
samples or in 
histograms. Exemplar-only 
TimeSeries messages MUST NOT be sent.
Histograms MUST use the 
Histogram message (e.g. using Native Histograms with Custom Buckets instead of classic histogram 
_bucket, 
_sum, and 
_count float 
Sample series).
Metadata.type MUST be specified and MUST NOT be 
METRIC_TYPE_UNSPECIFIED (
0). For series with an unknown metric type (such as 
untyped scraped metrics or recording rule outputs), Senders SHOULD assume:
METRIC_TYPE_GAUGE, unless the metric name has a 
_total suffix, in which case Senders SHOULD assume 
METRIC_TYPE_COUNTER.
Metadata.help_ref set to 
"(type inferred from untyped)" if empty, or with 
" (type inferred from untyped)" appended to the existing help text.
stateless-st Mode
When operating in 
stateless-st mode, Senders MUST:
Use content type of 
Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request;mode=stateless-st.
Satisfy all 
TimeSeries requirements of 
stateless mode.
Ensure that 
start_timestamp is set (non-zero) on every 
Sample and 
Histogram (except stale markers) for metric types 
METRIC_TYPE_COUNTER, 
METRIC_TYPE_HISTOGRAM, as well as 
METRIC_TYPE_SUMMARY 
_sum and 
_count 
Samples.
Out of Scope
The same as in 1.0.
Future Plans
This section contains speculative plans that are not considered part of protocol specification yet but are mentioned here for completeness. Note that 2.0 specification completed 2 of 3 future plans in the 1.0.
Transactionality There is still no transactionality defined for 2.0 specification, mostly because it makes a scalable Sender implementation difficult. Prometheus Sender aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with Remote-Write -- for instance, in the future we would like to "align" Remote-Write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single Remote-Write request.
However, Remote-Write 2.0 specification solves an important transactionality problem for the classic histogram buckets. This is done thanks to the native histograms supporting custom bucketing (NHCB) in the 
io.prometheus.write.v2.Request wire format, which 
stateless and 
stateless-st modes mandate instead of classic histograms. When no mode is specified, Senders might still translate classic histograms to native histograms this way, and Receivers MAY ignore certain metric types (e.g. classic histograms).
Alternative wire formats. The OpenTelemetry community has shown the validity of Apache Arrow (and potentially other columnar formats) for over-wire data transfer with their OTLP protocol. We would like to do experiments to confirm the compatibility of a similar format with Prometheus’ data model and include benchmarks of any resource usage changes. We would potentially maintain both a protobuf and columnar format long term for compatibility reasons and use our content negotiation to add different Protobuf Messages for this purpose.
Global symbols. Pre-defined string dictionary for interning The protocol could pre-define a static dictionary of ref->symbol that includes strings that are considered common, e.g. “namespace”, “le”, “job”, “seconds”, “bytes”, etc. Senders could refer to these without the need to include them in the request’s symbols table. This dictionary could incrementally grow with minor version releases of this protocol.
Related
FAQ
Why did you not use gRPC?
Because the 1.0 protocol does not use gRPC, breaking it would increase friction in the adoption. See 1.0 reason.
Why not stream protobuf messages?
If you use persistent HTTP/1.1 connections, they are pretty close to streaming. Of course, headers have to be re-sent, but that is less expensive than a new TCP set up.
Why do we send samples in order?
The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is optimized for append-only workloads. However, this requirement is also shared across many other databases and vendors in the ecosystem. In fact, Prometheus with OOO feature enabled, allows out-of-order writes, but with the performance penalty, thus reserved for rare events. To sum up, Receivers may support out-of-order write, though it is not permitted by the specification. In the future e.g. 2.x spec versions, we could extend content type to negotiate the out-of-order writes, if needed.
What's the purpose of 
stateless and 
stateless-st modes?
By default (
none mode, when no mode or 
;mode=none is specified), Remote-Write 2.0 allows any valid permutation of fields in 
io.prometheus.write.v2.Request to preserve compatibility and minimal overhead for existing Prometheus TSDB pipelines.
However, stateless or strongly-typed receivers often require structured histograms, attached exemplars, metric types, or start timestamps to ingest metrics without stateful caches or mid-stream 
400 HTTP Bad Request rejections. Modes have two goals:
Simplify communication between senders and receivers, such as configuring required sender features as a single setting, rejecting unsupported modes early on the receiver with 
415 HTTP Unsupported Media Type, and documenting the cost and requirements of each mode.
Help the ecosystem adopt stricter semantics, such as letting receivers advertise which mode they require and letting operators monitor mode adoption across senders.
We defined two progressive modes, 
stateless and 
stateless-st, rather than a single mode or fine-grained capability flags:
stateless mode rules (attaching exemplars to samples or histograms, converting classic histograms to NHCB, and requiring 
Metadata.type with a 
METRIC_TYPE_GAUGE or 
_total-based 
METRIC_TYPE_COUNTER fallback for unknown metric types) make every 
TimeSeries message self-contained and typed. Any sender can enforce 
stateless mode without dropping data or tracking cross-scrape counter start timestamps, and receivers can validate histogram structure because 
Metadata.type is always set.
stateless-st mode adds the 
start_timestamp requirement on cumulative metric types (
METRIC_TYPE_COUNTER, 
METRIC_TYPE_HISTOGRAM, and summary 
_sum and 
_count). It is kept separate from 
stateless because 
start_timestamp depends on upstream instrumentation exposing created timestamps or stateful sender-side synthesis, and adds per-sample wire overhead on cumulative series that not all receivers require.
stateless-st includes 
stateless because receivers that require 
start_timestamp on cumulative metrics also require 
Metadata.type, structured histograms, and attached exemplars, and validating 
start_timestamp itself requires knowing 
Metadata.type. Following RFC 9413, three progressive levels (
none, 
stateless, 
stateless-st) avoid 
2^N protocol permutations, keep 
Content-Type matching to a single parameter value, and keep compliance testing linear.
In future specification versions, we might propose making 
stateless mode the default when 
;mode= is omitted while keeping 
none and 
stateless-st explicitly selectable, depending on ecosystem feedback and adoption.
How can we parallelise requests with the in-order constraint?
Samples must be in-order for a given series. However, even if a Receiver does not support out-of-order write, the Remote-Write requests can be sent in parallel as long as they are for different series. Prometheus shards the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.
What are the differences between Remote-Write 2.0 and OpenTelemetry's OTLP protocol?
OpenTelemetry OTLP is a protocol for transporting of telemetry data (such as metrics, logs, traces and profiles) between telemetry sources, intermediate nodes and telemetry backends. The recommended transport involves gRPC with protobuf, but HTTP with protobuf or JSON are also described. It was designed from scratch with the intent to support a variety of different observability signals, data types and extra information. For metrics that means additional non-identifying labels, flags, temporal aggregations types, resource or scoped metrics, schema URLs and more. OTLP also requires the semantic convention to be used.
Remote-Write was designed for simplicity, efficiency and organic growth. The first version was officially released in 2023, when already dozens of battle-tested adopters in the CNCF ecosystem had been using this protocol for years. Remote-Write 2.0 iterates on the previous protocol by adding a few new elements (metadata, exemplars, start timestamp and native histograms) and string interning. Remote-Write 2.0 is always stateless, focuses only on metrics and is opinionated; as such it is scoped down to elements that the Prometheus community considers enough to have a robust metric solution. The intention is to ensure the Remote-Write is a stable protocol that is cheaper and simpler to adopt and use than the alternatives in the observability ecosystem.