Operations

Manage authentication

Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services.

The simple scalable and microservices deployment modes require a reverse proxy to be deployed in front of Loki, to direct client API requests to the various components.

By default the Loki Helm chart includes a default reverse proxy configuration, using an nginx container to handle routing traffic and authorization.

A list of open-source reverse proxies you can use:

{{< admonition type="note" >}} Loki runs in multi-tenant mode by default (auth_enabled: true). In this mode, Loki requires the HTTP header X-Scope-OrgID to be set to a string identifying the tenant on every request; the responsibility of populating this value should be handled by the authenticating reverse proxy. A request that does not include this header fails with an HTTP 401 Unauthorized error and the message no org id. To learn how to troubleshoot this error, refer to No org ID. For more information, read the multi-tenancy documentation.{{< /admonition >}}

For information on configuring authentication for your log shipping agent, see the Grafana Alloy documentation.

Mutual TLS

Loki's HTTP and gRPC servers can require clients to present a valid TLS certificate before a connection is accepted. This is a form of client authentication that Loki supports natively, without a reverse proxy.

Mutual TLS (mTLS) authenticates the client at the transport level. It does not populate X-Scope-OrgID, so you still need a reverse proxy, your log shipping agent, or your client to set that header when auth_enabled: true.

To require client certificates, you must first enable TLS on the server by setting a certificate and a key, then set a client authentication policy and the certificate authority (CA) that signs your client certificates. For example:

server:
  http_tls_config:
    cert_file: /etc/loki/certs/server.crt
    key_file: /etc/loki/certs/server.key
    # Reject any client that does not present a certificate signed by this CA.
    client_auth_type: RequireAndVerifyClientCert
    client_ca_file: /etc/loki/certs/ca.crt

Use client_ca_file for the path to a CA certificate file, or client_ca to include the certificate contents directly in the configuration file. The grpc_tls_config block accepts the same options for the gRPC server. The equivalent command line flags are -server.http-tls-client-auth and -server.http-tls-ca-path, plus -server.grpc-tls-client-auth and -server.grpc-tls-ca-path.

The supported values for client_auth_type are:

Use RequireAndVerifyClientCert for the strictest setting. It rejects a client that sends no certificate, and it also rejects a certificate that the CA did not sign. The other values check less. For example, VerifyClientCertIfGiven allows a client that sends no certificate at all. For the full list of TLS options, refer to the server block of the configuration reference.

Enable basic authentication for Loki using nginx

This section describes the process of enabling basic authentication for Loki using nginx.

Prerequisites

Configure nginx

You must create a new nginx configuration file for the Loki instance.

This example assumes the following:

If you used different configuration parameters for Loki, adjust the examples to match your configuration.

loki.conf configuration:

upstream loki {
  server 127.0.0.1:3100;
  keepalive 15;
}

server {
  listen 80;
  server_name loki.localhost;

  auth_basic "loki auth";
  auth_basic_user_file /opt/homebrew/etc/nginx/passwords;

  location / {
    proxy_read_timeout 1800s;
    proxy_connect_timeout 1600s;
    proxy_pass http://loki;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
    proxy_redirect off;
  }

  location /ready {
    proxy_pass http://loki;
    proxy_http_version 1.1;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
    proxy_redirect off;
    auth_basic "off";
  }
}

{{< admonition type="tip" >}} This example expects the client to send the X-Scope-OrgID header itself, in addition to the basic authentication credentials. If you run Loki with auth_enabled: true, you can instead have nginx set the tenant ID from the authenticated username. Add this line to the location / {} block:

proxy_set_header X-Scope-OrgID $remote_user;

With this line, nginx overwrites any tenant header sent by the client, so a client cannot read or write the data of another tenant. This requires each username in your password file to match a Loki tenant ID. The Loki Helm chart's gateway uses this same pattern.{{< /admonition >}}

This configuration must be included in your main nginx configuration, for example, by including it in nginx.conf like:

include /opt/homebrew/etc/nginx/loki.conf;

Restart the nginx server to ensure all configuration changes are updated.

Validate your nginx configuration

To validate the nginx configuration for Loki, you can send a curl request to two endpoints:

% curl -i http://loki.localhost/ready

HTTP/1.1 200 OK
Server: nginx/1.29.2
Date: Thu, 16 Oct 2025 14:28:31 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 6
Connection: keep-alive
X-Content-Type-Options: nosniff

ready

curl -i http://loki.localhost/

HTTP/1.1 401 Unauthorized
Server: nginx/1.29.2
Date: Thu, 16 Oct 2025 14:32:43 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive
WWW-Authenticate: Basic realm="loki auth"

<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.29.2</center>
</body>
</html>

Update passwords

The password file can be seeded using whatever mechanism you may use for other web services.

In this example, htpasswd is utilized:

% htpasswd -c /opt/homebrew/etc/nginx/passwords loki123

New password:
Re-type new password:
Adding password for user loki123

Restart the nginx server to ensure all configuration changes are updated.

Validate passwords

Enter your password into a temporary file, such as:

% vi lokipw

Then, store it as an environment variable::

% pass=$(cat lokipw)

You can validate basic authentication is then working by issuing a curl command to the protected resource:

curl -i -u loki123:$pass -H "X-Scope-OrgID:fake" "http://loki.localhost/loki/api/v1/labels"

HTTP/1.1 200 OK
Server: nginx/1.29.2
Date: Thu, 16 Oct 2025 14:46:09 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 21
Connection: keep-alive

{"status":"success"}

Manage large volume log streams with automatic stream sharding

Automatic stream sharding can keep streams under a desired_rate by adding new labels and values to existing streams. When properly tuned, this can eliminate issues where log producers are rate limited due to the per-stream rate limit.

Automatic stream sharding is enabled by default (shard_streams.enabled defaults to true). To tune it or confirm it's on:

  1. Check or set the global limits_config in the Loki configuration file:

    limits_config:
  shard_streams:
    enabled: true

  2. Optionally lower the desired_rate in bytes if you find that the system is still hitting the per_stream_rate_limit:

    limits_config:
  shard_streams:
    enabled: true
    desired_rate: 2097152 #2MiB

    The desired_rate defaults to 1536KB (1.5MB), which is below the default per_stream_rate_limit of 3MB. This gives Loki some headroom to shard a stream before it hits the per-stream rate limit.

  3. Optionally enable time_sharding_enabled if you need to ingest old, out-of-order logs, for example during a backfill:

    limits_config:
  shard_streams:
    enabled: true
    time_sharding_enabled: true
    time_sharding_ignore_recent: 40m

    Time-based sharding adds a __time_shard__ label to streams, splitting log entries into buckets based on their timestamp. Log entries with a timestamp newer than time_sharding_ignore_recent (40 minutes by default) are still ingested, but Loki does not add the __time_shard__ label to them. This lets very old logs be ingested without triggering out-of-order errors. Refer to How automatic stream sharding works for more detail.

  4. Optionally enable logging_enabled for debugging stream sharding. {{< admonition type="note" >}} This may affect the ingestion performance of Loki. {{< /admonition >}}

    limits_config:
  shard_streams:
    enabled: true
    logging_enabled: true

When to use automatic stream sharding

Large log streams present several problems for Loki, namely increased and uneven resource usage on Ingesters and Distributors. The general recommendation is to explore existing log streams for additional label values that are both useful for querying and sufficiently low cardinality. There are many cases, however, where no more labels can be extracted, or cardinality for a label is dangerously large. To protect itself from such volume leading to operational failure, Loki implements per-stream rate limits; but the result is that some data is lost. The per-stream limit also needs human intervention to change, which is not ideal when log volumes increase and decrease.

Loki uses automatic stream sharding to avoid rate limiting and large streams for any log stream by ensuring it is close to a configured desired_rate.

How automatic stream sharding works

Loki supports two independent sharding mechanisms, rate-based sharding and time-based sharding. You can enable either one on its own, or enable both together.

Rate-based sharding

Rate-based sharding works by adding a new label, __stream_shard__, to streams and incrementing its value to try and keep all streams below a configured desired_rate.

The feature adds a new API to Ingesters that reports the size of all existing log streams. Once per second, Distributors query the API to get a picture of all stream rates in the system. Distributors use the existing stream-rate data and a configured desired_rate to determine how many shards a given stream should have. The desired number of new log streams are created with the label __stream_shard__ and logs are divided evenly among the streams.

The once-per-second query interval is a Distributor setting, not a shard_streams limit. Use the -distributor.rate-store.stream-rate-update-interval flag, or the equivalent rate_store block in the Distributor configuration, to change how often Distributors refresh their view of stream rates.

Because rate-based sharding is reactive and relies on successive calls to Ingesters, the view of current rates is always somewhat behind. As a result, the actual size of sharded streams will always be higher than the desired_rate. In practice, this is still sufficient to keep log producers from being rate limited by per-stream rate limits.

Time-based sharding

Time-based sharding works by adding a different label, __time_shard__, to streams. Loki calculates the value of this label from each log entry's timestamp, divided into buckets of max_chunk_age/2. This lets Loki accept out-of-order log entries that are older than would normally be allowed, because each time bucket becomes its own stream instead of extending an existing one.

Loki does not apply time-based sharding to log entries with a timestamp newer than the configured time_sharding_ignore_recent value. Those entries are still ingested, but they keep their original labels, so that recent logs are not needlessly split into extra streams.

If you enable both rate-based sharding and time-based sharding, Loki applies time-based sharding first, and then applies rate-based sharding to each of the resulting time-sharded streams.

Automatic stream sharding metrics

Use these metrics to help tune Loki so that it is sharding streams aggressively enough to avoid the per-stream rate limit:

Manage varying workloads at scale with autoscaling queriers

A microservices deployment of a Loki cluster that runs on Kubernetes typically handles a workload that varies throughout the day. To make Loki easier to operate and optimize the cost of running Loki at scale, we have designed a set of resources to help you autoscale your Loki queriers.

Prerequisites

You need to run Loki in Kubernetes as a set of microservices. You need to use the query-scheduler.

We recommend using Kubernetes Event-Driven Autoscaling (KEDA) to configure autoscaling based on Prometheus metrics. Refer to Deploying KEDA to learn more about setting up KEDA in your Kubernetes cluster.

Scaling metric

Because queriers pull queries from the query-scheduler queue and process them on the querier workers, you should scale metrics based on:

The query-scheduler exposes the loki_query_scheduler_inflight_requests metric. It tracks the sum of queued queries plus the number of queries currently running in the querier workers. The following query is useful to scale queriers based on the inflight requests.

sum(
  max_over_time(
    loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="<Q>"}[<R>]
  )
)

Use the quantile (Q) and the range (R) parameters to fine-tune the metric. The higher Q is, the more sensitive the metric is to short-lasting spikes. As R increases, you can reduce the variation over time in the metric. A higher R-value helps avoid the autoscaler from modifying the number of replicas too frequently.

In our experience, we have found that a Q of 0.75 and an R of 2 minutes work well. You can adjust these values according to your workload.

loki_query_scheduler_inflight_requests only tracks a fixed set of quantiles: 0.5, 0.75, 0.8, 0.9, 0.95, and 0.99. If you query the metric with any other quantile value, Prometheus returns no data. Choose Q from this list.

The metric already smooths its value over a 60 second sliding window before Prometheus scrapes it. Because of this, an R value below about one minute adds little extra smoothing. We recommend keeping R at one minute or higher.

Cluster capacity planning

To scale the Loki queries, you configure the following settings:

Querier workers process queries from the queue. You can configure each Loki querier to run several workers with the -querier.max-concurrent flag (max_concurrent in YAML configuration), which defaults to 4. To reserve workforce headroom to address workload spikes, our recommendation is not to use more than 75% of the workers. For example, if you configure the Loki queriers to run 6 workers, set a threshold of floor(0.75 * 6) = 4.

-querier.max-concurrent sets the total number of workers for a querier, but the querier splits that total across its connections to query-schedulers (or query-frontends). If the configured value is lower than the number of query-schedulers a querier connects to, each connection still gets at least one worker, so the real worker count can be higher than the configured value. Check the actual number with the loki_querier_worker_concurrency metric before you calculate the threshold.

To determine the minimum number of queries that you should run, run at least one querier and determine the average number of inflight requests the system processes 75% of the time over seven days. The target utilization of the queries is 75%. So if we use 6 workers per querier, we will use the following query:

clamp_min(ceil(
    avg(
        avg_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.75"}[7d])
    ) / scalar(floor(vector(6 * 0.75)))
), 1)

The maximum number of queriers to run is equal to the number of queriers required to process all inflight requests 50% of the time during a seven-day timespan. As for the previous example, if each querier runs 6 workers, divide the inflight requests by 6. The resulting query becomes:

ceil(
    max(
        max_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.5"}[7d])
    ) / 6
)

To minimize the scenario where Loki scales up shortly after scaling down, set a stabilization window for scaling down.

KEDA configuration

If you deploy Loki with the Helm chart, you don't need to write your own KEDA ScaledObject. The chart can generate one for you through the querier.kedaAutoscaling values, which already default to a Prometheus trigger on loki_query_scheduler_inflight_requests with a Q of 0.75 and an R of 2 minutes. Set querier.kedaAutoscaling.enabled to true, then set minReplicas and maxReplicas for your workload. You also need to set defaults.kedaAutoscaling.prometheusAddress to the address of your Prometheus server, because it is empty by default and the trigger cannot query the metric without it. To replace the default trigger, set querier.kedaAutoscaling.triggers. Refer to Helm chart values for the full list of values.

If you don't use the Helm chart, or need a trigger the chart doesn't support, follow the example below to write your own KEDA ScaledObject.

This KEDA ScaledObject example configures autoscaling for the querier deployment in the loki-cluster namespace. The example shows the minimum number of replicas set to 10 and the maximum number of replicas set to 50. Because each querier runs 6 workers, aiming to use 75% of those workers, the threshold is set to 4. The metric is served at http://prometheus.default:9090/prometheus. We configure a stabilization window of 30 minutes.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: querier
  namespace: loki-cluster
spec:
  maxReplicaCount: 50
  minReplicaCount: 10
  scaleTargetRef:
    kind: Deployment
    name: querier
  triggers:
  - metadata:
      metricName: querier_autoscaling_metric
      query: sum(max_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.75"}[2m]))
      serverAddress: http://prometheus.default:9090/prometheus
      threshold: "4"
    type: prometheus
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 1800

Prometheus alerting when at capacity

Because the configured maximum might not be sufficient, a Prometheus alert can identify when the quantity of queriers has been at its configured maximum for an extended time. The following example specifies three hours (3h) as the extended time:

name: LokiAutoscalerMaxedOut
expr: kube_horizontalpodautoscaler_status_current_replicas{namespace=~"loki-cluster"} == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"loki-cluster"}
for: 3h
labels:
  severity: warning
annotations:
  description: HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} has been running at max replicas for longer than 3h; this can indicate underprovisioning.
  summary: HPA has been running at max replicas for an extended time

Block unwanted queries

In certain situations, you may not be able to control the queries being sent to your Loki installation. These queries may be intentionally or unintentionally expensive to run, and they may affect the overall stability or cost of running your service.

You can block queries using per-tenant overrides, like so:

overrides:
  "tenant-id":
    blocked_queries:
      # block this query exactly
      - pattern: 'sum(rate({env="prod"}[1m]))'

      # block any query matching this regex pattern 
      - pattern: '.*prod.*'
        regex: true

      # block all metric queries
      - types: metric

      # block any filter or limited queries matching this regex pattern 
      - pattern: '.*prod.*'
        regex: true
        types: filter,limited

      # block any query that matches this query hash
      - hash: 2943214005          # hash of {stream="stdout",pod="loki-canary-9w49x"}
        types: filter,limited

      # block queries originating from specific sources via X-Query-Tags
      # Keys and values are matched case-insensitively.
      - pattern: '.*'             # optional; if pattern and regex are omitted they will default to '.*' and true
        regex: true
        query_tags:
          source: grafana
          feature: beta

{{< admonition type="note" >}} Changes to these configurations do not require a restart; they are defined in the runtime configuration file. {{< /admonition >}}

The available query types are:

The hash option uses a 32-bit FNV-1 hash of the query string, represented as a 32-bit unsigned integer. This can often be easier to use than query strings that are long or require lots of string escaping. A query_hash field is logged with every query request in the query-frontend and querier logs, for easy reference. Here's an example log line:

level=info ts=2023-03-30T09:08:15.2614555Z caller=metrics.go:152 component=frontend org_id=29 latency=fast 
query="{stream=\"stdout\",pod=\"loki-canary-9w49x\"}" query_hash=2943214005 query_type=limited range_type=range ...

{{< admonition type="note" >}} The order of patterns is preserved, and Loki stops at the first rule whose pattern or hash matches the query text. If that rule's types or query_tags constraints then don't match, the query is not blocked, and Loki does not continue checking any later rules, even if a later rule would have matched and blocked the query.

For example, with this configuration:

overrides:
  "tenant-id":
    blocked_queries:
      - pattern: '{env="prod"}'
        types: metric
      - pattern: '{env="prod"}'

A query for {env="prod"} (a limited query) matches the first rule's pattern, but not its types: metric constraint, so it is not blocked. Loki does not fall through to the second rule, which has no types restriction and would otherwise have blocked it. To block a query with more than one rule that share a pattern, combine the constraints into a single rule instead of relying on a later rule as a fallback. {{< /admonition >}}

Observing blocked queries

Blocked queries are logged, as well as counted in the loki_blocked_queries metric on a per-tenant basis.

When a policy matches by pattern/hash/regex, Loki logs whether the query type and request tags matched that policy:

level=warn msg="query blocker matched with regex policy" user=29 type=metric pattern=".*rate\\(.*\\).*" query="sum(rate({app=\"foo\"}[5m]))" typesMatched=true tagsMatched=false blocked=false

If tag constraints fail to match, Loki emits a debug log showing the missing key and the raw header value that was received:

level=debug msg="query blocker tags mismatch: missing or mismatched key" key=feature tagsRaw="Source=grafana,Feature=alpha"

Scope

Query blocking is enforced by the LogQL query engine when it evaluates a query. This covers:

Query blocking does not apply to:

Tag-based blocking

You can scope a blocked query rule to requests that include specific key=value pairs in the X-Query-Tags header.

Examples:

overrides:
  tenant-a:
    blocked_queries:
      # Block only metric queries from a beta feature flag
      - types: metric
        query_tags:
          feature: beta

      # Combine with regex to narrow scope further
      - pattern: '.*rate\\(.*\\).*'
        regex: true
        query_tags:
          source: grafana

Ruler query tags

When the ruler evaluates alerting and recording rules in remote evaluation mode (-ruler.evaluation.mode=remote), it automatically sets X-Query-Tags on the query it sends to source=ruler,rule_name=<rule name>,rule_type=<rule type>. You can match on these tags to block or scope a policy to rule evaluations specifically, for example:

overrides:
  "tenant-id":
    blocked_queries:
      # block all queries that come from rule evaluation
      - query_tags:
          source: ruler

{{< admonition type="note" >}} The ruler doesn't set X-Query-Tags in local evaluation mode, so you can't match on these tags there.

Be careful when you match on rule_name, because a rule name that contains a comma or an equals sign doesn't survive tag parsing. A comma ends the value early, so rule_name=my,rule is read as rule_name=my. An equals sign makes the pair invalid, so rule_name=a=b is dropped completely. Refer to Tag-based blocking for the full parsing rules. {{< /admonition >}}

Manage bloom filter building and querying (Deprecated)

{{< admonition type="warning" >}} In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is deprecated and will be removed in a future release. Engineering and on-call support is not available. No SLA is provided. Note that this feature is intended for users who are ingesting more than 75TB of logs a month, as it is designed to accelerate queries against large volumes of logs. {{< /admonition >}}

Loki leverages bloom filters to speed up queries by reducing the amount of data Loki needs to load from the store and iterate through. Loki is often used to run "needle in a haystack" queries; these are queries where a large number of log lines are searched, but only a few log lines match the query. Some common use cases are searching all logs tied to a specific trace ID or customer ID.

An example of such queries would be looking for a trace ID on a whole cluster for the past 24 hours:

{cluster="prod"} | traceID="3c0e3dcd33e7"

Without accelerated filtering, Loki downloads all the chunks for all the streams matching {cluster="prod"} for the last 24 hours and iterates through each log line in the chunks, checking if the structured metadata key traceID with value 3c0e3dcd33e7 is present.

With accelerated filtering, Loki is able to skip most of the chunks and only process the ones where we have a statistical confidence that the structured metadata pair might be present.

To learn how to write queries to use bloom filters, refer to Query acceleration.

Enable bloom filters

{{< admonition type="warning" >}} Building and querying bloom filters are by design not supported in single binary deployment, it can only be used with distributed microservice mode. The reason is that bloom filters also come with a relatively high cost for both building and querying the bloom filters that only pays off at large scale deployments. {{< /admonition >}}

To start building and using blooms you need to:

# Configuration block for the bloom creation.
bloom_build:
  enabled: true
  planner:
    planning_interval: 8h
  builder:
    planner_address: bloom-planner.<namespace>.svc.cluster.local:9095

# Configuration block for bloom filtering.
bloom_gateway:
  enabled: true
  client:
    addresses: dnssrvnoa+_bloom-gateway-grpc._tcp.bloom-gateway-headless.<namespace>.svc.cluster.local

# Enable blooms creation and filtering for all tenants by default
# or do it on a per-tenant basis.
limits_config:
  bloom_creation_enabled: true
  bloom_split_series_keyspace_by: 1024
  bloom_gateway_enable_filtering: true

For more configuration options refer to the Bloom Gateway, Bloom Build and per tenant-limits configuration docs. We strongly recommend reading the whole documentation for this deprecated feature before using it.

Bloom Planner and Builder

Building bloom filters from the chunks in the object storage is done by two components: the Bloom Planner and the Bloom Builder, where the planner creates tasks for bloom building, and sends the tasks to the builders to process and upload the resulting blocks. Bloom filters are grouped in bloom blocks spanning multiple streams (also known as series) and chunks from a given day. To learn more about how blocks and metadata files are organized, refer to the Building blooms section below.

The Bloom Planner runs as a single instance and calculates the gaps in fingerprint ranges for a certain time period for a tenant for which bloom filters need to be built. It dispatches these tasks to the available builders. The planner also applies the blooms retention.

{{< admonition type="warning" >}} Do not run more than one instance of the Bloom Planner. {{< /admonition >}}

The Bloom Builder is a stateless horizontally scalable component and can be scaled independently of the planner to fulfill the processing demand of the created tasks.

You can find all the configuration options for these components in the Configure section for the Bloom Builder. Refer to the Enable bloom filters section above for a configuration snippet enabling this feature.

Retention

The Bloom Planner applies bloom block retention on object storage. Retention is disabled by default. When enabled, retention is applied to all tenants. The retention for each tenant is the longest of its configured general retention (retention_period) and the streams retention (retention_stream).

For example, in the following example, tenant A has a bloom retention of 30 days, and tenant B a bloom retention of 40 days for the {namespace="prod"} stream.

overrides:
    "A":
        retention_period: 30d
    "B":
        retention_period: 30d
        retention_stream:
            - selector: '{namespace="prod"}'
              priority: 1
              period: 40d

Planner and Builder sizing and configuration

The single planner instance runs the planning phase for bloom blocks for each tenant in the given interval and puts the created tasks to an internal task queue. Builders process tasks sequentially by pulling them from the queue. The amount of builder replicas required to complete all pending tasks before the next planning iteration depends on the value of -bloom-build.split-keyspace-by, the number of tenants, and the log volume of the streams.

The maximum block size is configured per tenant via -bloom-build.max-block-size. The actual block size might exceed this limit given that we append streams blooms to the block until the block is larger than the configured maximum size. The maximum size of a single stream's bloom is configured per tenant via -bloom-build.max-bloom-size. A stream whose bloom exceeds this size is discarded rather than added to a block. Blocks are created in memory and as soon as they are written to the object store they are freed. Chunks and TSDB files are downloaded from the object store to the file system. We estimate that builders are able to process 4MB worth of data per second per core.

Planning strategies

The Bloom Planner supports two strategies for splitting the bloom-building work into tasks, configured per tenant via bloom_planning_strategy:

A few additional per-tenant limits let you tune task scheduling, block encoding, and gateway prefetching:

Refer to the per-tenant limits configuration docs for the full list of flags, defaults, and descriptions.

Bloom Gateway

Bloom Gateways handle chunks filtering requests from the index gateway. The service takes a list of chunks and a filtering expression and matches them against the blooms, filtering out those chunks not matching the given filter expression.

This component is horizontally scalable and every instance only owns a subset of the stream fingerprint range for which it performs the filtering. The sharding of the data is performed on the client side using DNS discovery of the server instances and the jumphash algorithm for consistent hashing and even distribution of the stream fingerprints across Bloom Gateway instances.

You can find all the configuration options for this component in the Configure section for the Bloom Gateways. Refer to the Enable bloom filters section above for a configuration snippet enabling this feature.

Gateway sizing and configuration

Bloom Gateways use their local file system as a Least Recently Used (LRU) cache for blooms that are downloaded from object storage. The size of the blooms depend on the ingest volume and number of unique structured metadata key-value pairs, as well as on build settings of the blooms, namely false-positive-rate. With default settings, bloom filters make up <1% of the raw structured metadata size.

Since reading blooms depends heavily on disk IOPS, Bloom Gateways should make use of multiple, locally attached SSD disks (NVMe) to increase I/O throughput. Multiple directories on different disk mounts can be specified using the -bloom.shipper.working-directory setting when using a comma separated list of mount points, for example:

-bloom.shipper.working-directory="/mnt/data0,/mnt/data1,/mnt/data2,/mnt/data3"

Bloom Gateways need to deal with relatively large files: the bloom filter blocks. Even though the binary format of the bloom blocks allows for reading them into memory in smaller pages, the memory consumption depends on the number of pages that are concurrently loaded into memory for processing. The product of three settings control the maximum amount of bloom data in memory at any given time: -bloom-gateway.worker-concurrency, -bloom-gateway.block-query-concurrency, and -bloom.max-query-page-size.

Example, assuming 4 CPU cores:

-bloom-gateway.worker-concurrency=4      // 1x NUM_CORES
-bloom-gateway.block-query-concurrency=8 // 2x NUM_CORES
-bloom.max-query-page-size=64MiB

4 x 8 x 64MiB = 2048MiB

Here, the memory requirement for block processing is 2GiB. To get the minimum requirements for the Bloom Gateways, you need to double the value.

Building blooms

Bloom filters are built per stream and aggregated together into block files. Streams are assigned to blocks by their fingerprint, following the same ordering scheme as Loki’s TSDB and sharding calculation. This gives a data locality benefit when querying as streams in the same shard are likely to be in the same block.

In addition to blocks, builders maintain a list of metadata files containing references to bloom blocks and the TSDB index files they were built from. Gateways and the planner use these metadata files to discover existing blocks.

Every -bloom-build.planner.interval, the planner will load the latest TSDB files for all tenants for which bloom building is enabled, and compares the TSDB files with the latest bloom metadata files. If there are new TSDB files or any of them have changed, the planner will create a task for the streams and chunks referenced by the TSDB file.

The builder pulls a task from the planner's queue and processes the containing streams and chunks. For a given stream, the builder will iterate through all the log lines inside its new chunks and build a bloom for the stream. In case of changes for a previously processed TSDB file, builders will try to reuse blooms from existing blocks instead of building new ones from scratch. The builder converts structured metadata from each log line of each chunk of a stream and appends the hash of each key, and key-value pair to the bloom, followed by the hashes combined with the chunk identifier. The first set of hashes allows gateways to skip whole streams, while the latter is for skipping individual chunks.

For example, given structured metadata foo=bar in the chunk c6dj8g, we append to the stream bloom the following hashes: hash("foo"), hash("foo=bar"), hash("c6dj8g" + "foo") and hash("c6dj8g" + "foo=bar").

Query sharding

Query acceleration does not just happen while processing chunks, but also happens from the query planning phase where the query frontend applies query sharding. Loki 3.0 introduces a new per-tenant configuration flag tsdb_sharding_strategy which defaults to computing shards as in previous versions of Loki by using the index stats to come up with the closest power of two that would optimistically divide the data to process in shards of roughly the same size. Unfortunately, the amount of data each stream has is often unbalanced with the rest, therefore, some shards end up processing more data than others.

Query acceleration introduces a new sharding strategy: bounded, which uses blooms to reduce the chunks to be processed right away during the planning phase in the query frontend, as well as evenly distributes the amount of chunks each sharded query will need to process.

Configure caches to speed up queries

Loki supports caching for query results, chunks, and index lookups to speed up query performance and reduce calls to the storage layer. Memcached is included in the Loki Helm chart and enabled by default for the chunksCache and resultsCache. This section describes the recommended Memcached configuration to enable caching for chunks and query results.

Cache backends

Every cache in Loki (chunk_cache_config, results_cache, and so on) can use one of three interchangeable backends: Memcached, Redis, or an embedded (in-process) cache. Memcached is the backend used throughout this page and is the recommended choice for production, multi-replica deployments, because it is shared across all instances of a component. Redis is supported as an alternative external cache. The embedded cache stores data in the memory of the Loki process itself, so it isn't shared between replicas; it is best suited to single-binary or small deployments. Configure only one backend per cache. If you configure both Memcached and Redis for the same cache, Loki fails to start.

If you don't configure Memcached or Redis for a cache, Loki automatically enables the embedded cache for it, so that caching still works out of the box. This applies to the chunks cache and to the results caches. It does not apply to the index queries cache or the write dedupe cache, which stay disabled unless you configure them. Because the embedded cache is per-process, this automatic fallback is not a substitute for Memcached or Redis in a deployment with more than one replica.

For more information about which caching backend to use for each cache, refer to Configure caches and Components. For help diagnosing cache configuration problems, refer to Troubleshoot operations.

Results cache

The results cache stores results so that Loki can reuse them on subsequent queries, and it supports negative caching for log queries. It is sometimes called the frontend cache in some configurations. The results cache is consulted by query-frontends. If the cached results are incomplete, the query frontend calculates the required sub-queries and sends them further along to be executed in queriers, then also caches those results. To orchestrate all of the above, the results cache uses a query hash as the key that is computed and stored in the headers.

The results cache is really a set of six independently configurable caches under query_range, one per query type:

Query type Enable with Cache config
Metric and log queries cache_results results_cache
Index-stats queries cache_index_stats_results index_stats_results_cache
Volume queries cache_volume_results volume_results_cache
Instant metric queries cache_instant_metric_results instant_metric_results_cache
Series queries cache_series_results series_results_cache
Label queries cache_label_results label_results_cache

If a per-query-type cache is enabled but its cache config is left empty, Loki falls back to the results_cache configuration for that query type. This means you only need to configure results_cache to get caching for all query types, and you only need to give a query type its own cache config if you want it to use a different Memcached, Redis, or embedded cache instance. For details of each supported request type, refer to the Components section.

The index lookup cache only supports the legacy BoltDB index storage and is configured to be in-memory by default. Since moving to the TSDB indexes the attached disks/persistent volumes are utilized as cache and in-memory index lookup cache is obsolete.

Chunks cache

The chunks are cached using the chunkRef as the cache key, which is the unique reference to a chunk when it's cut in the Loki ingesters. The chunk cache is consulted by queriers each time a set of chunkRefs are calculated to serve the query, before going to the storage layer.

Query results are significantly smaller compared to chunks. As the Loki cluster gets bigger in ingested volume, the results cache can continue to perform, whereas the chunks cache will need to grow in proportion to demand more memory. To be able to support the growing needs of a cluster, in 2023 we introduced support for memcached-extstore. Extstore is an additional feature on Memcached which supports attaching SSD disks to memcached pods to maximize their capacity.

Please see this blog post on Loki's experience with memcached-extstore for our SaaS offering, Grafana Cloud. For more information on how to tune memcached-extstore please consult the open source memcached documentation.

L2 chunks cache

As an alternative, or a complement, to memcached-extstore, Loki supports a second, "L2" chunk cache tier configured with chunk_cache_config_l2 and l2_chunk_cache_handoff. The l2_chunk_cache_handoff setting is an age threshold. Chunks younger than this age are written to and read from the primary (L1) cache that you configure in chunk_cache_config. Chunks older than this age are written to and read from the L2 cache instead. A value of 0 disables the L2 cache.

Loki does not copy or move entries between the two tiers. Each chunk is routed to one tier or the other based on its age at the time of the request. On the read path, Loki widens the threshold by 10% so that chunks near the boundary are still looked up in the L1 cache.

This lets you keep a small, fast cache of recent chunks while routing older, less frequently accessed chunks to a larger and cheaper second tier, such as a disk-backed Memcached instance.

chunk_store_config:
  l2_chunk_cache_handoff: 24h
  chunk_cache_config_l2:
    memcached:
      batch_size: 256
      parallelism: 10
    memcached_client:
      host: <l2 chunk cache memcached host>
      service: <port name of memcached service>

If the Helm chart is used, the L2 cache is configured under chunksCache.l2, which is disabled by default (chunksCache.l2.enabled: false) and defaults to a four day handoff (chunksCache.l2.l2ChunkCacheHandoff: 345600s). The Helm chart's chunksCache.l2.persistence settings can attach a persistent volume to the L2 cache pods, which is a chart-native alternative to memcached-extstore for adding disk-backed capacity.

Before you begin

Steps

To enable and configure Memcached:

  1. Deploy each Memcached service with at least three replicas and configure each as follows:

    1. Chunk cache

      --memory-limit=4096 --max-item-size=2m --conn-limit=1024

    2. Query result cache

      --memory-limit=1024 --max-item-size=5m --conn-limit=1024

  2. Configure Loki to use the cache.

    1. If the Helm chart is used

      Set chunksCache.addresses to the Memcached address for the chunk cache and resultsCache.addresses to the Memcached address for the query result cache, then set chunksCache.enabled=true and resultsCache.enabled=true.

      Ensure that the connection limit of Memcached is at least number_of_clients * max_idle_conns.

      By default, the chart also deploys and manages the Memcached servers themselves. To use your own Memcached instances instead, set memcached.enabled to false to stop the chart from deploying its built-in Memcached server, keep chunksCache.enabled and resultsCache.enabled set to true so that Loki still uses a Memcached-based cache, and point the addresses values at your own service. Use the same comma-separated DNS Service Discovery format that the chart uses by default:

      memcached:
  enabled: false
chunksCache:
  enabled: true
  addresses: "dnssrvnoa+_memcached-client._tcp.chunk-cache-memcached.loki.svc"
resultsCache:
  enabled: true
  addresses: "dnssrvnoa+_memcached-client._tcp.results-cache-memcached.loki.svc"

      The Helm chart's chunksCache and resultsCache values only manage the chunk cache and the main query-range results cache. The other five results caches (index_stats_results_cache, volume_results_cache, instant_metric_results_cache, series_results_cache, and label_results_cache) don't have dedicated Helm values. You can still enable and configure them by adding the settings to the chart's loki.query_range value, which the chart copies into the Loki configuration file as it is written. For example:

      loki:
  query_range:
    cache_index_stats_results: true
    cache_volume_results: true

    2. If the Loki configuration is used, modify the following two sections in the Loki configuration file.

      1. Configure the chunk cache

        chunk_store_config:
  chunk_cache_config:
    memcached:
      batch_size: 256
      parallelism: 10
    memcached_client:
      host: <chunk cache memcached host>
      service: <port name of memcached service>

      2. Configure the query result cache

        query_range:
  cache_results: true
  results_cache:
    cache:
      memcached_client:
        consistent_hash: true
        host: <memcached host>
        service: <port name of memcached service>
        max_idle_conns: 16
        timeout: 200ms
        update_interval: 1m

Kafka-based ingestion (Experimental)

{{< admonition type="warning" >}} Kafka-based ingestion is an experimental feature. Engineering and on-call support is not available. No SLA is provided. Configuration options and metric names may change in a future release. {{< /admonition >}}

Grafana Loki can use Apache Kafka, or a Kafka-protocol-compatible system such as WarpStream, as an optional durable buffer on the write path. This topic explains what Kafka-based ingestion is, when to use it, how it works, and how to configure and monitor it.

What it is

By default, the distributor sends log streams directly to ingesters over gRPC. When you enable Kafka-based ingestion, the distributor instead (or also) writes log streams as records to a Kafka topic. Each ingester then reads, or consumes, records from one partition of that topic and processes them the same way it processes directly pushed data.

Kafka sits between the distributor and the ingester as a durable, replicated buffer. This decouples the two components so that a slow or restarting ingester doesn't block or fail writes.

Loki uses the franz-go Kafka client library. If you need to debug Kafka client behavior in your logs, look for the component name kafka_client. Kafka client metrics use the loki_kafka_client_ prefix.

When and why to use it

Consider enabling Kafka-based ingestion if you want to:

Kafka-based ingestion also underpins the ingest-limits service and the experimental data object builders used for columnar storage. These features require Kafka even if you don't otherwise need the durability benefits above.

If you don't have an existing Kafka deployment and don't need these benefits, you can continue to use the default gRPC write path. Enabling Kafka adds an operational dependency: you must run, or pay for, a Kafka cluster (or a compatible managed service) and monitor it.

Architecture summary

{{< mermaid >}} flowchart LR A[Distributor] -->|produces records| B[(Kafka topic, partitioned)] B -->|partition 0| C1[Ingester A] B -->|partition 1| C2[Ingester B] B -->|partition N| C3[Ingester N] D[Partition ring] -.assigns partition ownership.-> C1 D -.-> C2 D -.-> C3 {{< /mermaid >}}

The distributor writes each stream as one or more Kafka records to a topic. The topic has multiple partitions. Each ingester owns exactly one partition and consumes records only from that partition. Ingesters use the partition ring to coordinate which ingester owns which partition.

Every ingester process always registers in the classic hash ring it uses for gRPC writes, whether or not Kafka-based ingestion is enabled. When you also enable ingester.kafka_ingestion.enabled, that same process additionally joins the partition ring. Loki supports two ways to deploy this:

Because the two rings assign ownership of a tenant's streams differently (token-hash with replication, compared to shuffle-sharded partitions with a single owner), the querier has to be told explicitly which ring to use when it looks up ingesters. See Read path and querier.query_partition_ingesters under Configuration overview.

{{< admonition type="note" >}} You can use any system that implements the Kafka wire protocol, such as WarpStream, instead of running Apache Kafka yourself. Loki only needs the broker address and, optionally, SASL credentials. No code changes are required. {{< /admonition >}}

Read path

Enabling Kafka-based ingestion primarily changes the write path. LogQL queries still read recent data from ingester memory and older data from long-term object storage, the same way they do when Kafka-based ingestion is disabled.

However, the querier must be told which ring to use to find the ingesters that hold recent data: the classic hash ring (the default), or the partition ring that Kafka-based ingestion uses. Set querier.query_partition_ingesters: true to switch the querier, and any locally evaluating rulers, over to the partition ring. Until you set this, the querier keeps using the classic ring even if the distributor and ingesters are fully configured for Kafka-based ingestion. For the full migration sequence and when to flip this setting, refer to Migrate to Kafka-based ingestion.

Configuration overview

This section groups the relevant configuration blocks at a high level. For the full list of fields and their defaults, refer to the Configuration reference.

Authentication limitations

The only Kafka authentication mechanism Loki supports today is SASL PLAIN, configured with kafka_config.sasl_username and kafka_config.sasl_password. Both fields must be set together, or both left empty.

Loki has no built-in Transport Layer Security (TLS) or mutual TLS (mTLS) configuration for the Kafka connection. If your organization requires an encrypted connection to Kafka, you must terminate TLS at the network layer, for example with a sidecar proxy or a load balancer in front of your Kafka brokers.

Partition sizing guidance

The number of Kafka partitions in your topic limits how many ingesters can actively consume from it, because each ingester owns exactly one partition.

Keep the following in mind when you choose a partition count:

Offset management and restart behavior

When Kafka-based ingestion is enabled, each ingester follows this sequence to safely resume consuming after a restart:

  1. On startup, the ingester fetches the last offset it committed for its partition.
  2. If no committed offset exists, for example on first startup, the ingester starts from the earliest available offset.
  3. The ingester replays Kafka records from that offset until its consumption lag is within kafka_config.max_consumer_lag_at_startup (default: 15 seconds).
  4. Only after catching up does the ingester mark itself ACTIVE in the partition ring and pass its readiness check.
  5. During normal operation, the ingester commits its current offset to Kafka every kafka_config.consumer_group_offset_commit_interval (default: 1 second).

Because of this sequence, ingester restarts are safe: any data produced to Kafka while an ingester was down is still in the topic, and the ingester automatically replays it after it comes back up.

Monitoring Kafka ingestion

This is the full list of Kafka-related metrics, organized by component. For the highest-signal subset of these metrics and example alerting queries, refer to Key metrics for monitoring Loki.

Distributor (producer side):

Ingester (consumer side):

Kafka client internals (reader and writer):

Configuration examples

Dual-write for a safe rollout

Write to both Kafka and ingesters at the same time so you can validate Kafka-based ingestion before relying on it. Queries still use the classic hash ring at this stage (querier.query_partition_ingesters stays false).

distributor:
  kafka_writes_enabled: true
  ingester_writes_enabled: true

ingester:
  kafka_ingestion:
    enabled: true

kafka_config:
  topic: loki-logs
  reader_config:
    address: kafka:9092
  writer_config:
    address: kafka:9092

Kafka-only write path

Once you've validated dual-write and switched queriers to the partition ring, you can disable the direct gRPC writes to ingesters. Refer to Migrate to Kafka-based ingestion for the full, safely ordered sequence.

distributor:
  kafka_writes_enabled: true
  ingester_writes_enabled: false

ingester:
  kafka_ingestion:
    enabled: true

querier:
  query_partition_ingesters: true

kafka_config:
  topic: loki-logs
  reader_config:
    address: kafka:9092
  writer_config:
    address: kafka:9092

Managed Kafka with SASL authentication

This example connects to a managed Kafka-protocol-compatible service, such as WarpStream, using SASL PLAIN authentication and separate reader and writer broker addresses.

distributor:
  kafka_writes_enabled: true
  ingester_writes_enabled: true

ingester:
  kafka_ingestion:
    enabled: true

kafka_config:
  topic: loki-logs
  reader_config:
    address: reader.kafka.example.com:9092
  writer_config:
    address: writer.kafka.example.com:9092
  sasl_username: ${KAFKA_SASL_USERNAME}
  sasl_password: ${KAFKA_SASL_PASSWORD}

Use label-based access control with Loki

{{< admonition type="note" >}} This feature is generally available from v4.0. For the latest releases, refer to the Release notes. {{< /admonition >}}

Label-based access control (LBAC) restricts the logs a tenant can query to those that match one or more Prometheus label selectors. You associate a set of selectors with a tenant, and queries from that tenant only return data from log streams that match at least one of the selectors. Because selectors can be combined into "OR" groups, where each group is a list of things joined by "AND" you can express any required policy. For example, (A and B and C) or (D and E) or (F).

LBAC builds on Loki's multi-tenancy: policies are scoped per tenant, so you must run Loki in multi-tenant mode. For more information, refer to the multi-tenancy documentation.

Why use LBAC

LBAC lets you share a single Loki tenant across multiple teams or users while still limiting what each of them can see. Common reasons to use LBAC include:

Security model

Loki does not include its own authentication layer. Both the tenant (X-Scope-OrgID) and the label policy (X-Prom-Label-Policy) are read from trusted HTTP headers on incoming requests.

{{< admonition type="warning" >}} Never expose the Loki API directly to clients. A client that can reach Loki can set the X-Scope-OrgID and X-Prom-Label-Policy headers itself and bypass label-based access control entirely. You must run an authentication gateway in front of Loki that authenticates the request, sets the tenant, and attaches the correct label policy. {{< /admonition >}}

The gateway is responsible for stripping any user supplied X-Scope-OrgID and X-Prom-Label-Policy and replacing them with values it derives from the authenticated identity. One open-source gateway that sets these headers is db-auth-gateway. For general guidance on putting an authenticating reverse proxy in front of Loki, refer to the authentication documentation.

How LBAC works

At a high level, your gateway attaches a label policy to each request, and Loki carries that policy through the read path so that every component filters data to match it.

  1. Your authentication gateway authenticates the request and sets the X-Prom-Label-Policy HTTP header. Each value in the header has the form <TENANT>:<URL-ENCODED SELECTOR>, for example tenant1:%7Benv%3D%22dev%22%7D for the selector {env="dev"}.
  2. An HTTP middleware inside Loki parses this header on every incoming request and stores the resulting policies in the request context.
  3. As the request moves between Loki's internal components, for example from the query frontend to the queriers, ingesters, and index gateway, the policies travel with it as gRPC metadata. This means the same policy applies no matter which component ultimately reads the data.
  4. Loki enforces the policy at several points in the read path:
    • It filters chunks and streams while running the query, for both the original query engine and the newer query engine.
    • It filters the results of label-values queries against the store and the ingesters.
    • In the query frontend, it filters the response of volume queries, and it rewrites aggregated-metrics queries so they only return permitted data.

The following diagram shows the request flow for a query from a client through the gateway and into Loki:

sequenceDiagram
    participant Client
    participant Gateway as Authentication gateway
    participant QF as Query frontend
    participant Q as Querier / Ingester

    Client->>Gateway: Query request
    Gateway->>Gateway: Authenticate request
    Gateway->>QF: Query request + X-Scope-OrgID + X-Prom-Label-Policy
    QF->>Q: Query request (policy propagated over gRPC)
    Q->>Q: Filter streams and chunks by policy
    Q->>QF: Filtered results
    QF->>Gateway: Filtered response
    Gateway->>Client: Filtered response

LBAC and ingestion

Label-based access control is not enforced on write (push) requests. It only restricts what a tenant can read. A tenant that is allowed to write can push log lines with any labels, regardless of the label policy that applies to its queries.

LBAC and alertmanager and ruler

Label policies are not enforced by the Alertmanager or ruler HTTP endpoints. This means that the requests they serve contain everything for a particular tenant without applying label-based access control. For example, listing all rule groups in the ruler returns all rule groups for the tenant, even if a label selector in the policy would exclude some of the labels on the rules. However it does apply when you query the metrics those components generate, such as ALERTS.

LBAC and querying

LogQL queries use the following format:

{ log stream selector } | log pipeline

LBAC only applies to the log stream selector portion of a query, that is the part of the query that comes before the pipe (|). It does not apply to label filter expressions or other parts of the log pipeline.

For more information, refer to the Log stream selector and Label filter expression sections in the documentation about Log queries.

Enable label-based access control

LBAC requires multi-tenant mode, which is the default. Ensure authentication is enabled so that requests carry a tenant:

auth_enabled: true

Then enable LBAC:

lbac:
  enabled: true

This can also be set with the -lbac.enabled command-line flag. When enabled, Loki parses the X-Prom-Label-Policy header on incoming requests and enforces the policies it contains.

Configuration reference

LBAC currently has a single configuration setting. There are no per-tenant settings in Loki's YAML configuration: policies for each tenant arrive dynamically in the X-Prom-Label-Policy header, set by your authentication gateway, not through Loki's configuration file.

lbac:
  # Enables label based access control through the X-Prom-Label-Policy header.
  [enabled: <boolean> | default = false]
Setting CLI flag Default Description
lbac.enabled -lbac.enabled false Enables label-based access control through the X-Prom-Label-Policy header. This setting is experimental.

Setting up a label policy

A label policy is conveyed to Loki in the X-Prom-Label-Policy request header. This header is set by your authentication gateway, not by clients. Each header value has the form:

<TENANT>:<URL-ENCODED SELECTOR>

The selector is a standard label matcher set such as {env="dev"}, URL-encoded so that it is safe to carry in a header. For example, the policy {env="dev"} for tenant tenant1 is encoded as:

X-Prom-Label-Policy: tenant1:%7Benv%3D%22dev%22%7D

To associate multiple selectors with a tenant, set multiple values. You can either repeat the header or separate the encoded policies with a comma (,). The selectors are combined with OR into groups where each group is a list of things joined by AND: a stream is returned if any of the streams matches any one of the selector groups. For example, a policy written {a='true', b='true, c='true'} OR {x='true', y='true'} reads as {a='true' AND b='true'AND c='true'} OR {x='true'AND y='true'}.

Exclude a label

One common use case for an LBAC policy is to exclude logs that have a specific label. For example, to exclude all log lines with the label secret=true, use a selector with secret!="true":

{secret!="true"}

Use multiple selectors

To allow access to both the production and development environments while excluding logs with the label secret=true in the production environment, use multiple selectors:

{secret!="true", env="prod"}
{env="dev"}

These selectors enforce the policy as follows:

Real-world examples

The following examples use labels that Loki assigns by default when logs are collected through OpenTelemetry, such as service_name, k8s_namespace_name, and deployment_environment_name. For the full list of default OpenTelemetry-derived labels, refer to Default labels. Each example shows the LogQL selector and the resulting X-Prom-Label-Policy header value that your gateway would set, following the same format as Setting up a label policy.

Restrict a team to one namespace

Scope a tenant so that it can only see logs from the checkout Kubernetes namespace. This is useful when a platform team gives a single application team access to a shared tenant, but the application team should only see its own namespace.

Selector:

{k8s_namespace_name="checkout"}

Header:

X-Prom-Label-Policy: team-checkout:%7Bk8s_namespace_name%3D%22checkout%22%7D

Restrict by environment

Give separate policies to a production support team and a staging support team that share the same tenant.

Production team selector:

{deployment_environment_name="production"}

Header:

X-Prom-Label-Policy: team-prod:%7Bdeployment_environment_name%3D%22production%22%7D

Staging team selector:

{deployment_environment_name="staging"}

Header:

X-Prom-Label-Policy: team-staging:%7Bdeployment_environment_name%3D%22staging%22%7D

Restrict by service, with an exclusion

Combine a service restriction with an exclusion, similar to Use multiple selectors, but scoped to a specific service.

Allow access to the payments-api service, but exclude any of its streams labeled secret="true", and separately allow access to every other service:

{service_name="payments-api", secret!="true"}
{service_name!="payments-api"}

Header:

X-Prom-Label-Policy: team-payments:%7Bservice_name%3D%22payments-api%22%2C%20secret%21%3D%22true%22%7D,team-payments:%7Bservice_name%21%3D%22payments-api%22%7D

Restrict by region for a distributed team

Scope a regional on-call team so that it only sees logs from its own region.

Selector:

{cloud_region="us-west-1"}

Header:

X-Prom-Label-Policy: team-uswest:%7Bcloud_region%3D%22us-west-1%22%7D

Combine selectors across multiple namespaces

Grant a tenant access to logs from two Kubernetes namespaces by giving it two separate selectors for the same tenant:

{k8s_namespace_name="frontend"}
{k8s_namespace_name="checkout"}

Because a tenant's selectors are combined with OR, a stream is returned if it matches either one: the frontend namespace or the checkout namespace.

Header:

X-Prom-Label-Policy: team-multi:%7Bk8s_namespace_name%3D%22frontend%22%7D,team-multi:%7Bk8s_namespace_name%3D%22checkout%22%7D

Manage tenant isolation

Grafana Loki is a multi-tenant system; requests and data for tenant A are isolated from tenant B. Requests to the Loki API should include an HTTP header (X-Scope-OrgID) that identifies the tenant for the request.

Tenant IDs must not be longer than 150 bytes and can only include a specific set of characters; see Restrictions below for the full rules. Operators are recommended to use a reasonable limit for uniquely identifying tenants; 20 bytes is usually enough.

Loki defaults to running in multi-tenant mode. Multi-tenant mode is set in the configuration with auth_enabled: true, or with the equivalent command line flag -auth.enabled (default true).

When configured with auth_enabled: false, Loki uses a single tenant. The X-Scope-OrgID header is not required in Loki API requests. The single tenant ID defaults to fake and can be changed via the no_auth_tenant configuration option. On a fresh cluster with an empty bucket this is safe to change at any time; on a cluster that has already written data, changing the tenant ID requires migrating existing data to the new tenant path (see Loki Migrate Tool).

When auth_enabled: true, a request that does not include the X-Scope-OrgID header fails with an HTTP 401 Unauthorized error and the message no org id. For more information about setting this header, including examples of configuring a reverse proxy to add it, refer to Manage authentication. To troubleshoot this error, refer to No org ID.

Multi-tenant Queries

In multi-tenant mode, queries may gather results from multiple tenants. Set the querier configuration option multi_tenant_queries_enabled: true, or the equivalent command line flag -querier.multi-tenant-queries-enabled (default false), to enable queries across tenants. The query API request defines the tenants. Specify multiple tenants in the query request HTTP header X-Scope-OrgID by separating the tenant IDs with the pipe character (|). For example, a query for tenants A and B requires the header X-Scope-OrgID: A|B.

Only query endpoints support multi-tenant calls. Calls to GET /loki/api/v1/tail, POST /loki/api/v1/push, and POST /otlp/v1/logs will return an HTTP 400 error if more than one tenant is defined in the HTTP header.

Instant and range queries support label filtering using tenant IDs. For example, the query

{app="foo", __tenant_id__=~"a.+"} | logfmt

will return results for all tenants that have a tenant ID that begins with the character a.

The __tenant_id__ label is not stored with the log data. Loki adds it to the results only when a query spans more than one tenant. If the request names a single tenant, the results do not include this label.

If the label __tenant_id__ is already present in a log stream, it is prepended with the string original_.

Tenant ID filtering in stages is not supported. An example of a query that will not work:

{app="foo"} | __tenant_id__="1" | logfmt

Restrictions

Tenant IDs must not be longer than 150 bytes and can only include the following characters:

{{< admonition type="note" >}} For security reasons, . and .. aren't valid tenant IDs. {{< /admonition >}}

{{< admonition type="warning" >}} Do not include a colon (:) in the X-Scope-OrgID header. A colon is not a valid tenant ID character. Most Loki components stop reading the tenant ID at the first colon and ignore the rest of the value, and the ignored part is not checked against the rules on this page. Other components read the whole header value instead, so a header that contains a colon can produce inconsistent results. {{< /admonition >}}

Monitor tenant limits using the Overrides Exporter

Loki is a multi-tenant system that supports applying limits to each tenant as a mechanism for resource management. The overrides-exporter module exposes these limits as Prometheus metrics in order to help operators better understand tenant behavior.

Context

Configuration updates to tenant limits can be applied to Loki without restart via the runtime_config feature.

Example

The overrides-exporter module is disabled by default. We recommend running a single instance per cluster to avoid issues with metric cardinality. The overrides-exporter creates one metric for every scalar field in the limits configuration under the metric loki_overrides_defaults with the default value for that field after loading the Loki configuration. It also exposes another metric for every differing field for every tenant.

Using an example runtime.yaml:

overrides:
  "tenant_1":
    ingestion_rate_mb: 10
    max_streams_per_user: 100000
    max_chunks_per_query: 100000

Launch an instance of the overrides-exporter:

loki -target=overrides-exporter -runtime-config.file=runtime.yaml -config.file=basic_schema_config.yaml -server.http-listen-port=8080

To inspect the tenant limit overrides:

$ curl -sq localhost:8080/metrics | grep override
# HELP loki_overrides Resource limit overrides applied to tenants
# TYPE loki_overrides gauge
loki_overrides{limit_name="ingestion_rate_mb",user="tenant_1"} 10
loki_overrides{limit_name="max_chunks_per_query",user="tenant_1"} 100000
loki_overrides{limit_name="max_streams_per_user",user="tenant_1"} 100000
# HELP loki_overrides_defaults Default values for resource limit overrides applied to tenants
# TYPE loki_overrides_defaults gauge
loki_overrides_defaults{limit_name="block_ingestion_status_code"} 260
loki_overrides_defaults{limit_name="cardinality_limit"} 100000
loki_overrides_defaults{limit_name="creation_grace_period"} 6e+11
loki_overrides_defaults{limit_name="ingestion_burst_size_mb"} 6
loki_overrides_defaults{limit_name="ingestion_rate_mb"} 4
loki_overrides_defaults{limit_name="max_cache_freshness_per_query"} 6e+11
loki_overrides_defaults{limit_name="max_chunks_per_query"} 2e+06
loki_overrides_defaults{limit_name="max_concurrent_tail_requests"} 10
loki_overrides_defaults{limit_name="max_entries_limit_per_query"} 5000
loki_overrides_defaults{limit_name="max_global_streams_per_user"} 5000
loki_overrides_defaults{limit_name="max_label_name_length"} 1024
loki_overrides_defaults{limit_name="max_label_names_per_series"} 15
loki_overrides_defaults{limit_name="max_label_value_length"} 2048
loki_overrides_defaults{limit_name="max_line_size"} 262144
loki_overrides_defaults{limit_name="max_metadata_cache_freshness"} 8.64e+13
loki_overrides_defaults{limit_name="max_queriers_per_tenant"} 0
loki_overrides_defaults{limit_name="max_query_capacity"} 0
loki_overrides_defaults{limit_name="max_query_length"} 2.5956e+15
loki_overrides_defaults{limit_name="max_query_lookback"} 0
loki_overrides_defaults{limit_name="max_query_parallelism"} 32
loki_overrides_defaults{limit_name="max_query_series"} 500
loki_overrides_defaults{limit_name="max_streams_matchers_per_query"} 1000
loki_overrides_defaults{limit_name="max_streams_per_user"} 0
loki_overrides_defaults{limit_name="max_structured_metadata_size"} 65536
loki_overrides_defaults{limit_name="min_sharding_lookback"} 0
loki_overrides_defaults{limit_name="per_stream_rate_limit"} 3.145728e+06
loki_overrides_defaults{limit_name="per_stream_rate_limit_burst"} 1.572864e+07
loki_overrides_defaults{limit_name="per_tenant_override_period"} 1e+10
loki_overrides_defaults{limit_name="query_timeout"} 6e+10
loki_overrides_defaults{limit_name="reject_old_samples_max_age"} 6.048e+14
loki_overrides_defaults{limit_name="retention_period"} 0
loki_overrides_defaults{limit_name="ruler_max_rule_groups_per_tenant"} 0
loki_overrides_defaults{limit_name="ruler_max_rules_per_rule_group"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_batch_send_deadline"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_capacity"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_backoff"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_samples_per_send"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_shards"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_backoff"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_shards"} 0
loki_overrides_defaults{limit_name="ruler_remote_write_timeout"} 0
loki_overrides_defaults{limit_name="split_queries_by_interval"} 3.6e+12
loki_overrides_defaults{limit_name="tsdb_max_bytes_per_shard"} 6.291456e+08
loki_overrides_defaults{limit_name="tsdb_max_query_parallelism"} 128
loki_overrides_defaults{limit_name="volume_enabled"} 1
loki_overrides_defaults{limit_name="volume_max_series"} 1000

{{< admonition type="note" >}} This output is a partial example. The exact set of metrics depends on your Loki version, since loki_overrides_defaults includes one series for every scalar field in the limits_config block, and that block grows with each release. Query your own instance's /metrics endpoint for the authoritative, current list rather than relying on a fixed example. {{< /admonition >}}

Alerts can be created based on these metrics to inform operators when tenants are close to hitting their limits allowing for increases to be applied before the tenant limits are exceeded.

Manage recording rules

Recording rules are queries that run in an interval and produce metrics from logs that can be pushed to a Prometheus compatible backend.

Recording rules are evaluated by the ruler component. Each ruler acts as its own querier, in the sense that it executes queries against the store without using the query-frontend or querier components. It will respect all query limits put in place for the querier.

The Loki implementation of recording rules largely reuses Prometheus' code.

Samples generated by recording rules are sent to Prometheus using Prometheus' remote-write feature.

Write-Ahead Log (WAL)

All samples generated by recording rules are written to a WAL. The WALs main benefit is that it persists the samples generated by recording rules to disk, which means that if your ruler crashes, you won't lose any data. We are trading off extra memory usage and slower start-up times for this functionality.

A WAL is created per tenant; this is done to prevent cross-tenant interactions. If all samples were to be written to a single WAL, this would increase the chances that one tenant could cause data-loss for others. A typical scenario here is that Prometheus will, for example, reject a remote-write request with 100 samples if just 1 of those samples is invalid in some way.

Start-up

When the ruler starts up, it deletes (wipes) its remote-write WAL directory and starts each tenant with a fresh, empty WAL. WAL data left over from a previous run is not replayed, so any samples that had not yet been flushed to remote storage before the restart are discarded.

Truncation

WAL files are regularly truncated to reduce their size on disk. This guide from one of the Prometheus maintainers (Ganesh Vernekar) gives an excellent overview of the truncation, checkpointing, and replaying of the WAL.

Cleaner

The WAL Cleaner watches for abandoned WALs (tenants who no longer have recording rules associated) and deletes them. Enable this feature only if you are running into storage concerns with WALs that are too large. WALs should not grow excessively large due to truncation.

The WAL Cleaner is disabled by default. Enable it by setting -ruler.wal-cleaner.period to a non-zero value.

Scaling

See Mimir's guide for configuring Grafana Mimir hash rings for scaling the ruler using a ring.

{{< admonition type="note" >}} By default, the ruler shards by rule group, not by individual rules. This is an artifact of the fact that Prometheus recording rules need to run in order since one recording rule can reuse another, but this is not possible in Loki. To shard by individual rule instead, set -ruler.sharding-algo="by-rule". For details, refer to Tuning. {{< /admonition >}}

Deployment

The ruler needs to persist its WAL files to disk, and it incurs a bit of a start-up cost by reading these WALs into memory. As such, it is recommended that you try to minimize churn of individual ruler instances, since recording rule samples produced while a tenant's WAL is being read from disk are dropped rather than persisted (see Start-up).

Kubernetes

Remote-Write

Client configuration

Remote-write client configuration is fully compatible with prometheus configuration format.

remote_write:
  clients:
    mimir:
      url: http://mimir/api/v1/push
      write_relabel_configs:
      - action: replace
        target_label: job
        replacement: loki-recording-rules

Per-Tenant Limits

Remote-write can be configured at a global level in the base configuration, and certain parameters tuned specifically on a per-tenant basis. Most of the configuration options defined here have override options (which can be also applied at runtime!).

Tuning

Remote-write can be tuned if the default configuration is insufficient (see Failure Modes below).

There is a guide on the Prometheus website, all of which applies to Loki, too.

Rules can be evenly distributed across available rulers by using -ruler.enable-sharding=true and -ruler.sharding-algo="by-rule".

-ruler.enable-sharding and -ruler.sharding-strategy control which ruler instances own which tenants (ring-based sharding, with optional shuffle sharding across rulers). For details, refer to Shuffle sharding.

-ruler.sharding-algo controls how a single tenant's rule groups are split up for evaluation once sharding is enabled. Rule groups execute in order; this is a feature inherited from Prometheus' rule engine (which Loki uses), but Loki has no need for this constraint because rules cannot depend on each other. The default algorithm, by-group, shards by rule group, but this may be undesirable, since some rule groups could contain more expensive rules, which can lead to subsequent rules missing evaluations. The by-rule algorithm creates one rule group for each rule the ruler instance "owns" (based on its hash ring), and these groups are all executed concurrently.

Observability

Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus' observability.

Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with loki_ruler_wal_.

For example: prometheus_remote_storage_bytes_total → loki_ruler_wal_prometheus_remote_storage_bytes_total

Additional metrics are exposed, also with the prefix loki_ruler_wal_. All per-tenant metrics contain a tenant label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large.

Some key metrics to note are:

We've created a basic dashboard in our loki-mixin which you can use to administer recording rules.

Failure Modes

Remote-Write Lagging

Remote-write can lag behind for many reasons:

  1. Remote-write storage (Prometheus) is temporarily unavailable
  2. A tenant is producing samples too quickly from a recording rule
  3. Remote-write is tuned too low, creating backpressure

It can be determined by subtracting loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds from loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds.

In case 1, the ruler will continue to retry sending these samples until the remote storage becomes available again. Be aware that if the remote storage is down for longer than ruler.wal.max-age, data loss may occur after truncation occurs.

In cases 2 and 3, you should consider tuning remote-write appropriately.

Further reading: see this blog post by Prometheus maintainer Callum Styan.

Appender Not Ready

Each tenant's WAL has an "appender" internally; this appender is used to append samples to the WAL. The appender is marked as not ready until the WAL storage has been initialized upon startup. You can alert on loki_ruler_wal_appender_ready < 1 to detect tenants whose WAL is not yet ready to accept samples.

Corrupt WAL

If a disk fails or the ruler does not terminate correctly, there's a chance one or more tenant WALs can become corrupted. The ruler wipes its WAL directory on startup, so any WAL left over from a previous run — corrupt or not — is discarded and recreated fresh rather than replayed.

Found another failure mode?

Open an issue and tell us about it!

Enforce rate limits and push request validation

Loki will reject requests if they exceed a usage threshold (rate limit error) or if they are invalid (validation error).

All occurrences of these errors can be observed using the loki_discarded_samples_total and loki_discarded_bytes_total metrics. The sections below describe the various possible reasons specified in the reason label of these metrics.

Terminology

Rate-Limit Errors

Rate-limits are enforced when Loki cannot handle more requests from a tenant.

Rate-limit errors reject the whole request before Loki stores any of it. The tables below list Sample discarded: No for these errors because Loki never accepts the samples in the first place, but the request still increments loki_discarded_samples_total and loki_discarded_bytes_total. Because these errors are retryable, a client can resend the same data after the rate limit condition clears.

rate_limited

This rate limit is enforced when a tenant has exceeded their configured log ingestion rate limit.

One solution if you're seeing samples dropped due to rate_limited is simply to increase the rate limits on your Loki cluster. These limits can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. The config options to use are ingestion_rate_mb and ingestion_burst_size_mb.

Note that you'll want to make sure your Loki cluster has sufficient resources provisioned to be able to accommodate these higher limits. Otherwise your cluster may experience performance degradation as it tries to handle this higher volume of log lines to ingest.

Another option to address samples being dropped due to rate_limits is simply to decrease the rate of log lines being sent to your Loki cluster. Consider collecting logs from fewer targets or setting up drop stages in Alloy to filter out certain log lines. You can also use Alloy's rate limiting to control the volume of logs sent to your Loki cluster.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable Yes
Sample discarded No
Configurable per tenant Yes
HTTP status code 429 Too Many Requests

per_stream_rate_limit

This limit is enforced when a single stream reaches its rate limit.

Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to (the size of that set is equal to the replication_factor value).

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. The config options to adjust are per_stream_rate_limit and per_stream_rate_limit_burst. The default values are 3MB for per_stream_rate_limit and 15MB for per_stream_rate_limit_burst.

Another option you could consider to decrease the rate of samples dropped due to per_stream_rate_limit is to split the stream that is getting rate limited into several smaller streams. A third option is to use the Alloy stage.limit block to limit the rate of samples sent to the stream hitting the per_stream_rate_limit.

We typically recommend setting per_stream_rate_limit no higher than 5MB, and per_stream_rate_limit_burst no higher than 20MB.

Property Value
Enforced by ingester
Outcome Request rejected
Retryable Yes
Sample discarded No
Configurable per tenant Yes
HTTP status code 429 Too Many Requests

stream_limit

This limit is enforced when a tenant reaches their maximum number of active streams.

Active streams are held in memory buffers in the ingesters, and if this value becomes sufficiently large then it will cause the ingesters to run out of memory.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the allowable active streams, adjust max_global_streams_per_user. Alternatively, the number of active streams can be reduced by removing extraneous labels or removing excessive unique label values.

{{< admonition type="note" >}} Under normal conditions, this limit only blocks the creation of genuinely new streams. Log lines for streams that are already active continue to be accepted.

However, during ingester scaling (autoscaling, rollouts, or ring changes such as a partition-ingester migration), streams can be rebalanced to ingesters that were not previously handling them. The new ingester has no record of that stream in memory, so it treats the next write for that stream as a stream creation and checks it against its local share of the limit. If that share is already used up, for example because the same tenant is also adding new high-cardinality streams during the rebalance, writes to the already-active stream can be rejected with stream_limit, even though the tenant has not added new label cardinality for it.

This effect is temporary. It is bounded by the number of ingesters added or restarted, and it typically resolves once the ring is stable and stream ownership is recalculated. {{< /admonition >}}

Property Value
Enforced by ingester
Outcome Request rejected
Retryable Yes
Sample discarded No
Configurable per tenant Yes
HTTP status code 429 Too Many Requests

Validation Errors

Validation errors occur when a request violates a validation rule defined by Loki.

line_too_long

This error occurs when a log line exceeds the maximum allowable length in bytes. The HTTP response will include the stream to which the offending log line belongs as well as its size in bytes.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the maximum line size, adjust max_line_size. We recommend that you do not increase this value above 256kb for performance reasons. Alternatively, Loki can be configured to ingest truncated versions of log lines over the length limit by using the max_line_size_truncate option. When truncation is enabled, you can use max_line_size_truncate_identifier to append a marker string to each truncated log line, so that truncated lines are easy to identify later.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

invalid_labels

This error occurs when one or more labels in the submitted streams fail validation.

Loki uses the same validation rules as Prometheus for validating labels.

Label names may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label names beginning with __ are reserved for internal use.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant No
HTTP status code 400 Bad Request

missing_labels

This validation error is returned when a stream is submitted without any labels.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant No
HTTP status code 400 Bad Request

too_far_behind and out_of_order

The too_far_behind validation error is returned when lines in question are older than half of -ingester.max-chunk-age compared to the newest line in the stream.

{{< admonition type="note" >}} unordered_writes and its related flag -ingester.unordered-writes are deprecated and will be removed in a future major release. Once removed, out-of-order writes will be accepted unconditionally, and this setting will no longer be configurable. {{< /admonition >}}

More details can be found here about the Loki ordering constraints.

The unordered_writes config value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. max_chunk_age, on the other hand, is a global-only configuration and cannot be set per tenant.

This problem can be solved by ensuring that log delivery is configured correctly, or by increasing the max_chunk_age value.

Property Value
Enforced by ingester
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes, for unordered_writes. No, for max_chunk_age, which is global only.
HTTP status code 400 Bad Request

greater_than_max_sample_age

If the reject_old_samples config option is set to true (it is by default), then samples will be rejected with reason=greater_than_max_sample_age if they are older than the reject_old_samples_max_age value. You should not see samples rejected for reason=greater_than_max_sample_age if reject_old_samples=false.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. This error can be solved by increasing the reject_old_samples_max_age value, or investigating why log delivery is delayed for this particular stream. The stream in question will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

too_far_in_future

If a sample's timestamp is greater than the current timestamp, Loki allows for a certain grace period during which samples will be accepted. If the grace period is exceeded, the error will occur.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. This error can be solved by increasing the creation_grace_period value, or investigating why this particular stream has a timestamp too far into the future. The stream in question will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

max_label_names_per_series

If a sample is submitted with more labels than Loki has been configured to allow, it will be rejected with the max_label_names_per_series reason. Note that 'series' is the same thing as a 'stream' in Loki - the 'series' term is a legacy name.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. This error can be solved by increasing the max_label_names_per_series value. The stream to which the offending sample (i.e. the one with too many label names) belongs will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

label_name_too_long

If a sample is sent with a label name that has a length in bytes greater than Loki has been configured to allow, it will be rejected with the label_name_too_long reason.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. This error can be solved by increasing the max_label_name_length value, though we do not recommend raising it significantly above the default value of 1024 for performance reasons. The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

label_value_too_long

If a sample has a label value with a length in bytes greater than Loki has been configured to allow, it will be rejected for the label_value_too_long reason.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. This error can be solved by increasing the max_label_value_length value. The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

duplicate_label_names

If a sample is sent with two or more identical labels, it will be rejected for the duplicate_label_names reason.

The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant No
HTTP status code 400 Bad Request

request_body_too_large

This error occurs when the raw push request exceeds a size limit before Loki can parse individual streams and log lines out of it. Unlike the other validation errors on this page, request_body_too_large can be triggered in two different ways:

To resolve this error, reduce the size of the push request (for example by sending fewer streams per request, or by reducing the number or size of labels on a stream), or increase max_recv_msg_size and max_decompressed_size if your cluster has the resources to handle larger requests.

{{< admonition type="note" >}} This reason only appears in the loki_discarded_bytes_total metric, not in loki_discarded_samples_total. Loki rejects the request before it can parse the body, so it cannot count how many log lines the request contained. {{< /admonition >}}

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant No
HTTP status code 413 Request Entity Too Large

disallowed_structured_metadata

This error occurs when a stream includes structured metadata but the tenant is not allowed to send it.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To allow structured metadata, set allow_structured_metadata to true (this is the default). The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

structured_metadata_too_large

This error occurs when the structured metadata attached to a single log line exceeds the maximum allowable size in bytes.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the limit, adjust max_structured_metadata_size (default: 64kb). The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

structured_metadata_too_many

This error occurs when a single log line includes more structured metadata entries than Loki has been configured to allow.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the limit, adjust max_structured_metadata_entries_count (default: 128). The offending stream will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

blocked_ingestion and blocked_ingestion_policy

{{< admonition type="note" >}} This is an experimental feature. {{< /admonition >}}

These reasons occur when an operator has deliberately blocked ingestion for a tenant (blocked_ingestion) or for a specific policy (blocked_ingestion_policy), rather than because a client sent invalid or excessive data. Policies are defined using the policy_stream_mapping setting in the limits_config block.

This is configured using block_ingestion_until (for a tenant) or block_ingestion_policy_until (for a policy), both of which take a date in RFC3339 format. Ingestion is blocked until that date. These values can be set globally in the limits_config block, but they are normally set for a single tenant in the runtime overrides file, because a global value blocks ingestion for every tenant.

The block_ingestion_status_code option controls what HTTP status code Loki returns while ingestion is blocked. The default is a custom status code, 260, so that blocked requests can be distinguished from ordinary errors. If block_ingestion_status_code is set to 200, Loki silently drops the blocked data without returning an error to the client, although the data is still counted in loki_discarded_samples_total and loki_discarded_bytes_total.

Property Value
Enforced by distributor
Outcome Request rejected, or silently dropped if block_ingestion_status_code is 200
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 260 by default, configurable with block_ingestion_status_code

missing_enforced_labels

{{< admonition type="note" >}} This is an experimental feature. {{< /admonition >}}

This error occurs when a tenant has been configured to require certain labels on every stream, and a stream is submitted that is missing one or more of them.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. Use enforced_labels to require labels on all streams, or policy_enforced_labels to require them only on streams matching a specific policy defined by policy_stream_mapping. The stream and the missing labels will be returned in the body of the HTTP response.

Property Value
Enforced by distributor
Outcome Request rejected
Retryable No
Sample discarded Yes
Configurable per tenant Yes
HTTP status code 400 Bad Request

Manage larger production deployments

When needing to scale Loki due to increased log volume, operators should consider running several Loki processes partitioned by role (ingester, distributor, querier, and so on) rather than a single Loki process. Grafana Labs' production setup contains .libsonnet files that demonstrates configuring separate components and scaling for resource usage.

Separate Query Scheduler

The Query frontend has an in-memory queue that can be moved out into a separate process similar to the Grafana Mimir query-scheduler. This allows running multiple query frontends.

To run with the Query Scheduler, configure both the frontend and the querier worker with the scheduler's address.

Using CLI flags:

Using the configuration file:

frontend:
  scheduler_address: <scheduler-host:port>

frontend_worker:
  scheduler_address: <scheduler-host:port>

{{< admonition type="note" >}} The querier's scheduler_address is configured under frontend_worker, not under querier. The frontend_worker block configures the worker running within the querier that pulls and executes queries. {{< /admonition >}}

It is not valid to start the querier with both a configured frontend address and a scheduler address.

The query scheduler process itself can be started via the -target=query-scheduler option of the Loki Docker image. For instance, docker run grafana/loki:latest -config.file=/etc/loki/config.yaml -target=query-scheduler -server.http-listen-port=8009 -server.grpc-listen-port=9009 starts the query scheduler listening on ports 8009 and 9009.

Scheduler discovery using a ring

Instead of configuring a static scheduler_address on the frontend and the querier worker, you can have query schedulers register themselves in a hash ring. Queriers and query frontends then discover query schedulers through the ring instead of using a fixed address.

To enable this, set use_scheduler_ring: true, or use the -query-scheduler.use-scheduler-ring CLI flag:

query_scheduler:
  use_scheduler_ring: true

{{< admonition type="note" >}} Set this option in the configuration used by your query schedulers, your queriers, and your query frontends. Queriers and query frontends only read the ring if use_scheduler_ring is true in their own configuration. If all of your components share one configuration file, you only need to set it once. {{< /admonition >}}

The ring needs a key-value store. In most deployments you do not need to configure one for the query scheduler specifically, because the scheduler ring inherits the store from the common.ring block, and Loki uses memberlist for all rings when you configure a memberlist section. To set the store for the scheduler ring alone, use the scheduler_ring block:

query_scheduler:
  use_scheduler_ring: true
  scheduler_ring:
    kvstore:
      store: memberlist

If you do not configure a frontend address, a scheduler address, or a downstream URL anywhere in your configuration, Loki automatically enables the scheduler ring for you.

Each component that takes part in the ring exposes the ring state at the /scheduler/ring endpoint, which you can use to check that all query schedulers registered as expected.

Memory ballast

In compute-constrained environments, garbage collection can become a significant performance factor. Frequently-run garbage collection interferes with running the application by using CPU resources. The use of memory ballast can mitigate the issue. Memory ballast allocates extra, but unused virtual memory in order to inflate the quantity of live heap space. Garbage collection is triggered by the growth of heap space usage. The inflated quantity of heap space reduces the perceived growth, so garbage collection occurs less frequently.

Configure memory ballast using the ballast_bytes configuration option.

Remote rule evaluation

This feature was first proposed in LID-0002; it contains the design decisions which informed the implementation.

By default, the ruler component embeds a query engine to evaluate rules. This generally works fine, except when rules are complex or have to process a large amount of data regularly. Poor performance of the ruler manifests as recording rules metrics with gaps or missed alerts. This situation can be detected by alerting on the loki_prometheus_rule_group_iterations_missed_total metric when it has a non-zero value.

A solution to this problem is to externalize rule evaluation from the ruler process. The ruler embedded query engine is single-threaded, meaning that rules are not split, sharded, or otherwise accelerated like regular Loki queries. The query-frontend component exists explicitly for this purpose and, when combined with a number of querier instances, can massively improve rule evaluation performance and lead to fewer missed iterations.

It is generally recommended to create a separate query-frontend deployment and querier pool from your existing one - which handles adhoc queries via Grafana, logcli, or the API. Rules should be given priority over adhoc queries because they are used to produce metrics or alerts which may be crucial to the reliable operation of your service; if you use the same query-frontend and querier pool for both, your rules will be executed with the same priority as adhoc queries which could lead to unpredictable performance.

To enable remote rule evaluation, set the following configuration options:

ruler:
  evaluation:
    mode: remote
    query_frontend:
      address: dns:///<query-frontend-service>:<grpc-port>

If the query-frontend connection requires TLS, set tls_enabled: true under query_frontend and configure the accompanying TLS options.

To reduce contention when many rules evaluate at the same time, set max_jitter to add a bounded, random delay before each rule evaluation. This option applies to both local and remote evaluation modes:

ruler:
  evaluation:
    max_jitter: 5s

Refer to the ruler configuration reference for further configuration options.

When you enable remote rule evaluation, the ruler component becomes a gRPC client to the query-frontend service; this will result in far lower ruler resource usage because the majority of the work has been externalized. The LogQL queries coming from the ruler will be executed against the given query-frontend service. Requests will be load-balanced across all query-frontend IPs if the dns:/// prefix is used.

{{< admonition type="note" >}} Queries that fail to execute are not retried. {{< /admonition >}}

Limits and Observability

Remote rule evaluation can be tuned with the following options:

Both of these can be specified globally in the limits_config section or on a per-tenant basis.

max_jitter, described earlier in this topic, is a global-only setting under ruler.evaluation rather than a per-tenant limit.

Remote rule evaluation exposes a number of metrics:

Each of these metrics are per-tenant, so cardinality must be taken into consideration.

Manage version upgrades

Speed up ingester rollout using zone awareness

The Loki zone aware ingesters are used by Grafana Labs in order to allow for easier rollouts of large Loki deployments. You can think of them as three logical zones, however with some extra Kubernetes configuration you could deploy them in separate zones.

By default, an incoming log stream's logs are replicated to 3 random ingesters. Except in the case of some replica scaling up or down, a given stream will always be replicated to the same 3 ingesters. This means that if one of those ingesters is restarted no data is lost. However two or more ingesters restarting can result in data loss and also impacts the systems ability to ingest logs because of an unhealthy ring status.

With zone awareness enabled, an incoming log line will be replicated to one ingester in each zone. This means that we're not only concerned about ingesters in multiple zones restarting at the same time, we can now rollout or lose an entire zone at once without impacting writes. This allows deployments with a large number of ingesters to be deployed much more quickly.

At Grafana Labs, we also make use of rollout-operator to manage rollouts to the 3 StatefulSets gracefully. The rollout-operator looks for labels on StatefulSets to know which ones are part of a certain rollout group, and coordinates rollouts of pods only from a single StatefulSet in the group at a time. See the README in the rollout-operator repo for a more in depth explanation.

{{< admonition type="note" >}} This guide describes the jsonnet deployment method used internally at Grafana Labs. New jsonnet deployments run zone aware ingesters by default, so this guide is only needed if you have an existing deployment that still uses a single ingester StatefulSet.

If you deploy Loki with the Helm chart instead, refer to Zone-aware replication in the Helm chart components documentation, which uses ingester.zoneAwareReplication Helm values rather than the jsonnet configuration described here.

The number of zones is fixed at 3 in the jsonnet library described in this guide. There is no configuration option to run a different number of zones. {{< /admonition >}}

Migration

Migrating from a single ingester StatefulSet to 3 zone aware ingester StatefulSets. The migration follows a few general steps, regardless of deployment method.

  1. Configure your existing ingesters to be part of a zone, for example zone-default, this will allow us to later exclude them from the write path while still allowing for graceful shutdowns.

  2. Prep for the increase in active streams (due to the way streams are split between ingesters) by increasing the number of active streams allowed for your tenants.

  3. Add and scale up your new zone-aware ingester StatefulSets such that each has 1/3rd of the total number of replicas you want to run.

  4. Enable zone awareness on the write path by setting distributor.zone-awareness-enabled to true for distributors and rulers.

  5. Wait some time to ensure that the new zone-aware ingesters have data for the time period they are queried for (query_ingesters_within).

  6. Enable zone awareness on the read path by setting distributor.zone-awareness-enabled to true for queriers.

    {{< admonition type="note" >}} distributor.zone-awareness-enabled and distributor.excluded-zones are CLI flag names. Despite the distributor. prefix, they configure the ingester ring, so each process that uses that ring (distributor, ruler, and querier) reads the same flag. In YAML, the equivalent keys are zone_awareness_enabled and excluded_zones under ingester.lifecycler.ring, not under a top-level distributor: block. {{< /admonition >}}

  7. Configure distributors and rulers to exclude ingesters in the zone-default so those ingesters no longer receive write traffic using distributor.excluded-zones.

  8. Use the shutdown endpoint to flush data from the default ingesters, then scale down and remove the associated StatefulSet.

  9. Clean up any config remaining from the migration.

Detailed Migration Steps

The following are steps to live migrate (no downtime) an existing Loki deployment from a single ingester StatefulSet to 3 zone aware ingester StatefulSets.

These instructions assume you are using the zone aware ingester jsonnet deployment code from this repo, see here. If you are not using jsonnet see the relevant annotations in some steps that describe how to perform that step manually.

  1. Configure the zone for the existing ingester StatefulSet as zone-default by setting multi_zone_default_ingester_zone: true, this allows us to later filter out that zone from the write path.

    If you're not using jsonnet, this is done by setting the -ingester.availability-zone=zone-default flag (or the equivalent availability_zone field under the ingester's lifecycler YAML configuration) on the existing ingesters.

  2. Configure ingester-pdb with maxUnavailable as 0 and deploy 3x zone-aware StatefulSets with 0 replicas by setting

    _config+:: {
    multi_zone_ingester_enabled: true,
    multi_zone_ingester_migration_enabled: true,
    multi_zone_ingester_replicas: 0,
    // These last two lines are necessary now that we enable zone aware ingester by default
    // so that newly created cells will not be migrated later on. If you miss them you will
    // break writes in the cell.
    multi_zone_ingester_replication_write_path_enabled: false,
    multi_zone_ingester_replication_read_path_enabled: false,
},

    Setting multi_zone_ingester_migration_enabled: true already configures the existing ingester PodDisruptionBudget (PDB) with maxUnavailable: 0 for you, so no manual PDB change is needed when using jsonnet.

    If you're not using jsonnet, manually set maxUnavailable to 0 on the existing ingester-pdb. The new ingester StatefulSets should have a label with rollout-group: ingester, annotation rollout-max-unavailable: x (put a placeholder value in, later you should set the value of this to be some portion of the StatefulSets total replicas, for example in jsonnet we template this so that each StatefulSet runs 1/3 of the total replicas and the max unavailable is 1/3 of each StatefulSets replicas), set the update strategy to OnDelete, and set -ingester.availability-zone=zone-<a|b|c> for each of the three new StatefulSets.

  3. Diff ingester and ingester-zone-a StatefulSets and make sure all config matches

    kubectl get statefulset -n loki-dev-008 ingester -o yaml > ingester.yaml
kubectl get statefulset -n loki-dev-008 ingester-zone-a -o yaml > ingester-zone-a.yaml
diff ingester.yaml ingester-zone-a.yaml

    Expected in diffs are values like: creation time and revision number, the zone, fields used by rollout operator, number of replicas, anything related to kustomize/flux, and PVC for the WAL since the containers don't exist yet.

  4. Temporarily double max series limits for users that are using more than 50% of their current limit, the queries are as follows (add label selectors as appropriate):

    sum by (tenant)(sum (loki_ingester_memory_streams) by (cluster, namespace, tenant) / on (namespace) group_left max by(namespace) (loki_distributor_replication_factor))
>
on (tenant) (
max by (tenant) (label_replace(loki_overrides{limit_name="max_global_streams_per_user"} / 2.5, "tenant", "$1", "user", "(.+)"))
)

    (sum (loki_ingester_memory_streams) by (cluster, namespace, tenant) / on (namespace) group_left max by(namespace) (loki_distributor_replication_factor)
) / ignoring(tenant) group_left max by (cluster, namespace)(loki_overrides_defaults{limit_name="max_global_streams_per_user"}) > 0.4)
unless on (tenant) (
(label_replace(loki_overrides{limit_name="max_global_streams_per_user"},"tenant", "$1", "user", "(.+)")))

  5. Scale up zone-aware StatefulSets until they have 1/3rd of replicas each. In smaller cells you can do this all at once, in larger cells it is safer to do it in chunks. The config value you need to change is multi_zone_ingester_replicas: 6, the value will be split across the three StatefulSets. In this case, each StatefulSet would run 2 replicas.

    If you're not using jsonnet, this is the step where you would also set the annotation rollout-max-unavailable to some value that is less than or equal to the number of replicas each StatefulSet is running.

    By default, each zone's StatefulSet uses pod anti-affinity so that ingesters from two different zones are never scheduled on the same node. Two ingesters from the same zone are still allowed to share a node. This anti-affinity rule is controlled by ingester_allow_multiple_replicas_on_same_node, which defaults to false.

    To also spread the ingesters within a single zone evenly across nodes, set multi_zone_ingester_use_topology_spread: true. This adds a topology spread constraint to each zone's StatefulSet, which you can tune with multi_zone_ingester_topology_spread_max_skew and multi_zone_ingester_topology_spread_when_unsatisfiable.

  6. Enable zone awareness on the write path by setting multi_zone_ingester_replication_write_path_enabled: true, this causes distributors and rulers to reshuffle series to distributors in each zone. Be sure to check that all the distributors and rulers have restarted properly.

    If you're not using jsonnet, enable zone awareness on the write path by setting distributor.zone-awareness-enabled to true for distributors and rulers.

  7. Wait for query_ingesters_within configured hours. The default is 3h. This ensures that no data will be missing if we query a new ingester. However, because we cut chunks at least every 30m due to chunk_idle_period we can likely reduce this amount of time.

  8. Check that rule evaluations are still correct on the migration, look for increases in the rate for metrics with names with the following suffixes:

    rule_evaluations_total
rule_evaluation_failures_total
rule_group_iterations_missed_total

  9. Enable zone-aware replication on the read path multi_zone_ingester_replication_read_path_enabled: true. If you're not using jsonnet, set distributor.zone-awareness-enabled to true for queriers.

  10. Check that queries are still executing correctly, for example look at loki_logql_querystats_latency_seconds_count to see that you don't have a big increase in latency or error count for a specific query type.

  11. Configure distributor / ruler to exclude ingesters in the zone-default so those ingesters no longer receive write traffic by setting multi_zone_ingester_exclude_default: true. If you're not using jsonnet set distributor.excluded-zones on distributors and rulers.

    It is a good idea to check rules evaluations again at this point, and also that the zone aware ingester StatefulSet is now receiving all the write traffic, you can compare sum(loki_ingester_memory_streams{cluster="<cluster>",job=~"(<namespace>)/ingester"}) to sum(loki_ingester_memory_streams{cluster="<cluster>",job=~"(<namespace>)/ingester-zone.*"})

  12. If you're using an automated reconciliation or deployment system like flux, disable it now (for example using flux ignore) if possible for just the default ingester StatefulSet.

  13. Shutdown flush the default ingesters, unregistering them from the ring, you can do this by port-forwarding each ingester Pod and using the endpoint: "http://url:PORT/ingester/shutdown?flush=true&delete_ring_tokens=true&terminate=false"

    See Flush in-memory chunks and shut down for details on this endpoint. For Kubernetes-native scale-down automation, you can instead use POST /ingester/prepare_shutdown, which arms the ingester to flush and unregister on its next SIGTERM rather than shutting it down immediately.

  14. manually scale down the default ingester StatefulSet to 0 replicas, we do this via tk apply but you could do it via modifying the yaml.

  15. merge a PR to your central config repo to keep the StatefulSet 0'd, and then remove the flux ignore.

  16. clean up any remaining temporary config from the migration, for example multi_zone_ingester_migration_enabled: true is no longer needed.

  17. ensure that all the old default ingester PVC/PV are removed.

Deploy Loki meta-monitoring with chart version 3

The primary method for collecting and monitoring a Loki cluster is to use the Kubernetes Monitoring Helm chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying version 3.8.x of the Kubernetes Monitoring Helm chart to monitor your Loki cluster.

{{< admonition type="note" >}} This page covers chart version 3.8.x, which is a maintained stable release line. If you do not already have a reason to stay on chart 3.8.x, use Deploy Loki meta-monitoring instead, which covers the current chart 4.x line. For help moving from chart 3.x to 4.x, refer to the Kubernetes Monitoring Helm chart migration guide. {{< /admonition >}}

{{< admonition type="note" >}} We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure. {{< /admonition >}}

Prerequisites

Preparing your environment

Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment.

  1. Add the Grafana Helm repository:

    helm repo add grafana https://grafana.github.io/helm-charts

  2. Update your Helm repositories:

    helm repo update

  3. Create a namespace for the monitoring stack:

    kubectl create namespace meta

Authentication

The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example.

  1. Create a new Cloud Access Policy in Grafana Cloud.

    1. Sign into Grafana Cloud.
    2. In the main menu, select Security > Access Policies.
    3. Click Create access policy.
    4. Give the policy a Name and select the following permissions:
      • Metrics: Write
      • Logs: Write
    5. Click Create.
    6. Click Add Token. Give the token a name and click Create. Save the token for later use.

  2. Collect url and username for Prometheus and Loki.

    1. Navigate to the Grafana Cloud Portal Overview page.
    2. Click the Details button for your Prometheus instance.
      1. From the Sending metrics using Grafana Alloy section, collect the instance username and url.
      2. Navigate back to the Overview page.
    3. Click the Details button for your Loki instance.
      1. From the Sending Logs to Grafana Cloud using Grafana Alloy section, collect the instance username and url.
      2. Navigate back to the Overview page.

  3. Create the Kubernetes Secrets with the collected credentials from Grafana Cloud.

    kubectl create secret generic metrics -n meta \
 --from-literal=username=<PROMETHEUS-USER> \
 --from-literal=password=<CLOUD-TOKEN>

kubectl create secret generic logs -n meta \
 --from-literal=username=<LOKI-USER> \
 --from-literal=password=<CLOUD-TOKEN>

Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including:

For further information on how to configure these methods, see the Kubernetes Monitoring Helm examples.

Deploy the Kubernetes Monitoring Helm chart

Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a values.yaml file to our local machine and modify it to include the necessary configuration.

  1. Download the values-chart-v3.yaml file from the Kubernetes Monitoring Helm chart repository:

    curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values-chart-v3.yaml

  2. Open the values-chart-v3.yaml file in a text editor of your choosing and add the Prometheus and Loki endpoints.

    destinations:
  - name: prometheus
    type: prometheus
    url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: metrics
      namespace: meta

  - name: loki
    type: loki
    url: https://<LOKI-ENDPOINT>/loki/api/v1/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: logs
      namespace: meta

  3. (Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud.

    # Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster.
cluster:
  name: loki-meta-monitoring-cluster

  4. The default values file assumes that you have deployed Loki in the loki namespace and will deploy the Kubernetes monitoring stack in the meta namespace. If you have deployed Loki in a different namespace, or the monitoring stack in a namespace other than meta, you will need to update several keys in values-chart-v3.yaml. There is no single namespaces key; namespace scoping is set independently for each feature:

    Key Format Purpose
    integrations.loki.instances[0].namespaces list Namespace(s) to discover Loki instances in.
    integrations.alloy.instances[0].namespaces list Namespace(s) to discover the monitoring stack's own Alloy collectors in.
    clusterEvents.namespaces list Namespace(s) to capture Kubernetes events from.
    clusterMetrics.cadvisor.metricsTuning.includeNamespaces list Namespace(s) to keep cadvisor metrics for. This is a metric filter, not a discovery scope.
    clusterMetrics.kube-state-metrics.namespaces list, or comma-separated string Namespace(s) to collect Kubernetes resource state from.
    podLogs.namespaces list Namespace(s) to collect pod logs from.

    For example, to add a namespace to the Loki discovery scope:

    integrations:
  loki:
    instances:
      - name: loki
        namespaces:
          - loki

    Each of these keys defaults to an empty list, which means "collect from all namespaces". Removing a key restores that default for that feature only; it does not affect the other keys in this table.

    Also note that destinations[].secret.namespace (used for both the metrics and logs destinations) must match the namespace where you created the metrics and logs secrets, which is meta by default. Update it if you installed the monitoring stack into a different namespace.

  5. Deploy the Kubernetes Monitoring Helm chart using the modified values-chart-v3.yaml file, pinning the chart to the 3.x line:

    helm install meta-loki grafana/k8s-monitoring \
 --namespace meta \
 --version "^3" \
 -f values-chart-v3.yaml

  6. Verify that the Kubernetes Monitoring Helm chart has been deployed successfully:

     kubectl get pods -n meta

    You should see a list of pods running in the meta namespace.

    NAME                                           READY   STATUS    RESTARTS ...        
meta-loki-alloy-singleton-6d7f8d8b86-sg4wx     2/2     Running   0        ...       
meta-loki-kube-state-metrics-64bdcfcbd-5snqz   1/1     Running   0        ...       
meta-loki-node-exporter-855l5                  1/1     Running   0        ...       
meta-loki-node-exporter-b976b                  1/1     Running   0        ...       
meta-loki-node-exporter-vsm4s                  1/1     Running   0        ...

Next Steps

You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see Install the Loki Mixin.

Deploy Loki meta-monitoring

The primary method for collecting and monitoring a Loki cluster is to use the Kubernetes Monitoring Helm chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying the Kubernetes Monitoring Helm chart to monitor your Loki cluster.

{{< admonition type="note" >}} We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure. If you run Loki in monolithic mode outside of this Helm chart, refer to Single binary meta-monitoring instead. {{< /admonition >}}

{{< admonition type="note" >}} This page covers chart version 4.x, the current line. If you need to stay on the maintained 3.8.x stable line instead, use Deploy Loki meta-monitoring with chart version 3. {{< /admonition >}}

Prerequisites

Preparing your environment

Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment.

  1. Add the Grafana Helm repository:

    helm repo add grafana https://grafana.github.io/helm-charts

  2. Update your Helm repositories:

    helm repo update

  3. Create a namespace for the monitoring stack:

    kubectl create namespace meta

Authentication

The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example.

  1. Create a new Cloud Access Policy in Grafana Cloud.

    1. Sign into Grafana Cloud.
    2. In the main menu, select Security > Access Policies.
    3. Click Create access policy.
    4. Give the policy a Name and select the following permissions:
      • Metrics: Write
      • Logs: Write
    5. Click Create.
    6. Click Add Token. Give the token a name and click Create. Save the token for later use.

  2. Collect url and username for Prometheus and Loki.

    1. Navigate to the Grafana Cloud Portal Overview page.
    2. Click the Details button for your Prometheus instance.
      1. From the Sending metrics using Grafana Alloy section, collect the instance username and url.
      2. Navigate back to the Overview page.
    3. Click the Details button for your Loki instance.
      1. From the Sending Logs to Grafana Cloud using Grafana Alloy section, collect the instance username and url.
      2. Navigate back to the Overview page.

  3. Create the Kubernetes Secrets with the collected credentials from Grafana Cloud.

    kubectl create secret generic metrics -n meta \
 --from-literal=username=<PROMETHEUS-USER> \
 --from-literal=password=<CLOUD-TOKEN>

kubectl create secret generic logs -n meta \
 --from-literal=username=<LOKI-USER> \
 --from-literal=password=<CLOUD-TOKEN>

Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including:

For further information on how to configure these methods, see the Kubernetes Monitoring Helm examples.

Deploy the Kubernetes Monitoring Helm chart

Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a values.yaml file to our local machine and modify it to include the necessary configuration.

  1. Download the values.yaml file from the Kubernetes Monitoring Helm chart repository:

    curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values.yaml

  2. Open the values.yaml file in a text editor of your choosing and add the Prometheus and Loki endpoints. Chart 4.x configures destinations as a map keyed by name, rather than a list:

    destinations:
  prometheus:
    type: prometheus
    url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: metrics
      namespace: meta

  loki:
    type: loki
    url: https://<LOKI-ENDPOINT>/loki/api/v1/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: logs
      namespace: meta

  3. (Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud.

    # Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster.
cluster:
  name: loki-meta-monitoring-cluster

  4. Confirm that at least one collector is defined under collectors:. Chart 4.x requires this; the install fails if collectors: is empty or missing. The default values.yaml already defines the alloy-singleton collector that every feature in this file is assigned to:

    collectors:
  alloy-singleton:
    presets: [singleton]

    If you add or rename a collector, update every collector: alloy-singleton reference in the file to match, or the install fails with an error naming the missing collector.

  5. The default values file assumes that you have deployed Loki in the loki namespace and will deploy the Kubernetes monitoring stack in the meta namespace. If you have deployed Loki in a different namespace, or the monitoring stack in a namespace other than meta, you will need to update several keys in values.yaml. There is no single namespaces key; namespace scoping is set independently for each feature:

    Key Format Purpose
    integrations.loki.instances[0].namespaces list Namespace(s) to discover Loki instances in.
    integrations.alloy.instances[0].namespaces list Namespace(s) to discover the monitoring stack's own Alloy collectors in.
    clusterEvents.namespaces list Namespace(s) to capture Kubernetes events from.
    clusterMetrics.cadvisor.metricsTuning.includeNamespaces list Namespace(s) to keep cadvisor metrics for. This is a metric filter, not a discovery scope.
    telemetryServices.kube-state-metrics.namespaces list, or comma-separated string Namespace(s) to collect Kubernetes resource state from. In chart 4.x this key must be set under telemetryServices, not under clusterMetrics.
    podLogsViaKubernetesApi.namespaces list Namespace(s) to collect pod logs from.

    For example, to add a namespace to the Loki discovery scope:

    integrations:
  loki:
    instances:
      - name: loki
        namespaces:
          - loki

    Each of these keys defaults to an empty list, which means "collect from all namespaces". Removing a key restores that default for that feature only; it does not affect the other keys in this table.

    Also note that destinations.prometheus.secret.namespace and destinations.loki.secret.namespace must match the namespace where you created the metrics and logs secrets, which is meta by default. Update them if you installed the monitoring stack into a different namespace.

  6. Deploy the Kubernetes Monitoring Helm chart using the modified values.yaml file, pinning the chart to the 4.x line:

    helm install meta-loki grafana/k8s-monitoring \
 --namespace meta \
 --version "^4" \
 -f values.yaml

  7. Verify that the Kubernetes Monitoring Helm chart has been deployed successfully:

     kubectl get pods -n meta

    You should see a list of pods running in the meta namespace, similar to the following. Exact pod names and counts depend on your cluster size and which features you have enabled:

    NAME                                           READY   STATUS    RESTARTS ...
meta-loki-alloy-operator-7d8f9c6b5d-x2k9p      1/1     Running   0        ...
meta-loki-alloy-singleton-6d7f8d8b86-sg4wx     2/2     Running   0        ...
meta-loki-kube-state-metrics-64bdcfcbd-5snqz   1/1     Running   0        ...
meta-loki-node-exporter-855l5                  1/1     Running   0        ...
meta-loki-node-exporter-b976b                  1/1     Running   0        ...
meta-loki-node-exporter-vsm4s                  1/1     Running   0        ...

Next Steps

You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see Install the Loki Mixin.

Key metrics for monitoring Loki

Loki exposes many metrics, and each component behaves differently under load. This page focuses on the highest-signal metrics for detecting negative trends early.

{{< admonition type="note" >}} The example queries on this page are PromQL. Run them against the Prometheus-compatible data source where your Loki metrics are stored (for example, Prometheus, Mimir, or Grafana Cloud Metrics). {{< /admonition >}}

For setup and prebuilt dashboards and alerts, refer to:

Request error rate

Watch request failures first. A sustained increase in 5xx responses is usually the earliest sign of user-visible impact.

Key metric:

{{< admonition type="note" >}} cluster, namespace, and job in the example queries on this page come from your Prometheus scrape configuration, not from Loki itself. They are not labels on the metric. {{< /admonition >}}

Example query:

100 * sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[2m])) by (cluster, namespace, job, route)
/
sum(rate(loki_request_duration_seconds_count[2m])) by (cluster, namespace, job, route)

Abnormal behavior:

Request latency (p99)

Latency degradation can appear before hard failures. Track p99 for read and write routes.

Key metric:

Example query:

histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, cluster, namespace, job, route))

Abnormal behavior:

Panics

Panics are high-severity faults and should stay at zero.

Key metric:

Example query:

sum(increase(loki_panic_total[10m])) by (cluster, namespace, job)

Abnormal behavior:

Discarded samples

Discarded samples indicate data that Loki rejected or dropped. This is one of the most important ingestion-quality signals.

Key metric:

Example query:

topk(10, sum by (tenant, reason) (rate(loki_discarded_samples_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval])))

Abnormal behavior:

Compaction health

Compaction issues can silently degrade read performance and retention behavior over time.

{{< admonition type="note" >}} Prior to Loki 4.0 the compaction and retention metrics used a loki_boltdb_shipper_ prefix for historical reasons. The compactor emitted these metrics regardless of which index type you use, including TSDB. After removing the legacy BoltDB Shipper index, the prefix has been renamed to loki_tsdb_shipper_. {{< /admonition >}}

Key metrics:

Example queries:

sum(loki_tsdb_shipper_compactor_running) by (cluster, namespace)

time() - (loki_tsdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds > 0)

Abnormal behavior:

Ingester health and flush behavior

Ingester pressure often appears as memory growth, poor chunk utilization, or flush backlog.

Key metrics:

Example queries:

sum(loki_ingester_memory_streams{cluster="$cluster", namespace="$namespace"})

The share of in-memory streams that were created by stream sharding in the distributors:

sum(loki_ingester_memory_stream_shards{cluster="$cluster", namespace="$namespace"})
/
sum(loki_ingester_memory_streams{cluster="$cluster", namespace="$namespace"})

sum(loki_ingester_flush_queue_length{cluster="$cluster", namespace="$namespace"})

Abnormal behavior:

Distributor throughput

Throughput changes help identify upstream sender issues, sudden traffic shifts, or ingestion bottlenecks.

Key metrics:

Example queries:

sum(rate(loki_distributor_bytes_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

sum(rate(loki_distributor_lines_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

Abnormal behavior:

Kafka-based ingestion (experimental)

{{< admonition type="note" >}} These metrics only apply if you've enabled Kafka-based ingestion, an experimental write-path feature. {{< /admonition >}}

Key metrics:

Example queries:

# Consumption lag p99 by partition state
histogram_quantile(0.99, sum(rate(loki_kafka_client_partition_reader_consumption_lag_seconds_bucket{phase="running", cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (le, cluster, namespace))

# Kafka produce error rate
sum(rate(loki_distributor_kafka_appends_total{status="fail", cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (cluster, namespace)

Abnormal behavior:

For a complete list of Kafka metrics, refer to Kafka-based ingestion.

Object store operations

Object store latency and failures directly impact query and retention workflows.

Key metrics:

Example queries:

sum by (operation) (rate(loki_objstore_bucket_operation_failures_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

histogram_quantile(0.99, sum(rate(loki_objstore_bucket_operation_duration_seconds_bucket{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (le, operation))

Abnormal behavior:

Connection metrics just for S3:

Abnormal behavior:

Resource and runtime health

Resource pressure can explain or predict service degradation before alert thresholds are crossed.

Common signals to track:

Abnormal behavior:

Loki Canary (end-to-end data verification)

If you run Loki Canary, use it as an end-to-end correctness signal, not only a performance signal.

Key metrics:

Example query:

sum(increase(loki_canary_missing_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range]))
/
sum(increase(loki_canary_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range]))
* 100

Abnormal behavior:

Internal error log rate

Internal logs provide fast context when metrics indicate degradation.

Key metric:

Use this metric with component logs to correlate where failures begin.

Retention and sweeper progress

Retention and sweeper lag can cause storage growth and delayed data lifecycle actions.

Key metrics:

Example query:

time() - (loki_tsdb_shipper_retention_sweeper_marker_file_processing_current_time{cluster="$cluster", namespace="$namespace"} > 0)

Abnormal behavior:

Next steps

Install dashboards, alerts, and recording rules

Loki is instrumented to expose metrics about itself via the /metrics endpoint, designed to be scraped by Prometheus. Each Loki release includes a mixin. The Loki mixin provides a set of Grafana dashboards, Prometheus recording rules and alerts for monitoring Loki.

To set up monitoring using the mixin, you need to:

  1. Deploy the Kubernetes Monitoring Helm chart. Follow the instructions in the Deploy Loki Meta-monitoring documentation.
  2. Be actively storing metrics from your Loki cluster in Grafana Cloud or a separate LGTM stack.

This procedure assumes that you have set up Loki using the Helm chart.

{{< admonition type="note" >}} Be sure to update the commands and configuration to match your own deployment. {{< /admonition >}}

Install Loki dashboards in Grafana

After Loki metrics are scraped by the Kubernetes Monitoring Helm chart and stored in a Prometheus compatible time-series database, you can monitor Loki’s operation using the Loki mixin.

Each Loki release includes a mixin that includes:

To install the mixins in Grafana and a Prometheus-compatible rules store, the general steps are as follows:

  1. Download the mixin dashboards from the Loki repository.

  2. Import the dashboards in your Grafana instance.

  3. Upload alerts.yaml and rules.yaml files to Prometheus or Mimir with mimirtool.

Download the loki-mixin dashboards

  1. First, clone the Loki repository from Github:

    git clone https://github.com/grafana/loki
cd loki

  2. Check out the tag that matches the version of Loki you are running, so that the dashboards, alerts, and recording rules match the metrics your Loki cluster exposes. For example, if you are running Loki 3.7.6:

    git checkout v3.7.6

  3. Once you have a local copy of the repository at the matching version, navigate to the production/loki-mixin-compiled directory.

    cd production/loki-mixin-compiled

This directory contains a compiled version of the alert and recording rules, as well as the dashboards.

{{< admonition type="note" >}} If you want to change any of the mixins, make your updates in the production/loki-mixin directory. Use the instructions in the README in that directory to regenerate the files. {{< /admonition >}}

Import the dashboards to Grafana

The dashboards directory includes the monitoring dashboards that can be installed into your Grafana instance. Refer to Import a dashboard in the Grafana documentation.

{{< admonition type="note" >}} The dashboards and alerts expect your metrics to carry these labels: cluster, namespace, job, container, pod, and instance. The Kubernetes Monitoring Helm chart provides them, because it adds the cluster name as a global label and collects container and node metrics using cAdvisor, kube-state-metrics, and Node Exporter.

If your setup uses different label names, or does not collect container and node metrics, some panels and alerts show no data until you relabel your metrics to match. {{< /admonition >}}

{{< admonition type="tip" >}} Install all dashboards. You can only import one dashboard at a time. Create a new folder in the Dashboards area, for example “Loki Monitoring”, as an easy location to save the imported dashboards. {{< /admonition >}}

To create a folder:

  1. Open your Grafana instance and select Dashboards.
  2. Click the New button.
  3. Select New folder from the New menu.
  4. Name your folder, for example, “Loki Monitoring”.
  5. Click Create.

To import a dashboard:

  1. Open your Grafana instance and select Dashboards.
  2. Click the New button.
  3. Select Import from the New menu.
  4. On the Import dashboard screen, select Upload dashboard JSON file.
  5. Browse to production/loki-mixin-compiled/dashboards and select the dashboard to import. Or, drag the dashboard file, for example, loki-operational.json, onto the Upload area of the Import dashboard screen.
  6. Select a folder in the Folder menu where you want to save the imported dashboard. For example, select "Loki Monitoring" created in the earlier steps.
  7. Click Import.

The imported files are listed in the Loki Monitoring dashboard folder.

To view the dashboards in Grafana:

  1. Select Dashboards in your Grafana instance.
  2. Select Loki Monitoring, or the folder where you uploaded the imported dashboards.
  3. Select any file in the folder to view the dashboard.

Add alerts and recording rules to Prometheus or Mimir

The rules and alerts need to be installed into a Prometheus instance, Mimir or a Grafana Enterprise Metrics cluster.

You can find the YAML files for alerts and rules in the following directories in the Loki repo:

You use mimirtool to load the mixin alerts and rules definitions into a Prometheus instance, Mimir, or a Grafana Enterprise Metrics cluster. The following examples show how to load the mixin alerts and rules into a Grafana Cloud instance.

  1. Download mimirtool.

  2. Export the authentication credentials for connecting to your Grafana Cloud Mimir instance.

    export MIMIR_ADDRESS=<CLOUD-MIMIR-URL>
export MIMIR_API_USER=<CLOUD-MIMIR-USER>
export MIMIR_API_KEY=<CLOUD-MIMIR-API-KEY>
export MIMIR_TENANT_ID=<CLOUD-MIMIR-USER> # Same as MIMIR_API_USER when using Grafana Cloud

    The best place to locate these credentials is to:

    1. Sign into Grafana Cloud and create a new access policy.
      1. In the main menu, select Security > Access Policies.
      2. Click Create access policy.
      3. Give the policy a Name and select the following permissions:
        • Alerts: Write & Read
        • Rules: Write & Read
      4. Click Create.
      5. Click Add Token. Give the token a name and click Create.
    2. Collect URL and user for Prometheus
      1. Navigate to the Grafana Cloud Portal Overview page.
      2. Click the Details button for your Alerts instance.
      3. From the Configuring your Alerting Stacks section, collect the instance User and URL for Metrics Authentication Settings.

  3. Using the same terminal we exported the Mimir environment variables into earlier, run the following command to load the recording rules:

    mimirtool rules load rules.yaml

  4. (Optional) Load the alert rules:

    mimirtool rules load alerts.yaml

Refer to the mimirtool documentation for more information.

Next steps

After you have installed the Loki mixin dashboards, alerts, and recording rules, you can now monitor your production Loki cluster using Grafana. Make sure you review the mixins when you upgrade Loki to make sure you are using the latest version of the mixin.

You can now move onto:

Single binary meta-monitoring

Meta monitoring for monolithic mode deployments involves some additional configuration. This approach does not use the Kubernetes Monitoring Helm chart.

Metrics

Configure Alloy or Prometheus to scrape the Loki metrics endpoint, adding the additional labels that are expected by the mixin dashboards, alerts, and recording rules:

Alloy

prometheus.scrape "loki" {
  targets = [{
    __address__ = "localhost:3100",
    cluster     = "prod",
    namespace   = "default",
    job         = "default/loki-single-binary",
    pod         = "loki-single-binary",
    container   = "loki",
  }]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"
  }
}

Prometheus

scrape_configs:
  - job_name: loki
    static_configs:
      - targets: ['localhost:3100']
        labels:
          cluster: prod
          namespace: default
          job: default/loki-single-binary
          pod: loki-single-binary
          container: loki

Mixins

To generate the mixins:

  1. Install jb.

  2. Install mixtool.

  3. Clone the Loki repository from Github.

    git clone https://github.com/grafana/loki
cd loki

  4. Check out the tag that matches the version of Loki you are running, so that the dashboards, alerts, and recording rules match the metrics your Loki cluster exposes. For example, if you are running Loki 3.7.6:

    git checkout v3.7.6

  5. Create the output directory.

    mkdir production/loki-mixin-compiled-single-binary

  6. Navigate to the mixin directory.

    cd production/loki-mixin

  7. Use jb to install mixin dependencies and generate the vendor directory.

    jb install

  8. Create the mixin configuration file single-binary.libsonnet.

    cat <<EOF > single-binary.libsonnet
local loki = import 'mixin.libsonnet';

loki + {
  _config+:: {
    meta_monitoring+: {
      enabled: true,
    },
    canary+: {
      // Whether or not to include the loki-canary dashboard.
      // Set to false if you don't run loki-canary.
      enabled: true,
    },
    promtail+: {
      // Whether or not to include promtail specific dashboards
      enabled: false,
    },
    // Tunes histogram recording rules to aggregate over this interval.
    // Set to at least twice the scrape interval; otherwise, recording rules will output no data.
    // Set to four times the scrape interval to account for edge cases: https://www.robustperception.io/what-range-should-i-use-with-rate/
    recording_rules_range_interval: '5m',
  },
}
EOF

  9. Generate dashboards, alerts, and recording rules.

    mixtool generate all \
  --output-alerts ../loki-mixin-compiled-single-binary/alerts.yaml \
  --output-rules ../loki-mixin-compiled-single-binary/rules.yaml \
  --directory ../loki-mixin-compiled-single-binary/dashboards \
  single-binary.libsonnet

  10. See the generated dashboards, alerts, and recording rules in the production/loki-mixin-compiled-single-binary directory.

  11. Follow the instructions in the Install Mixins documentation to set up the dashboards, alerts, and recordings.

Horizontal scaling of Compactor

{{< admonition type="caution" >}} Compactor horizontal scaling is an experimental feature. Use it with caution in your production environments. {{< /admonition >}}

Grafana Loki saw a major improvement in its operational complexity and cost-effectiveness with the introduction of object-storage-based indexes. This change also led to the addition of a singleton Compactor service, initially responsible only for index compaction. However, as new features like Custom Retention and Deletion of Logs with line filters were introduced, the Compactor's responsibilities grew. With increasing scale and more demanding features, especially log deletion with line filters, the singleton Compactor began to show its scaling limits.

Now, you can run the Loki Compactor in a horizontally scalable mode. Since log deletion with line filters is the compactor's most operationally intensive work, initially, this horizontally scalable architecture will be leveraged to speed up and distribute the workload specifically for its log deletion with line filters functionality.

How compactor horizontal scaling works

We have introduced two new modes to the Compactor for operating in horizontally scalable mode:

  1. Main:

    • Runs all Compactor functions and distributes chunk processing for log line deletion with filters to the workers.
    • Should be deployed as a singleton with access to a disk, like the current singleton deployment.

  2. Worker:

    • Connects to the Main Compactor over gRPC to get and execute the jobs.
    • Multiple replicas can be deployed to achieve higher job processing throughput.
    • Only needs access to Object Storage for reading/writing chunks.

Compactor horizontal scaling implementation details

Although, Horizontally Scalable Compactor currently only supports distributing the work of log line deletion with filters, in the future, we might add support for distributing more kind of work to the Workers. We are going to use the current functionality to see in detail some of the implementation details.

Main mode

For Distribution of Chunk processing work from Main compactor, there are following core components involved:

  1. Deletion Manifest Builder: The manifest builder works on a batch of delete request to discover all the chunks covered by them based on their labels filters and time range. Using the discovered chunks, it creates structured manifests and stores it to the Object Storage. Here is what comprises Deletion Manifest:

    • Segments: Groups of up to 100K chunks per segment. Also acts as partition for chunks by Loki Tenant/Table.
    • Manifest: Complete metadata about all segments and requests to process.

  2. Job Builder: The job builder converts manifests into discrete jobs and manages their lifecycle:

    • Job Creation: Breaks segments into jobs of up to 1K chunks each. Also includes line filters to apply on the chunks to remove the log lines requested for deletion.
    • Progress Tracking: Monitors job completion and stops processing of manifest when any of the jobs fail.
    • Job Queueing: Sends jobs to the Job Queue for processing.
    • Storage Updates: Collects storage updates suggested by Workers and stores them to Object Storage for each segment.
    • Post-processing Cleanup: Marks requests as processed and removes all the files from the storage.

  3. Job Queue: The job queue manages job distribution and Worker-Job Builder communication.

    • Job Distribution: Sends jobs to available workers via gRPC streaming.
    • Retry Logic: Automatically retries failed or timed-out jobs until allowed number of attempts.
    • Job Response: Sends the job processing response to Job Builder. Also notifies about failed jobs after running out of retries.

Worker mode

Workers connect to the Main Compactor via gRPC to fetch and execute jobs, returning results on the same gRPC stream. It uses compactor_grpc_address under the common config to connect to the Main Compactor.

When handling Deletion jobs, a worker downloads the listed chunks, applies the specified filters to rebuild them without the filtered lines, and then provides a comprehensive storage update as job execution response. The storage update details which chunks to delete from Object Storage and which newly created chunks to index.

Configure compactor mode

The horizontal_scaling_mode configuration option in the compactor controls the enablement of horizontally scalable compactor deployment. It supports setting the following modes:

Configure Main mode

To run Compactor in Main mode, the Horizontal Scaling Mode needs to be set to "main":

compactor:
  # CLI flag: -compactor.horizontal-scaling-mode
  horizontal_scaling_mode: "main"

Additionally, there are the below config options available for the Main compactor to configure some aspects of Job Building:

compactor:
  jobs_config:
    deletion:
      # Object storage path prefix for storing deletion manifests.
      # CLI flag: -compactor.jobs.deletion.deletion-manifest-store-prefix
      [deletion_manifest_store_prefix: <string> | default = "__deletion_manifest__/"]
      
      # Maximum time to wait for a job before considering it failed and retrying.
      # CLI flag: -compactor.jobs.deletion.timeout
      [timeout: <duration> | default = 15m]
      
      # Maximum number of times to retry a failed or timed out job.
      # CLI flag: -compactor.jobs.deletion.max-retries
      [max_retries: <int> | default = 3]

Configure Worker mode

To run Compactor in Worker mode, the Horizontal Scaling Mode needs to be set to "worker" and Main compactor's GRPC address needs to be set:

common:
  compactor_grpc_address: <HOST>:<GRPC_PORT>
compactor:
  # CLI flag: -compactor.horizontal-scaling-mode
  horizontal_scaling_mode: "worker"

Additionally, there are the below config options available for the Worker to configure some aspects of Job execution:

compactor:
  jobs_config:
    deletion:
      # Maximum number of chunks to process concurrently in each worker.
      # CLI flag: -compactor.jobs.deletion.chunk-processing-concurrency
      [chunk_processing_concurrency: <int> | default = 3]

  worker_config:
    # Number of sub-workers to run for concurrent processing of jobs. This
    # value must be greater than 0, or the compactor fails to start.
    # CLI flag: -compactor.worker.num-sub-workers
    [num_sub_workers: <int> | default = 4]

Filesystem object store

The filesystem object store is the easiest way to get started with Grafana Loki. This topic describes its pros and cons.

The filesystem object store keeps every object, such as chunks, in a directory that you specify. The recommended way to configure this directory is in the common.storage.filesystem block:

common:
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules

You can also configure the chunks directory directly under storage_config:

storage_config:
  filesystem:
    directory: /tmp/loki/

Loki creates a folder for every tenant. All the chunks for one tenant are stored in that tenant's folder.

If you run Loki in single-tenant mode, Loki puts all the chunks in a folder named fake. This is the synthesized tenant name that Loki uses in single-tenant mode.

See multi-tenancy for more information.

{{< admonition type="note" >}} Loki also supports an alternative, opt-in filesystem client based on thanos-io/objstore. Enable it by setting use_thanos_objstore: true and configuring storage_config.object_store.filesystem.dir. This client will become the default way to configure object store clients in a future release. For configuration examples, see Thanos storage examples and the storage client migration guide. {{< /admonition >}}

Pros

The filesystem object store is very simple. It requires no additional software to run Loki, and it works with TSDB, which is the recommended index store.

It's great for low volume applications, proof of concepts, and just playing around with Loki.

Cons

Grafana Labs does not support the filesystem object store for production environments, including for customers who have purchased a support contract.

Scaling

At some point there is a limit to how many chunks you can store in a single directory. For example, see issue #1502, which explains how a Loki user ran into a strange error with about 5.5 million chunk files in their file store. That issue also describes a workaround for the problem.

Loki writes one chunk per stream, so keeping the number of active streams low reduces the number of chunk files. You can also tune the following ingester settings to reduce how many chunks get flushed, although lower flush rates trade off for higher memory consumption:

It's still possible to store terabytes of log data with the filesystem store, but keep in mind the limitations on how many files a filesystem can efficiently store in a single directory.

Durability

The durability of your objects depends entirely on the underlying filesystem. Other object stores, such as S3 and GCS, do a lot of work behind the scenes to offer much higher durability for your data.

High availability

Running Loki as a cluster is not possible with the filesystem store, unless you share the filesystem in some way, for example over NFS. Using a shared filesystem is likely to give you a poor experience with Loki, just as it does with almost every other application.

Retention and deletion

When you use the filesystem chunk store, Loki does not delete chunks based on disk usage or free space. Loki only deletes data according to your retention configuration, so you must handle disk-full scenarios yourself, outside of Loki.

Retention-based deletion runs through the compactor. When you enable retention with compactor.retention_enabled, you must also set compactor.delete_request_store. Set it to filesystem to store delete requests in this object store.

For more information, see Retention and the storage retention section.

Log entry deletion

Grafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.

Log entry deletion is supported when the TSDB index is configured as the index store. It is also supported on the deprecated BoltDB Shipper index, but BoltDB Shipper is being removed in Loki 4.0, so new deployments should use TSDB.

The compactor component exposes REST endpoints that process delete requests. Hitting the endpoint specifies the streams and the time window. The deletion of the log entries takes place after a configurable cancellation time period expires.

Log entry deletion relies on configuration of the custom logs retention workflow as defined for the compactor. The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.

Configuration

Enable log entry deletion by setting retention_enabled to true in the compactor's configuration (-compactor.retention-enabled on the command line). The tenant's deletion_mode must also be filter-only or filter-and-delete, which is the case by default. delete_request_store also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests.

{{< admonition type="warning" >}} Be very careful when enabling retention. It is strongly recommended that you also enable versioning on your objects in object storage to allow you to recover from accidental misconfiguration of a retention setting. If you want to enable deletion but do not want to enforce retention, configure the retention_period setting with a value of 0s. {{< /admonition >}}

deletion_mode is a global and per-tenant setting in limits_config. Its default value is filter-and-delete, so log entry deletion is active for every tenant as soon as retention_enabled is true and delete_request_store is configured, unless you explicitly change the mode. Set deletion_mode in the main configuration file for a global default, or override it per tenant in the runtime configuration file.

deletion_mode supports three values:

A delete request may be canceled within a configurable cancellation period. Set the delete_request_cancel_period in the compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h. To cancel a delete request that has already started processing, pass the force=true query parameter to the cancellation endpoint.

Delete requests that include a line filter are split into smaller requests that each cover no more than delete_max_interval (24h by default). You can request a smaller split for a single request with the max_interval query parameter, but it cannot be larger than delete_max_interval. Delete requests without a line filter are not split. Refer to the HTTP API reference for the full set of request parameters.

Delete requests themselves are stored using the database type set by delete_request_store_db_type, which defaults to boltdb. You can instead use sqlite. When you migrate from one database type to another, you can set backup_delete_request_store_db_type to boltdb so that delete requests are also written to a backup database. Refer to the compactor configuration reference for details.

The deletion API endpoints are registered only when compactor.retention_enabled is true. When retention is not enabled, the endpoints are unavailable for every tenant, whatever the deletion_mode value is. When retention is enabled, use the deletion_mode tenant override to control which tenants can use the deletion API.

{{< admonition type="note" >}} Deletion of log lines with line filters is one of the compactor's most resource-intensive operations. If you delete large volumes of data with line filters, refer to Horizontal scaling of Compactor to distribute that work across multiple compactor instances. {{< /admonition >}}

Log retention

Retention in Grafana Loki is achieved through the Compactor. By default the compactor.retention-enabled flag is not set, so the logs sent to Loki live forever.

{{< admonition type="note" >}} If you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period. {{< /admonition >}}

Granular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.

Compactor

The Compactor is responsible for compaction of index files and applying log retention.

{{< admonition type="note" >}} Run the Compactor as a singleton (a single instance). {{< /admonition >}}

The Compactor runs compaction and retention on two independent schedules:

If the Compactor falls behind on either schedule, it runs the corresponding operation as soon as possible. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off.

{{< admonition type="note" >}} Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested. {{< /admonition >}}

The Compactor's algorithm to apply retention is as follows:

Chunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting -compactor.retention-delete-delay. Marker files are used to keep track of the chunks pending for deletion.

Chunks cannot be deleted immediately for the following reasons:

By default, marker files are written to local disk, under <working_directory>/retention/<object_store_type>_<period_from_date>/markers/. Marker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts. {{< admonition type="note" >}} Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files. {{< /admonition >}}

Alternatively, you can configure the Compactor to store marker files in object storage instead of local disk, by setting compactor.deletion-marker-object-store-prefix (deletion_marker_object_store_prefix) to a prefix that ends with /. When set, the Compactor writes marker files to the same per-period object store bucket used for chunks, under that prefix. On startup it also copies any existing marker files from local disk to that prefix, then removes the local copies. This lets you run the Compactor without a persistent volume for marker files. Leave this setting empty to keep using local disk.

Retention Configuration

This Compactor configuration example activates retention.

compactor:
  working_directory: /data/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: gcs
schema_config:
  configs:
    - from: "2020-07-31"
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v13
      store: tsdb
storage_config:
  tsdb_shipper:
    active_index_directory: /data/index
    cache_location: /data/index_cache
  gcs:
    bucket_name: loki

{{< admonition type="note" >}} Retention is only available if the index period is 24h. Single store TSDB requires a 24h index period. {{< /admonition >}}

retention_enabled should be set to true. Without this, the Compactor will only compact tables.

delete_request_store should be set to configure the store for delete requests. This is required when retention is enabled.

working_directory is the directory where marked chunks and temporary tables will be saved.

compaction_interval dictates how often compaction is applied. If the Compactor falls behind, compaction occurs as soon as possible.

apply_retention_interval dictates how often retention is applied, independently of compaction_interval. Refer to Compactor for how the default value is derived from compaction_interval.

retention_delete_delay is the delay after which the Compactor will delete marked chunks.

retention_delete_worker_count specifies the maximum quantity of goroutine workers instantiated to delete chunks.

A few other options are also relevant when tuning retention:

Refer to the Configuration reference for the full list of compactor options and their defaults.

Configuring the retention period

Retention period is configured within the limits_config configuration section.

There are two ways of setting retention policies:

{{< admonition type="note" >}} Each retention_stream rule's period must be at least 24h. Loki rejects configuration with a shorter value, both in the global retention_stream list and in any per-tenant override. This minimum does not apply to the retention_period setting, either globally or per tenant. {{< /admonition >}}

This example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides):

...
limits_config:
  retention_period: 744h
  retention_stream:
  - selector: '{namespace="dev"}'
    priority: 1
    period: 24h
runtime_config:
  file: /etc/overrides.yaml
...

{{< admonition type="note" >}} You can only use label matchers in the selector field of a retention_stream definition. Arbitrary LogQL expressions are not supported. {{< /admonition >}}

Per tenant retention can be defined by configuring runtime overrides. For example:

overrides:
    "29":
        retention_period: 168h
        retention_stream:
        - selector: '{namespace="prod"}'
          priority: 2
          period: 336h
        - selector: '{container="loki"}'
          priority: 1
          period: 72h
    "30":
        retention_stream:
        - selector: '{container="nginx", level="debug"}'
          priority: 1
          period: 24h

Retention period for a given stream is decided based on the first match in this list:

  1. If multiple per-tenant retention_stream selectors match the stream, retention period with the highest priority is picked.
  2. If multiple global retention_stream selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant retention_stream is set.
  3. If a per-tenant retention_period is specified, it will be applied.
  4. The global retention_period will be applied if none of the above match.
  5. If no global retention_period is specified, the default value of 0s is used, which means logs are kept indefinitely.

{{< admonition type="note" >}} The larger the priority value, the higher the priority. If two matching retention_stream rules have the same priority, Loki applies the shorter (lowest) of the two periods. {{< /admonition >}}

Stream matching uses the same syntax as Prometheus label matching:

The example configurations defined above will result in the following retention periods:

{{< admonition type="note" >}} If you are a Grafana Cloud customer, you can use the config self-serve API to configure your tenant retention. {{< /admonition >}}

Example Configuration

Example configuration with GCS with a 28 day retention:

schema_config:
  configs:
    - from: 2018-04-15
      store: tsdb
      object_store: gcs
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  gcs:
    bucket_name: GCS_BUCKET_NAME

limits_config:
  max_query_lookback: 672h # 28 days
  retention_period: 672h   # 28 days

compactor:
  working_directory: /data/retention
  delete_request_store: gcs
  retention_enabled: true

Object store lifecycle policies

Loki stores chunks and indexes as objects in the configured object store (S3, GCS, Azure Blob Storage, and so on). Chunks are deleted by the Compactor according to your configured retention, but the Compactor only removes chunk objects: it never deletes the objects that hold cluster state, such as the index or the cluster seed.

If you also configure an object store lifecycle policy on the bucket, scope it carefully. A blanket "delete everything older than N days" rule will eventually remove objects that Loki must keep, which corrupts the store and causes query failures. The note at the top of this page applies here as well: any lifecycle expiration must be longer than your retention period.

{{< admonition type="warning" >}} Never apply a lifecycle rule with an empty prefix (one that targets the whole bucket). Always scope lifecycle rules to the per-tenant chunk prefixes described below. {{< /admonition >}}

Bucket object layout

The following objects are written to the configured bucket. The exact prefixes depend on your schema_config and on whether multi-tenancy is enabled, so confirm them against your own configuration before writing a lifecycle rule.

Object or prefix Content Safe to expire with a lifecycle policy?
<tenant_id>/ (for example, fake/ in single-tenant mode, or your org ID per tenant) Chunk objects (the log data). Yes. These are the only objects a lifecycle rule should target.
Index objects (under the index path prefix, index/ by default, set by index.path_prefix in schema_config) TSDB or BoltDB index files, written by ingesters and rewritten by the Compactor. The index.prefix value (for example index_ or loki_index_) is the table-name prefix used inside this path, not the path itself, so match a lifecycle rule against the path prefix. No. Loki manages the index lifecycle internally. Deleting index objects removes references to live chunks and breaks queries.
loki_cluster_seed.json A small file at the bucket root holding the cluster seed used by usage reporting. It is written once and never rotated. No. A blanket age-based rule will delete it once it ages past the time to live (TTL).
Delete-request store objects (a delete_requests/ table under delete_request_store_key_prefix, index/ by default, in the bucket configured by delete_request_store) The Compactor's database of delete requests submitted through the Log entry deletion API, and their processing status. No. Removing these makes the Compactor lose track of pending delete requests.
Deletion marker objects (only present if you set deletion_marker_object_store_prefix; otherwise markers are kept on the Compactor's local disk and never appear in the bucket) The Compactor's record of chunks that are marked for deletion but not yet swept. Written under the configured prefix in the same per-period bucket used for chunks. No. Removing these makes the Compactor lose track of chunks pending deletion.
Ruler store objects (if the Ruler is backed by the same bucket) Recording and alerting rule files. No. These are configuration, not time-series data.

Compactor retention versus object store lifecycle policies

Compactor retention and object store lifecycle policies are two independent mechanisms, and they should not be configured in ways that conflict:

For most deployments, enabling Compactor retention is sufficient and an object store lifecycle policy is not required. If you do add a lifecycle policy as a cost safety net, set its expiration longer than your retention period plus compactor.retention-delete-delay, so the Compactor always deletes chunks first and the lifecycle rule only ever catches objects the Compactor has already orphaned.

Example: Amazon S3 lifecycle policy

This policy expires only chunk objects under a single tenant prefix. Replace fake/ with your tenant prefix (use one rule per tenant, or your org ID prefix, when multi-tenancy is enabled). The Days value must be larger than your retention period plus the delete delay.

{
  "Rules": [
    {
      "ID": "loki-chunk-expiration",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "fake/"
      },
      "Expiration": {
        "Days": 395
      }
    }
  ]
}

Apply it with the AWS CLI:

aws s3api put-bucket-lifecycle-configuration \
  --bucket YOUR_LOKI_BUCKET \
  --lifecycle-configuration file://lifecycle.json

Example: Google Cloud Storage lifecycle rule

The equivalent GCS rule expires objects by age and prefix. As with S3, scope matchesPrefix to the tenant chunk prefix and keep age larger than your retention period plus the delete delay.

{
  "rule": [
    {
      "action": {"type": "Delete"},
      "condition": {
        "age": 395,
        "matchesPrefix": ["fake/"]
      }
    }
  ]
}

Apply it with the Google Cloud CLI:

gcloud storage buckets update gs://YOUR_LOKI_BUCKET --lifecycle-file=lifecycle.json

For Azure Blob Storage, use a management policy with a prefixMatch set to the tenant chunk prefix and the same "expiration must exceed retention plus delete delay" rule.

Single Store TSDB (tsdb)

Starting with Loki v2.8, TSDB is the recommended Loki index. It is heavily inspired by the Prometheus TSDB sub-project. This new index is more efficient, faster, and more scalable. It also resides in object storage like the boltdb-shipper index which preceded it.

For the on-disk binary layout of the index files, see TSDB index format.

Example Configuration

To get started using TSDB, add the following configurations to your config.yaml:

schema_config:
  configs:
    # Old boltdb-shipper schema. Included in example for reference but does not need changing.
    - from: "2023-01-03" # <---- A date in the past
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v12
      store: boltdb-shipper
    # New TSDB schema below
    - from: "2023-01-05" # <---- A date in the future
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v13
      store: tsdb

storage_config:
  # Old boltdb-shipper configuration. Included in example for reference but does not need changing.
  boltdb_shipper:
    active_index_directory: /data/index
    build_per_tenant_index: true
    cache_location: /data/boltdb-cache
    index_gateway_client:
      # only applicable if using microservices where index-gateways are independently deployed.
      # This example is using kubernetes-style naming.
      server_address: dns:///index-gateway.<namespace>.svc.cluster.local:9095
  # New tsdb-shipper configuration
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache
    index_gateway_client:
      # only applicable if using microservices where index-gateways are independently deployed.
      # This example is using kubernetes-style naming.
      server_address: dns:///index-gateway.<namespace>.svc.cluster.local:9095

query_scheduler:
  # the TSDB index dispatches many more, but each individually smaller, requests. 
  # We increase the pending request queue sizes to compensate.
  max_outstanding_requests_per_tenant: 32768

querier:
  # Each `querier` component process runs a number of parallel workers to process queries simultaneously.
  # You may want to adjust this up or down depending on your resource usage
  # (more available cpu and memory can tolerate higher values and vice versa),
  # but we find the most success running at around `16` with tsdb
  max_concurrent: 16

Operations

Limits

We've added a user per-tenant limit called tsdb_max_query_parallelism in the limits_config. This functions the same as the prior max_query_parallelism configuration but applies to tsdb queries instead. Since the TSDB index will create many more smaller queries compared to the other index types before it, we've added a separate configuration so they can coexist. This is helpful when transitioning between index types. The default parallelism is 128 which should work well for most cases, but you can extend it globally in the limits_config or per-tenant in the overrides file as needed.

Dynamic Query Sharding

Previously we would statically shard queries based on the index row shards configured here. TSDB does Dynamic Query Sharding based on how much data a query is going to be processing. We additionally store size(KB) and number of lines for each chunk in the TSDB index which is then used by the Query Frontend for planning the query. Based on our experience from operating many Loki clusters, we have configured TSDB to aim for processing 300-600 MBs of data per query shard. This means with TSDB we will be running more, smaller queries.

The per-tenant tsdb_max_bytes_per_shard setting in limits_config controls this target size. It defaults to 600MB. Each time a query would need to process more data than this target, Loki doubles the number of shards for that query. As a result, each shard typically ends up processing between half and all of the target amount. This is why, with the default target of 600MB, query shards typically process between 300 MB and 600 MB of data each.

Loki also supports the following related per-tenant settings in limits_config:

Index Caching not required

TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of retention or your configured max_query_lookback under limits_config. After that, we suggest running without an index cache (it isn't used in TSDB).

Write Ahead Log

Ingesters temporarily store data in memory. In the event of a crash, there could be data loss. The Write Ahead Log (WAL) helps fill this gap in reliability.

The WAL in Grafana Loki records incoming data and stores it on the local file system in order to guarantee persistence of acknowledged data in the event of a process crash. Upon restart, Loki will "replay" all of the data in the log before registering itself as ready for subsequent writes. This allows Loki to maintain the performance and cost benefits of buffering data in memory and durability benefits (it won't lose data once a write has been acknowledged).

This section will use Kubernetes as a reference deployment paradigm in the examples.

Disclaimer and WAL nuances

The Write Ahead Log in Loki takes a few particular tradeoffs compared to other WALs you may be familiar with. The WAL aims to add additional durability guarantees, but not at the expense of availability. Particularly, there are two scenarios where the WAL sacrifices these guarantees.

  1. Corruption/Deletion of the WAL prior to replaying it

    In the event the WAL is corrupted/partially deleted, Loki will not be able to recover all of its data. In this case, Loki will attempt to recover any data it can, but will not prevent Loki from starting.

    You can use the Prometheus metric loki_ingester_wal_corruptions_total to track and alert when this happens.

  2. No space left on disk

    By default, Loki monitors disk usage on the volume that stores the WAL. If usage reaches 90% of capacity, the ingester rejects new writes instead of accepting data it cannot persist. This check runs every 10 seconds and measures the entire filesystem that holds the WAL directory, not just the size of the WAL files. Because of this, you should not share the WAL volume with other data. Writes resume automatically once disk usage drops back below the threshold. You can change the threshold with the --ingester.wal-disk-full-threshold flag, or set it to 0 to disable this protection. This throttling behavior isn't available on Windows.

    If you disable this threshold, or if a write reaches the disk in the short window before the check runs, the underlying disk can still become completely full. In that case, Loki will not fail the incoming write, but neither will it log the entry to the WAL. In this case, the persistence guarantees across process restarts will not hold for that entry.

    You can use the Prometheus metric loki_ingester_wal_disk_full_failures_total to track and alert when this happens. This counter increases both when disk usage crosses the throttling threshold and when a WAL write fails because the disk is completely full, so a nonzero value means the WAL disk needs attention, but it does not count each individual rejected write. You can also monitor the gauge loki_ingester_wal_disk_usage_percent to see current disk usage and how close it is to the threshold.

Backpressure

The WAL also includes a backpressure mechanism to allow a large WAL to be replayed within a smaller memory bound. This is helpful after bad scenarios (i.e. an outage) when a WAL has grown past the point it may be recovered in memory. In this case, the ingester will track the amount of data being replayed and once it's passed 90% of the ingester.wal-replay-memory-ceiling threshold, will flush to storage. When this happens, it's likely that the Loki attempt to deduplicate chunks via content addressable storage will suffer. We deemed this efficiency loss an acceptable tradeoff considering how it simplifies operation and that it should not occur during regular operation (rollouts, rescheduling) where the WAL can be replayed without triggering this threshold. Setting ingester.wal-replay-memory-ceiling to 0 disables backpressure entirely.

Metrics

The following metrics are available for monitoring the WAL:

Changes to deployment

  1. Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on StatefulSet with fixed volumes.

  2. The following flags are relevant when deploying with a persistent volume:

    • --ingester.wal-enabled enables writing to WAL during ingestion. It defaults to true, so this typically doesn't need to be set explicitly.
    • --ingester.wal-dir sets the directory where the WAL data should be stored and/or recovered from. Note that this should be on the mounted volume.
    • --ingester.checkpoint-duration to the interval at which checkpoints should be created.
    • --ingester.wal-replay-memory-ceiling (default 4GB) may be set higher/lower depending on your resource settings. It handles memory pressure during WAL replays, allowing a WAL many times larger than available memory to be replayed. This is provided to minimize reconciliation time after very bad situations, i.e. an outage, and will likely not impact regular operations/rollouts at all. We suggest setting this to a high percentage (~75%) of available memory.
    • --ingester.wal-disk-full-threshold (default 0.90) sets the fraction of disk usage (0.0 to 1.0) at which the ingester throttles incoming writes to protect WAL durability. Set to 0 to disable this protection. For details, refer to Disclaimer and WAL nuances.

Changes in lifecycle when WAL is enabled

Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL. If you need to ensure that data is always flushed to the chunk store when your pod shuts down, you can set the --ingester.flush-on-shutdown flag to true.

Disk space requirements

Based on tests in real world:

You should not target 100% disk utilization.

Migrating from stateless deployments

The ingester Deployment without WAL and StatefulSet with WAL should be scaled down and up respectively in sync without transfer of data between them to ensure that any ingestion after migration is reliable immediately.

Let's take an example of 4 ingesters. The migration would look something like this:

  1. Bring up one stateful ingester ingester-0 and wait until it's ready (accepting read and write requests).
  2. Scale down the old ingester deployment to 3 and wait until the leaving ingester flushes all the data to chunk store.
  3. Once that ingester has disappeared from kc get pods ..., add another stateful ingester and wait until it's ready. Now you have ingester-0 and ingester-1.
  4. Repeat step 2 to reduce remove another ingester from old deployment.
  5. Repeat step 3 to add another stateful ingester. Now you have ingester-0 ingester-1 ingester-2.
  6. Repeat step 4 and 5, and now you will finally have ingester-0 ingester-1 ingester-2 ingester-3.

How to scale up/down

Scale up

Scaling up is same as what you would do without WAL or StatefulSets. Nothing to change here.

Scale down

When scaling down, we must ensure existing data on the leaving ingesters are flushed to storage instead of just the WAL. This is because we won't be replaying the WAL on an ingester that will no longer exist and we need to make sure the data is not orphaned.

Consider you have 4 ingesters ingester-0 ingester-1 ingester-2 ingester-3 and you want to scale down to 2 ingesters, the ingesters which will be shut down according to StatefulSet rules are ingester-3 and then ingester-2.

Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the /ingester/shutdown?flush=true endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted.

After hitting the endpoint for ingester-2 ingester-3, scale down the ingesters to 2.

Also you can set the --ingester.flush-on-shutdown flag to true. This enables chunks to be flushed to long-term storage when the ingester is shut down.

Additional notes

Kubernetes hacking

StatefulSets are significantly more cumbersome to work with, upgrade, and so on. Much of this stems from immutable fields on the specification. For example, if one wants to start using the WAL with single store Loki and wants separate volume mounts for the WAL and the TSDB, you may see immutability errors when attempting updates the Kubernetes StatefulSets.

In this case, try kubectl -n <namespace> delete sts ingester --cascade=false. This will leave the Pods alive but delete the StatefulSet. Then you may recreate the (updated) StatefulSet and one-by-one start deleting the ingester-0 through ingester-n Pods in that order, allowing the StatefulSet to spin up new pods to replace them.

Scaling Down Using /flush_shutdown Endpoint and Lifecycle Hook

  1. StatefulSets for Ordered Scaling Down: The Loki ingesters should be scaled down one by one, which is efficiently handled by Kubernetes StatefulSets. This ensures an ordered and reliable scaling process, as described in the Deployment and Scaling Guarantees documentation.

  2. Using PreStop Lifecycle Hook: During the Pod scaling down process, the PreStop lifecycle hook triggers the /flush_shutdown endpoint on the ingester. This action flushes the chunks and removes the ingester from the ring, allowing it to register as unready and become eligible for deletion.

  3. Using terminationGracePeriodSeconds: Provides time for the ingester to flush its data before being deleted, if flushing data takes more than 30 minutes, you may need to increase it.

  4. Cleaning Persistent Volumes: Persistent volumes are automatically cleaned up by leveraging the enableStatefulSetAutoDeletePVC feature in Kubernetes.

By following the above steps, you can ensure a smooth scaling down process for the Loki ingesters while maintaining data integrity and minimizing potential disruptions.

Non-Kubernetes or baremetal deployments

Troubleshoot Logs Drilldown

{{< docs/shared lookup="troubleshoot-logs-drilldown.md" source="explore-logs" version="next" >}}

Troubleshoot log ingestion (WRITE)

This guide helps you troubleshoot errors that occur when ingesting logs into Loki and writing logs to storage. When Loki rejects log ingestion requests, it's typically due to exceeding rate limits, violating validation rules, or encountering storage issues.

Before you begin, ensure you have the following:

Monitoring ingestion errors

All ingestion errors are tracked using Prometheus metrics:

The reason label on these metrics indicates the specific error type. Set up alerts on these metrics to detect ingestion problems.

Rate limit errors

Rate limits protect Loki from being overwhelmed by excessive log volume. When rate limits are exceeded, Loki returns HTTP status code 429 Too Many Requests.

Error: rate_limited

Error message:

ingestion rate limit exceeded for user <tenant> (limit: <limit> bytes/sec) while attempting to ingest <lines> lines totaling <bytes> bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased

Cause:

The tenant has exceeded their configured ingestion rate limit. This is a global per-tenant limit enforced by the distributor.

Default configuration:

Resolution:

Properties:

Error: per_stream_rate_limit

Error message:

Per stream rate limit exceeded (limit: <limit>/sec) while attempting to ingest for stream '<stream_labels>' totaling <bytes>, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased

Cause:

A single stream has exceeded its own rate limit. Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to. This limit is enforced by the ingester, not the distributor.

Default configuration:

Resolution:

Properties:

Error: stream_limit

Error message:

maximum active stream limit exceeded when trying to create stream <stream_labels>, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: <tenant>

Cause:

The tenant has reached the maximum number of active streams. Active streams are held in memory on ingesters, and excessive streams can cause out-of-memory errors.

{{< admonition type="note" >}} Under normal conditions, this limit only blocks the creation of genuinely new streams. Log lines for streams that are already active continue to be accepted.

However, during ingester scaling (autoscaling, rollouts, or ring changes such as a partition-ingester migration), streams can be rebalanced to ingesters that were not previously handling them. The new ingester has no record of that stream in memory, so it treats the next write for that stream as a stream creation and checks it against its local share of the limit. If that share is already used up, for example because the same tenant is also adding new high-cardinality streams during the rebalance, writes to the already-active stream can be rejected with stream_limit, even though the tenant has not added new label cardinality for it.

This effect is temporary. It is bounded by the number of ingesters added or restarted, and it typically resolves once the ring is stable and stream ownership is recalculated. If you see stream_limit errors that line up with ingester scaling or rollouts rather than with new label values, this is the likely cause. {{< /admonition >}}

Default configuration:

Terminology:

Resolution:

{{< admonition type="note" >}} Do not increase stream limits to accommodate high cardinality labels, this can result in Loki flushing extremely high numbers of small files which will make for extremely poor query performance. As volume and the size of the infrastructure being monitored increase, it would be expected to increase the stream limit. However even for hundreds of TBs per day of logs you should avoid exceeding 300,000 max global streams per user. {{< /admonition >}}

Properties:

Validation errors

Validation errors occur when log data doesn't meet Loki's requirements. These return HTTP status code 400 Bad Request and are not retryable.

Error: line_too_long

Error message:

max entry size <max_size> bytes exceeded for stream <stream_labels> while adding an entry with length <entry_size> bytes

Cause:

A log line exceeds the maximum allowed size.

Default configuration:

Resolution:

Properties:

Error: invalid_labels

Error message:

error parsing labels <labels> with error: <parse_error>

Cause:

Label names or values contain invalid characters or don't follow Prometheus naming conventions:

Default configuration:

This validation is always enabled and cannot be disabled.

Resolution:

Properties:

Error: missing_labels

Error message:

error at least one label pair is required per stream

Cause:

A log stream was submitted without any labels. Loki requires at least one label pair for each log stream.

Default configuration:

This validation is always enabled and cannot be disabled. Every stream must have at least one label.

Resolution:

Ensure all log streams have at least one label. Update your log shipping configuration to include labels:

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.source.file "logs" {
  targets = [
    {
      __path__ = "/var/log/*.log",
      job      = "myapp",
      environment = "production",
    },
  ]
  forward_to = [loki.write.default.receiver]
}

Properties:

Error: too_far_behind

These errors occur when log entries arrive in the wrong chronological order.

Error messages:

Cause:

Logs are being ingested with timestamps that violate Loki's ordering constraints:

Default configuration:

Resolution:

Properties:

Error: greater_than_max_sample_age

Error message:

entry for stream <stream_labels> has timestamp too old: <timestamp>, oldest acceptable timestamp is: <cutoff>

Cause:

The log entry's timestamp is older than the configured maximum sample age. This prevents ingestion of very old logs.

Default configuration:

Resolution:

Properties:

Error: too_far_in_future

Error message:

entry for stream <stream_labels> has timestamp too new: <timestamp>

Cause:

The log entry's timestamp is further in the future than the configured grace period allows.

Default configuration:

Resolution:

Properties:

Error: max_label_names_per_series

Error message:

entry for stream <stream_labels> has <count> label names; limit <limit>

Cause:

The stream has more labels than allowed.

Default configuration:

Resolution:

{{< admonition type="warning" >}} We strongly recommend against increasing max_label_names_per_series, doing so creates a larger index which hurts query performance as well as opens the door for cardinality explosions. You should be able to categorize your logs with 15 labels or typically much less. In all our years of running Loki, out of thousands of requests to increase this value, the number of valid exceptions we have seen can be counted on one hand. {{< /admonition >}}

Properties:

Error: label_name_too_long

Error message:

stream <stream_labels> has label name too long: <label_name>

Cause:

A label name exceeds the maximum allowed length.

Default configuration:

Resolution:

Properties:

Error: label_value_too_long

Error message:

stream <stream_labels> has label value too long: <label_value>

Cause:

A label value exceeds the maximum allowed length.

Default configuration:

Resolution:

Properties:

Error: duplicate_label_names

Error message:

stream <stream_labels> has duplicate label name: <label_name>

Cause:

The stream has two or more labels with identical names.

Default configuration:

This validation is always enabled and cannot be disabled. Duplicate label names are never allowed.

Resolution:

Properties:

Error: request_body_too_large

Error message:

Request body too large: <size> bytes, limit: <limit> bytes

Cause:

The HTTP compressed push request body exceeds the configured limit in your gateway/reverse proxy or service that fronts Loki after decompression.

Default configuration:

Resolution:

Properties:

Storage errors

Storage errors occur when Loki cannot write data to its storage backend or the Write-Ahead Log (WAL).

Error: Disk space full

Error message:

no space left on device

Cause:

Local storage has run out of space. This can affect ingester WAL, chunk cache, or temporary files.

Default configuration:

Disk space is not limited by Loki configuration; it depends on your infrastructure provisioning.

Resolution:

Properties:

Error: Chunk write failure

Error message:

failed to store chunks: storage unavailable

Common causes and errors:

Resolution:

Properties:

Error: WAL disk full

Error logged:

Error writing to WAL, disk full, no further messages will be logged for this error

Metric: loki_ingester_wal_disk_full_failures_total

Cause:

By default, the ingester monitors disk usage on the volume that stores the WAL and rejects new writes once usage reaches 90% of capacity (-ingester.wal-disk-full-threshold), so that it never accepts data it cannot persist. This error and the associated log message describe a different, rarer case: the disk still filled up completely, either because this throttling is disabled (threshold set to 0), or because a write reached the disk in the short window before the check could catch it. In that case, the ingester continues accepting writes but doesn't log them to the WAL, losing durability guarantees for those entries.

Default configuration:

Resolution:

Properties:

{{< admonition type="note" >}} This section describes the disk becoming completely full. In normal operation, the ingester's disk usage throttle (-ingester.wal-disk-full-threshold, default 90%) rejects writes to that ingester before the disk actually fills up, so that durability guarantees are preserved. Because Loki replicates each write to multiple ingesters, a client write can still succeed even while one ingester is throttled. Use metric loki_ingester_wal_disk_usage_percent to monitor disk usage. {{< /admonition >}}

Error: WAL corruption

Error message:

encountered WAL read error, attempting repair

Metric: loki_ingester_wal_corruptions_total

Cause:

The WAL has become corrupted, possibly due to:

Default configuration:

Resolution:

Properties:

{{< admonition type="note" >}} Loki prioritizes availability over complete data recovery. The replication factor provides redundancy if one ingester loses WAL data. Recovered data may be incomplete if corruption is severe. {{< /admonition >}}

Network and connectivity errors

These errors occur due to network issues or service unavailability.

Error: Connection refused

Error message:

connection refused when connecting to loki:3100

Cause:

The Loki service is unavailable or not listening on the expected port.

Default configuration:

Resolution:

Properties:

Error: Context deadline exceeded

Error message:

context deadline exceeded

Cause:

Requests are timing out due to slow response times or network issues.

Default configuration:

Resolution:

Properties:

Error: Client canceled request (Code 499)

Error message:

level=warn method=/logproto.Querier/GetChunkIDs duration=<duration> msg=gRPC err="rpc error: code = Code(499) desc = the request was cancelled by the client"

Cause:

A downstream client (such as Grafana, a load balancer, or another Loki component) canceled a gRPC request before the server finished processing it. Code(499) is a non-standard gRPC status code derived from the HTTP 499 status ("Client Closed Request"), which Loki uses internally when it detects a Go context.Canceled error. Because it is not the standard gRPC Canceled code (code 1), the gRPC logging middleware logs it at warn level instead of debug level.

The method field indicates which gRPC operation was interrupted. Common methods include /logproto.Querier/GetChunkIDs, /logproto.Querier/Query, and /logproto.Querier/Series. Although these are query-path operations, the error appears in ingester logs because ingesters serve gRPC read requests from queriers.

Common triggers include:

Default configuration:

Resolution:

Properties:

Error: Kafka producer backpressure

These errors occur only when the distributor is configured to write to Kafka (-distributor.kafka-writes-enabled=true), and indicate that the Kafka producer could not keep up with the incoming write volume.

Error messages:

Cause:

The distributor's Kafka producer wasn't able to deliver records to the Kafka brokers within kafka_config.write_timeout, or the amount of unacknowledged, buffered data reached kafka_config.producer_max_buffered_bytes. Both are transient backpressure conditions: the Kafka brokers or the network path to them are slower than the rate at which the distributor is trying to produce records, for example during a broker rollout, a partition rebalance, or a broker-side incident.

Default configuration:

Resolution:

Properties:

Error: Service unavailable

Error message:

service unavailable (503)

Cause:

Loki is temporarily unable to handle requests due to high load or maintenance. This can occur when ingesters are unhealthy or the ring is not ready.

Default configuration:

Resolution:

Properties:

Structured metadata errors

These errors occur when using structured metadata incorrectly.

Error: Structured metadata disabled

Error message:

stream <stream_labels> includes structured metadata, but this feature is disallowed. Please see limits_config.allow_structured_metadata or contact your Loki administrator to enable it

Cause:

allow_structured_metadata defaults to true, so this error usually means it has been explicitly disabled, either globally or as a per-tenant override.

Default configuration:

Resolution:

Properties:

Error: Structured metadata too large

Error message:

stream '{job="app"}' has structured metadata too large: '70000' bytes, limit: '65536' bytes. Please see limits_config.max_structured_metadata_size or contact your Loki administrator to increase it

Cause:

The structured metadata size exceeds the configured limit.

Default configuration:

Resolution:

Properties:

Error: Too many structured metadata entries

Error message:

stream '{job="app"}' has too many structured metadata labels: '150', limit: '128'. Please see limits_config.max_structured_metadata_entries_count or contact your Loki administrator to increase it

Cause:

The number of structured metadata entries exceeds the limit.

Default configuration:

Resolution:

Properties:

Blocked ingestion errors

These errors occur when ingestion is administratively blocked.

Ingestion blocked

Error message:

ingestion blocked for user <tenant> until <time> with status code 260

Or for policy-specific blocks:

ingestion blocked for user <tenant> (policy: <policy>) until <time> with status code 260

Cause:

Ingestion has been administratively blocked for the tenant, typically due to policy violations, billing issues, or maintenance. Status code 260 is a custom Loki status code for blocked ingestion.

Default configuration:

Resolution:

Properties:

Troubleshooting workflow

Follow this workflow when investigating ingestion issues:

Additional resources

Troubleshoot Loki operations

This guide helps you troubleshoot errors that occur during Loki operations, including configuration issues, storage backend problems, cluster communication failures, and service component errors. These errors are distinct from ingestion (write path) and query (read path) errors covered in separate troubleshooting topics.

Before you begin, ensure you have the following:

Configuration errors

Configuration errors occur during Loki startup or when loading runtime configuration. These errors prevent Loki from starting or operating correctly.

Error: Multiple config errors found

Error message:

MULTIPLE CONFIG ERRORS FOUND, PLEASE READ CAREFULLY
<list of configuration errors>

Cause:

Multiple configuration validation errors were detected during startup. Loki aggregates all configuration errors rather than failing on the first one.

Resolution:

Properties:

Error: Too many storage configs

Error message:

too many storage configs provided in the common config, please only define one storage backend

Cause:

Multiple storage backends are configured in the common configuration section. Loki requires a single storage backend for the common config.

Resolution:

Properties:

Error: Persist tokens path prefix required

Error message:

if persist_tokens is true, path_prefix MUST be defined

Cause:

The persist_tokens option is enabled for a ring but no path_prefix is specified. Loki needs a path to store the token file.

Resolution:

Properties:

Error: Conflicting gRPC client configs

Error message:

both `grpc_client_config` and (`query_frontend_grpc_client` or `query_scheduler_grpc_client`) are set at the same time. Please use only `query_frontend_grpc_client` and `query_scheduler_grpc_client`

Cause:

Both the deprecated grpc_client_config and the newer specific gRPC client configs are set. These are mutually exclusive.

Resolution:

Properties:

Error: Schema v13 required for structured metadata

Error message:

CONFIG ERROR: schema v13 is required to store Structured Metadata and use native OTLP ingestion, your schema version is <version>. Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update to schema v13 or newer before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure

Cause:

Structured metadata is enabled but the active schema version is older than v13. Structured metadata requires schema v13 or newer.

Resolution:

Properties:

Error: TSDB index type required for structured metadata

Error message:

CONFIG ERROR: `tsdb` index type is required to store Structured Metadata and use native OTLP ingestion, your index type is `<type>` (defined in the `store` parameter of the schema_config). Set `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false` and restart Loki. Then proceed to update the schema to use index type `tsdb` before re-enabling this config, search for 'Storage Schema' in the docs for the schema update procedure

Cause:

Structured metadata is enabled but the active index type is not TSDB. Structured metadata requires the TSDB index type.

Resolution:

Properties:

Error: TSDB directories not configured

Error message:

CONFIG ERROR: `tsdb` index type is configured in at least one schema period, however, `storage_config`, `tsdb_shipper`, `active_index_directory` is not set, please set this directly or set `path_prefix:` in the `common:` section

Or:

CONFIG ERROR: `tsdb` index type is configured in at least one schema period, however, `storage_config`, `tsdb_shipper`, `cache_location` is not set, please set this directly or set `path_prefix:` in the `common:` section

Cause:

The TSDB index type is configured in the schema but required local directories for index files are not set.

Resolution:

Properties:

Error: Compactor working directory empty

Error message:

CONFIG ERROR: `compactor:` `working_directory:` is empty, please set a valid directory or set `path_prefix:` in the `common:` section

Cause:

The compactor requires a working directory for index compaction, but none is configured.

Resolution:

Properties:

Error: Unrecognized index or store type

Error message:

unrecognized `store` (index) type `<type>`, choose one of: <supported_types>

Or:

unrecognized `object_store` type `<type>`, which also does not match any named_stores. Choose one of: <supported_types>. Or choose a named_store

Cause:

The schema configuration references an index type or object store type that Loki does not recognize.

Resolution:

Properties:

Error: Overrides exporter requires runtime configuration

Error message:

overrides-exporter has been enabled, but no runtime configuration file was configured

Cause:

The overrides-exporter target is enabled but no runtime configuration file is provided. The overrides-exporter needs a runtime config to expose tenant-specific limit overrides as metrics.

Resolution:

Properties:

Error: Invalid override for tenant

Error message:

invalid override for tenant <tenant>: <details>

Cause:

The runtime configuration file contains an invalid override for a specific tenant. The override failed validation.

Resolution:

Properties:

Error: Retention period too short

Error message:

retention period must be >= 24h was <duration>

Cause:

A stream-level retention rule specifies a retention period shorter than 24 hours, which is the minimum allowed.

Resolution:

Properties:

Error: Invalid query store max look back period

Error message:

it is an error to specify a non zero `query_store_max_look_back_period` value when using any object store other than `filesystem`

Cause:

The query_store_max_look_back_period is set to a non-zero value with a storage backend other than filesystem. This setting only applies to local filesystem storage.

Resolution:

Properties:

Authentication and tenant errors

Authentication and tenant errors occur when requests are missing required tenant identification or when tenant IDs are invalid. In multi-tenant mode, every request must include a valid tenant ID.

Error: No org ID

Error message:

no org id

Cause:

A request was made to Loki without the required X-Scope-OrgID header. In multi-tenant mode, every request must identify the tenant.

Resolution:

Properties:

Error: Multiple org IDs present

Error message:

multiple org IDs present

Cause:

The request contains multiple different tenant IDs, but the operation requires a single tenant. This can happen when a request is forwarded through multiple proxies that each inject a tenant ID.

Resolution:

Properties:

Error: Tenant ID too long

Error message:

tenant ID is too long: max 150 characters

Cause:

The tenant ID exceeds the maximum allowed length of 150 characters.

Resolution:

Properties:

Error: Unsafe tenant ID

Error message:

tenant ID is '.' or '..'

Cause:

The tenant ID is set to . or .., which are reserved filesystem path components and could cause path traversal issues.

Resolution:

Properties:

Error: Tenant ID contains unsupported character

Error message:

tenant ID '<id>' contains unsupported character '<char>'

Cause:

The tenant ID contains characters that are not allowed. Tenant IDs must consist of alphanumeric characters, hyphens, underscores, and periods.

Resolution:

Properties:

Error: Deletion not available for tenant

Error message:

deletion is not available for this tenant

Cause:

A delete request was submitted for a tenant, but the deletion API endpoints aren't reachable. This happens when retention_enabled is false in the compactor configuration (its default), or when the tenant's deletion_mode override has been explicitly set to disabled.

Resolution:

Properties:

Storage backend errors

Storage backend errors occur when Loki cannot communicate with or properly configure object storage (Amazon S3, Google Cloud Services, Microsoft Azure, Swift, or filesystem).

Error: Unsupported storage backend

Error message:

unsupported storage backend

Cause:

The specified storage backend type is not recognized. This typically occurs when a typo exists in the storage type configuration.

Resolution:

Properties:

Error: Invalid characters in storage prefix

Error message:

storage prefix contains invalid characters, it may only contain digits, English alphabet letters and dashes

Cause:

The storage path prefix contains invalid characters. Only alphanumeric characters and dashes are allowed.

Resolution:

Properties:

Error: Unsupported S3 SSE type

Error message:

unsupported S3 SSE type

Cause:

The S3 server-side encryption (SSE) type is not supported. Loki supports specific SSE types.

Resolution:

Properties:

Error: Invalid S3 SSE encryption context

Error message:

invalid S3 SSE encryption context

Cause:

The SSE-KMS encryption context is malformed and cannot be parsed as valid JSON.

Resolution:

Properties:

Error: Invalid S3 endpoint prefix

Error message:

the endpoint must not prefixed with the bucket name

Cause:

The S3 endpoint incorrectly includes the bucket name as a prefix. This can cause path-style vs virtual-hosted-style URL issues.

Resolution:

Properties:

Error: Invalid STS endpoint

Error message:

sts-endpoint must be a valid url

Cause:

The AWS STS (Security Token Service) endpoint URL is malformed or invalid.

Resolution:

Properties:

Error: Azure connection string malformed

Error message:

connection string is either blank or malformed. The expected connection string should contain key value pairs separated by semicolons. For example 'DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>;EndpointSuffix=core.windows.net'

Cause:

The Azure storage connection string is missing or doesn't follow the expected format.

Resolution:

Properties:

Error: Unrecognized named storage config

Error message:

unrecognized named storage config <name>

Or for specific backends:

unrecognized named s3 storage config <name>
unrecognized named gcs storage config <name>
unrecognized named azure storage config <name>
unrecognized named filesystem storage config <name>
unrecognized named swift storage config <name>

Or for an unrecognized store type:

unrecognized named storage type: <storeType>

Cause:

A named storage configuration referenced in the schema config doesn't exist in the named stores configuration.

Resolution:

Properties:

Cache errors

Cache errors occur when Loki cannot connect to or communicate with caching backends (Memcached, Redis).

Error: Redis client setup failed

Error message:

redis client setup failed: <details>

Cause:

Loki cannot establish a connection to the Redis server. Common causes include:

Resolution:

Properties:

Error: Could not lookup Redis host

Error message:

could not lookup host: <hostname>

Cause:

DNS resolution failed for the Redis hostname.

Resolution:

Properties:

Error: Unexpected Redis PING response

Error message:

redis: Unexpected PING response "<response>"

Cause:

The Redis server returned an unexpected response to a PING command. This could indicate:

Resolution:

Properties:

Error: Multiple cache systems not supported

Error message:

use of multiple cache storage systems is not supported

Cause:

Both Memcached and Redis cache backends are configured for the same cache type. Only one caching backend is supported per cache type.

Resolution:

Properties:

Error: No cache configured

Error message:

no cache configured

Cause:

A results cache is required for the query frontend but no cache configuration was provided.

Resolution:

Properties:

Error: Error loading cache generation numbers

Error message:

msg="error loading cache generation numbers" err="unexpected status code: 403"

Or from the compactor HTTP client:

msg="error getting cache gen numbers from the store" err="unexpected status code: 403"

Metric: loki_delete_cache_gen_load_failures_total

Cause:

Loki uses cache generation numbers to invalidate query caches when log deletion requests are processed. The cache generation number loader periodically fetches these numbers from the compactor. When the compactor returns HTTP 403, it means the deletion API is not enabled for the tenant. The loader logs this error and increments the loki_delete_cache_gen_load_failures_total metric.

Other non-403 causes include:

Resolution:

Properties:

Ring and cluster communication errors

Ring errors occur when Loki components cannot properly communicate through the hash ring, which is used to distribute work across instances. The ring is fundamental to Loki's distributed operation.

Error: Too many unhealthy instances in the ring

Error message:

too many unhealthy instances in the ring

Cause:

The ring contains too many unhealthy instances to satisfy the replication factor. For example, with a replication factor of 3, at least 3 healthy instances must be available.

Resolution:

Properties:

Error: Empty ring

Error message:

empty ring

Cause:

No instances are registered in the ring. This typically occurs during initial cluster startup, for example if your ingesters are OOM crashing, or due to misconfiguration.

Resolution:

Properties:

Error: Instance not found in the ring

Error message:

instance <id> not found in the ring

Cause:

A specific instance is expected to be in the ring but isn't registered. This can happen after a restart if the instance hasn't re-joined the ring yet.

Resolution:

Properties:

Error: Instance owns no tokens

Error message:

this instance owns no tokens

Cause:

The instance has joined the ring but hasn't claimed any tokens. Without tokens, the instance cannot receive any work. This can happen if:

Resolution:

Properties:

Error: Error talking to the KV store

Error message:

error talking to the KV store

Cause:

The instance cannot communicate with the key-value store used for ring state. The KV store (Consul, etcd, or memberlist) is required for ring coordination.

Resolution:

Properties:

Error: No ring returned from the KV store

Error message:

no ring returned from the KV store

Cause:

The KV store responded but returned an empty or invalid ring descriptor. This can happen if the KV store was recently initialized or its data was cleared.

Resolution:

Properties:

Error: Failed to join memberlist cluster

Error message:

failed to join memberlist cluster on startup

Or:

joining memberlist cluster failed

Cause:

The instance could not join the memberlist gossip cluster. Common causes:

Resolution:

Properties:

Error: Re-joining memberlist cluster failed

Error message:

re-joining memberlist cluster failed

Cause:

After being disconnected from the memberlist cluster, the instance failed to rejoin. This can happen during network partitions or after prolonged network issues.

Resolution:

Properties:

Component readiness errors

Readiness errors occur when Loki components are not ready to serve requests. These errors are returned by the /ready health check endpoint and prevent load balancers from routing traffic to unready instances.

Error: Application is stopping

Error message:

Application is stopping

Cause:

Loki is shutting down and no longer accepting new requests. This is normal during graceful shutdown.

Resolution:

Properties:

Error: Some services are not running

Error message:

Some services are not Running:
<state>: <count>
<state>: <count>

For example:

Some services are not Running:
Starting: 1
Failed: 2

Cause:

One or more internal Loki services have failed to start or have stopped unexpectedly. The error message lists each service state with a count of services in that state.

Resolution:

Properties:

Error: Ingester not ready

Error message:

Ingester not ready: <details>

When the ingester's own state check fails, <details> contains the ingester state, giving the full message:

Ingester not ready: ingester not ready: <state>

Where <state> is the service state, for example Starting, Stopping, or Failed.

Cause:

The ingester is not in a ready state to accept writes or serve reads. The detail message indicates the specific reason, such as:

Resolution:

Properties:

Error: No queriers connected to query frontend

Error message:

Query Frontend not ready: not ready: number of queriers connected to query-frontend is 0

Cause:

The query frontend has no querier workers connected. Without queriers, the frontend cannot process any queries. This typically occurs when:

Resolution:

Properties:

Error: No schedulers connected to frontend worker

Error message:

Query Frontend not ready: not ready: number of schedulers this worker is connected to is 0

Cause:

The query frontend worker has no active connections to any query scheduler. This prevents the frontend from dispatching queries.

Resolution:

Properties:

gRPC and message size errors

gRPC errors occur during inter-component communication. Loki components communicate using gRPC for ring coordination, query execution, and data transfer.

Error: Message size too large

Error message:

message size too large than max (<size> vs <max>)

Or for the decompressed body:

decompressed message size too large than max (<size> vs <max>)

Cause:

The compressed or decompressed body of an HTTP push request to the distributor exceeds the configured limit.

Default configuration:

Resolution:

Properties:

Error: Response larger than max message size

Error message:

response larger than the max message size (<size> vs <max>)

Cause:

A query result from the querier to the frontend exceeds the maximum allowed gRPC response size. This typically happens with queries that return very large result sets.

Default configuration:

Resolution:

Properties:

Error: Compressed message size exceeds limit

Error message:

compressed message size <size> exceeds limit <limit>

Cause:

The compressed body of an HTTP push request exceeds the distributor's configured limit. This check runs after the request body has been fully read and validates the total compressed size against the configured maximum.

Default configuration:

Resolution:

Properties:

TLS and certificate errors

TLS errors occur when Loki or its clients cannot establish secure connections due to certificate issues.

Error: TLS certificate loading failed

Error message:

error loading ca cert: <path>

Or:

error loading client cert: <path>

Or:

error loading client key: <path>

Or:

failed to load TLS certificate <cert_path>,<key_path>

Cause:

Loki cannot load TLS certificates from the specified paths. Common causes:

Resolution:

Properties:

Error: TLS configuration error

Error message:

error generating http tls config: <details>

Or:

error generating grpc tls config: <details>

Where <details> may include messages such as TLS version %q not recognized, cipher suite %q not recognized, or unknown TLS version: <version>.

Cause:

The TLS configuration is invalid. This can happen when:

Resolution:

Properties:

DNS resolution errors

DNS errors occur when Loki cannot resolve hostnames for service discovery or backend connections.

Error: DNS lookup timeout

Error message:

msg="failed to resolve server addresses" err="... DNS lookup timeout: [<address>] ..."

Cause:

DNS resolution exceeded the 5-second timeout when trying to resolve addresses for Loki service discovery or backend connections. This error is emitted by the index gateway and bloom gateway DNS discovery loops. The DNS lookup timeout: [<address>] string is the context cause embedded within the err field; the full address list is formatted as a Go slice (for example, [dns+loki-index-gateway.loki.svc.cluster.local:9095]).

Resolution:

Properties:

Scheduler and frontend errors

These errors relate to query scheduling, frontend workers, and queue management.

Error: Scheduler is not running

Error message:

scheduler is not running

Cause:

The query scheduler service is not in a running state. This can occur when:

Resolution:

Properties:

Error: Too many outstanding requests

Error message:

too many outstanding requests

Cause:

The query queue has reached its maximum capacity. This indicates the system is overloaded with queries.

Resolution:

Properties:

Error: Querying is disabled

Error message:

querying is disabled, please contact your Loki operator

Cause:

Query parallelism has been set to zero, effectively disabling all queries. This is typically done intentionally during maintenance.

Resolution:

Properties:

Error: No frontend address

Error message:

no frontend address

Cause:

The scheduler received a request from a frontend but no frontend address was provided for sending responses back.

Resolution:

Properties:

Error: Scheduler shutting down

Error message:

scheduler is shutting down

Cause:

The frontend scheduler worker detected that the scheduler is in shutdown mode and cannot accept new requests.

Resolution:

Properties:

Index gateway errors

Index gateway errors occur when queriers cannot communicate with index gateways for index lookups.

Error: Index gateway unhealthy in ring

Error message:

index-gateway is unhealthy in the ring

Cause:

The index gateway instance detects itself as unhealthy in the ring and refuses to process queries. This is a self-check: before handling tenant requests, the gateway verifies it appears in the set of healthy ring members.

Resolution:

Properties:

Error: No index gateway instances found

Error message:

no index gateway instances found for tenant <tenant>

Cause:

No index gateway instances are available in the ring to serve the tenant's request. This could be due to:

Resolution:

Properties:

Compactor and retention errors

Compactor errors occur during index compaction or retention enforcement.

Error: No chunks found in table

Error message:

no chunks found in table, please check if there are really no chunks and manually drop the table or see if there is a bug causing us to drop whole index table

Cause:

The compactor found an empty index table during retention processing. This could indicate:

Resolution:

Properties:

Error: Delete request store not configured

Error message:

compactor.delete-request-store should be configured when retention is enabled

Cause:

Retention is enabled but no store is configured for tracking delete requests.

Resolution:

Properties:

Error: Max compaction parallelism invalid

Error message:

max compaction parallelism must be >= 1

Cause:

The compactor's parallelism setting is configured to zero or a negative number.

Resolution:

Properties:

Error: Delete request not found

Error message:

could not find delete request with given id

Cause:

An attempt to cancel a delete request failed because no matching request exists.

Resolution:

Properties:

Error: Retention is not enabled

Error message:

Retention is not enabled

Cause:

A delete request was submitted but retention is not enabled in the compactor configuration. Delete requests require retention to be enabled.

Resolution:

Properties:

Error: Invalid delete request time format

Error message:

invalid start time: require unix seconds or RFC3339 format

Or:

invalid end time: require unix seconds or RFC3339 format

Cause:

The start or end time in a delete request is not in a valid format.

Resolution:

Properties:

Error: Delete request already processed

Error message:

deletion of request which is in process or already processed is not allowed

Cause:

An attempt was made to cancel a delete request that is already being processed or has completed processing.

Resolution:

Properties:

Error: Invalid max_interval for delete request

Error message:

invalid max_interval: valid time units are 's', 'm', 'h'

Or:

max_interval can't be greater than <configured-limit>

Or:

max_interval can't be greater than the interval to be deleted (<duration>)

Cause:

The max_interval parameter on a delete request has an invalid value, exceeds the configured delete_max_interval limit, or exceeds the time range of the delete request itself.

Resolution:

Properties:

Ruler errors

Ruler errors occur when evaluating alerting rules or recording rules.

Error: Invalid ruler evaluation config

Error message:

invalid ruler evaluation config: <details>

Cause:

The ruler evaluation mode configuration is invalid.

Resolution:

Properties:

Error: Ruler remote write config conflict

Error message:

ruler remote write config: both 'client' and 'clients' options are defined; 'client' is deprecated, please only use 'clients'

Cause:

Both the deprecated client and the new clients configuration options are set for ruler remote write.

Resolution:

Properties:

Error: Remote write enabled but no URL configured

Error message:

remote-write enabled but no clients URL are configured

Or when multiple clients are configured in the clients map and one entry is missing a URL:

remote-write enabled but client '<name>' URL for tenant <client-id> is not configured

Cause:

Remote write is enabled for the ruler but no destination URL is configured. The first variant occurs when the clients map is empty. The second occurs when a named entry in the clients map has no url set; <client-id> is the map key for that entry, not a tenant ID.

Resolution:

Properties:

Error: Rule result is not a vector or scalar

Error message:

rule result is not a vector or scalar

Cause:

A rule evaluation returned an unexpected result type. Both recording rules and alerting rules must produce vector or scalar results. A plain log-stream expression (one that returns log lines rather than a numeric metric) triggers this error in either rule type.

Resolution:

Properties:

Error: Ruler WAL closed

Error message:

WAL storage closed

Cause:

An operation was attempted on the ruler's write-ahead log (WAL) after it was closed. This typically occurs during shutdown.

Resolution:

Properties:

Kafka integration errors

These errors occur when Loki is configured to use Kafka for ingestion.

Error: Missing Kafka address

Error message:

the Kafka address has not been configured

Cause:

Kafka ingestion is enabled but no Kafka broker address is configured.

Resolution:

Properties:

Error: Missing Kafka topic

Error message:

the Kafka topic has not been configured

Cause:

Kafka ingestion is enabled but no topic name is configured.

Resolution:

Properties:

Error: Inconsistent SASL username and password

Error message:

both sasl username and password must be set

Cause:

Only one of the Simple Authentication and Security Layer (SASL) username or password is configured. Both must be set together.

Resolution:

Properties:

Error: Kafka enabled in distributor but not in ingester

Error message:

kafka is enabled in distributor but not in ingester

Cause:

Kafka is configured for the distributor but the ingester isn't configured to read from Kafka. Both must be configured together.

Resolution:

Properties:

Error: Invalid producer max record size bytes

Error message:

the configured producer max record size bytes must be a value between 1048576 and 15983616

Cause:

The kafka_config.producer_max_record_size_bytes value is outside the allowed range.

Resolution:

Properties:

Error: Invalid producer max inflight requests per broker

Error message:

the configured producer max inflight requests per broker must be greater than or equal to 1

Cause:

The kafka_config.producer_max_inflight_requests_per_broker value is less than 1.

Resolution:

Properties:

Error: Invalid producer linger

Error message:

the configured producer linger must be greater than or equal to 0

Cause:

The kafka_config.producer_linger value is negative.

Resolution:

Properties:

Error: Partition ring required for Kafka writes

Error message:

partition ring is required for kafka writes

Cause:

distributor.kafka_writes_enabled is true, but the distributor wasn't started with a partition ring. This usually means the partition ring key-value store isn't configured consistently across distributors and ingesters.

Resolution:

Properties:

Error: Single entry size exceeds maximum

Error message:

single entry size (%d) exceeds maximum allowed size (%d)

Cause:

A single log line is larger than the configured kafka_config.producer_max_record_size_bytes. Loki splits large write requests into multiple Kafka records, but it can't split a single log line across records.

Resolution:

Properties:

Error: Dataobj builder topic required

Error message:

topic is required

Cause:

The experimental data object builder, used for columnar storage, is enabled but no Kafka topic is configured for it.

Resolution:

Properties:

Error: Ingest-limits configuration errors

Error messages:

num partitions must be greater than 0
consumer group must be set
topic must be set

Cause:

The optional ingest-limits service is enabled (ingest_limits.enabled: true), but one or more of its required Kafka settings are missing or invalid.

Resolution:

Properties:

Error: Ingest-limits readiness errors

Error messages:

waiting to be assigned some partitions
partitions are not ready

Cause:

The ingest-limits service hasn't finished joining the partition ring yet, or one or more of its assigned partitions aren't ready. This is usually a transient condition during startup or a rolling restart.

Resolution:

Properties:

Error: Generic Kafka protocol errors

Error messages:

unexpected response type
malformed response

Cause:

The Kafka client received a response from the broker that it didn't expect, or couldn't parse. This usually points to a Kafka, or Kafka-protocol-compatible broker, that uses a protocol version Loki doesn't fully support, or to a network issue that corrupted the response.

Resolution:

Properties:

Bloom gateway errors

Bloom gateway errors occur when using bloom filters for query acceleration.

Error: Invalid bloom gateway addresses

Error message:

addresses requires a list of comma separated strings in DNS service discovery format with at least one item

Cause:

The bloom_gateway.client.addresses configuration field is empty or unset.

Resolution:

Properties:

Error: Request time range must span exactly one day

Error message:

request time range must span exactly one day

Cause:

Bloom gateway requests must be for exactly one day of data due to how bloom blocks are organized.

Resolution:

Properties:

Error: From time must not be after through time

Error message:

from time must not be after through time

Cause:

The bloom gateway received a request where the start time (from) is later than the end time (through).

Resolution:

Properties:

Write-ahead log (WAL) errors

WAL errors occur when the ingester cannot properly manage its write-ahead log.

Error: WAL is stopped

Error message:

wal is stopped

Cause:

An operation was attempted on the WAL after it was stopped. This typically occurs during shutdown or after a fatal error.

Resolution:

Properties:

Error: Invalid checkpoint duration

Error message:

invalid checkpoint duration: <duration>

Cause:

The WAL checkpoint duration is set to an invalid value (likely zero or negative).

Resolution:

Properties:

Error: Invalid disk full threshold

{{< admonition type="note" >}} The disk_full_threshold configuration option was introduced in Loki 3.7. This error does not occur in earlier releases. {{< /admonition >}}

Error message:

invalid disk full threshold: <value> (must be between 0 and 1)

Cause:

The WAL disk full threshold is set to a value outside the valid range. Valid values are between 0 and 1 (inclusive), where 0 disables throttling and values greater than 0 represent the fraction of disk capacity at which writes are throttled.

Resolution:

Properties:

Ingester lifecycle errors

Ingester lifecycle errors occur during ingester startup, shutdown, or state transitions.

Error: Ingester is shutting down

Error message:

Ingester is shutting down

Cause:

The ingester is in the process of shutting down and is no longer accepting writes. This error (also known as ErrReadOnly) is returned when a push request arrives during graceful shutdown. During this period the ingester may still serve reads for data it holds in memory.

Resolution:

Properties:

Error: Ingester is stopping or already stopped

Error message:

Ingester is stopping or already stopped.

Cause:

The ingester's shutdown management endpoint (POST /loki/api/v1/ingester/shutdown) was called when the ingester was not in a Running state. This happens when the endpoint is called a second time during an in-progress shutdown or after the ingester has already stopped. This error is returned by the shutdown endpoint, not by the log-write or query paths.

Resolution:

Properties:

Error: Failed to start partition reader

Error message:

failed to start partition reader: <details>

Cause:

The ingester could not start its Kafka partition reader. This occurs when Kafka ingestion is enabled but the partition reader fails to initialize.

Resolution:

Properties:

Error: Failed to start partition ring lifecycler

Error message:

failed to start partition ring lifecycler: <details>

Cause:

The ingester could not start its Kafka partition ring lifecycler during startup. This is a separate component from the partition reader; it manages the ingester's membership in the partition ring. This only occurs when Kafka ingestion is enabled.

Resolution:

Properties:

Error: Lifecycler failed

Error message:

lifecycler failed: <details>

Cause:

The ingester's lifecycler (which manages ring membership) encountered a fatal error. This prevents the ingester from participating in the ring.

Resolution:

Properties:

Pattern ingester errors

Pattern ingester errors occur when using the pattern ingester for automatic log pattern detection.

Error: Pattern ingester replication factor must be 1

Error message:

pattern ingester replication factor must be 1

Cause:

The pattern ingester is configured with a replication factor other than 1. Currently, the pattern ingester only supports a replication factor of 1.

Resolution:

Properties:

Error: Pattern ingester retain-for too short

Error message:

retain-for (<duration>) must be greater than or equal to chunk-duration (<duration>)

Cause:

The pattern ingester's retain_for duration is shorter than max_chunk_age, which would cause data loss.

Resolution:

Properties:

Error: Pattern ingester chunk-duration too short

Error message:

chunk-duration (<duration>) must be greater than or equal to sample-interval (<duration>)

Cause:

The pattern ingester's max_chunk_age is shorter than pattern_sample_interval. Chunks must span at least one sample interval to hold any data.

Resolution:

Properties:

Error: Pattern ingester volume threshold out of range

Error message:

volume_threshold (<value>) must be between 0 and 1

Cause:

The volume_threshold value is outside the valid range of 0 to 1. This setting controls what fraction of log volume the pattern ingester tracks — only patterns representing the top X% of log volume are persisted.

Resolution:

Properties:

API parameter errors

These errors occur when API requests contain invalid parameters.

Error: Invalid direction

Error message:

invalid direction '<value>'

Cause:

The direction query parameter contains an invalid value.

Resolution:

Properties:

Error: Limit must be a positive value

Error message:

limit must be a positive value

Cause:

The limit parameter is zero or negative.

Resolution:

Properties:

Error: End timestamp must not be before start time

Error message:

end timestamp must not be before or equal to start time

Cause:

The query's end time is before or equal to its start time.

Resolution:

Properties:

Error: Delay for tailing too large

Error message:

delay_for can't be greater than <max>

Cause:

The delay_for parameter for tailing queries exceeds the maximum allowed value.

Resolution:

Properties:

Error: Query filtering requires compactor address

Error message:

query filtering for deletes requires 'compactor_grpc_address' or 'compactor_address' to be configured

Cause:

Query-time filtering for delete requests is enabled but Loki doesn't know how to reach the compactor to retrieve active delete requests.

Resolution:

Properties:

Troubleshoot log queries (READ)

{{< docs/shared lookup="troubleshoot-query.md" source="loki" version="<LOKI_VERSION>" >}}