# Configure # Configuration best practices Grafana Loki is under active development, and the Loki team is constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki. ## Configure authentication Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Loki (for example, nginx). Refer to [Manage authentication](https://grafana.com/docs/loki//operations/authentication/) for a list of open-source reverse proxies you can use. ## Configure caching Loki can cache data at many levels, which can drastically improve performance. Details of this will be in a future post. ## Time ordering of logs Loki [accepts out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) _by default_. This section identifies best practices when Loki is _not_ configured to accept out-of-order writes. One issue many people have with Loki is their client receiving errors for out of order log entries. This happens because of this hard and fast rule within Loki: - For any single log stream, logs must always be sent in increasing time order. If a log is received with a timestamp older than the most recent log received for that stream, that log will be dropped. There are a few things to dissect from that statement. The first is this restriction is per stream. Let’s look at an example: ```bash {job="syslog"} 00:00:00 i'm a syslog! {job="syslog"} 00:00:01 i'm a syslog! ``` If Loki received these two lines which are for the same stream, everything would be fine. But what about this case: ```bash {job="syslog"} 00:00:00 i'm a syslog! {job="syslog"} 00:00:02 i'm a syslog! {job="syslog"} 00:00:01 i'm a syslog! <- Rejected out of order! ``` What can you do about this? What if this was because the sources of these logs were different systems? You can solve this with an additional label which is unique per system: ```bash {job="syslog", instance="host1"} 00:00:00 i'm a syslog! {job="syslog", instance="host1"} 00:00:02 i'm a syslog! {job="syslog", instance="host2"} 00:00:01 i'm a syslog! <- Accepted, this is a new stream! {job="syslog", instance="host1"} 00:00:03 i'm a syslog! <- Accepted, still in order for stream 1 {job="syslog", instance="host2"} 00:00:02 i'm a syslog! <- Accepted, still in order for stream 2 ``` But what if the application itself generated logs that were out of order? Well, I'm afraid this is a problem. If you are extracting the timestamp from the log line with something like the [Alloy `stage.timestamp` block](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagetimestamp), you could instead _not_ do this and let your log shipping agent assign a timestamp to the log lines. Or you can hopefully fix it in the application itself. It's also worth noting that the batching nature of the Loki push API can lead to some instances of out of order errors being received which are really false positives. (Perhaps a batch partially succeeded and was present; or anything that previously succeeded would return an out of order entry; or anything new would be accepted.) ## Consider the `snappy` compression algorithm The default `chunk_encoding` is `gzip`, which compresses well but is comparatively slow, and can cause slow query responses under load. `Snappy` is a recommended override for `chunk_encoding` in higher-throughput environments. It performs much better than `gzip` for speed, but it is not as efficient in storage. Whether that tradeoff is acceptable depends on your workload. `LZ4` is a good compromise of speed and compression performance. While compression is slightly slower than `snappy`, the compression ratio is higher, resulting in smaller chunks in object storage. ## Use `chunk_target_size` Using `chunk_target_size` instructs Loki to try to fill all chunks to a target _compressed_ size of 1.5MB. These larger chunks are more efficient for Loki to process. Other configuration variables affect how full a chunk can get. Loki has a default `max_chunk_age` of 2h and `chunk_idle_period` of 30m to limit the amount of memory used as well as the exposure of lost logs if the process crashes. Depending on the compression used (for example, `snappy`, which has less compressibility but faster performance than the `gzip` default), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled. Lots of small, unfilled chunks negatively affect Loki. The team is always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks. If you have an application that can log fast enough to fill these chunks quickly (much less than `max_chunk_age`), then it becomes more reasonable to use dynamic labels to break that up into separate streams. ## Use `-print-config-stderr` or `-log-config-reverse-order` Loki has flags which will dump the entire config object to stderr or the log file when it starts. `-print-config-stderr` works well when invoking Loki from the command line, as you can get a quick output of the entire Loki configuration. `-log-config-reverse-order` is the flag Grafana runs Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana's Explore. ## Recommended production limits ```yaml limits_config: # Rate limits ingestion_rate_strategy: global ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 per_stream_rate_limit: 3MB per_stream_rate_limit_burst: 15MB # Stream limits max_global_streams_per_user: 10000 max_streams_per_user: 0 # Validation max_line_size: 256KB max_line_size_truncate: false max_label_name_length: 1024 max_label_value_length: 2048 max_label_names_per_series: 15 # Time constraints reject_old_samples: true reject_old_samples_max_age: 168h # 7 days creation_grace_period: 10m ``` ## Recommended Ingester configuration ```yaml ingester: # Chunk settings chunk_idle_period: 30m chunk_target_size: 1572864 # 1.5 MB chunk_encoding: snappy max_chunk_age: 2h # WAL settings wal: enabled: true dir: /loki/wal checkpoint_duration: 5m flush_on_shutdown: true ``` # Understanding Loki configuration limits Loki implements configuration limits to maintain system stability, ensure fair resource distribution, and protect performance across all users. You can adjust many limits for specific use cases. Other limits are hard boundaries that you should not exceed, to prevent performance degradation and system instability. This topic explains the critical configuration limits in Loki, why they exist, and the consequences of exceeding them. ## Why limits exist Configuration limits in Loki serve multiple essential purposes: * **System stability:** Limits prevent individual queries or ingestion patterns from exhausting system resources. Without limits, a single tenant or misconfigured application could consume all available memory, CPU, or I/O (input/output) capacity. This can cause cascading failures across the entire system. * **Performance protection:** Loki is optimized for specific usage patterns. Limits ensure that workloads stay within the parameters where Loki performs efficiently. Exceeding these limits often results in exponential performance degradation rather than linear scaling. * **Fair resource allocation:** In multi-tenant environments, limits ensure that no single tenant can monopolize shared resources. Query workers, ingester memory, and storage I/O must be distributed fairly across all users. * **Operational predictability:** Limits allow operators to plan capacity and scale infrastructure appropriately. When operators understand the boundaries of expected workloads, they can provision resources that meet service level agreement (SLA) requirements. Exceeding a limit doesn't just affect the tenant who exceeded it. It can also impact overall system performance and stability, often affecting all tenants in a shared environment. ## Default ingest limits The following limits apply when you push logs to Loki. These are the default values in the Loki open source release. If a request exceeds one of these limits, Loki rejects the affected log lines or streams and returns an error. | Limit | Default value | What happens when you exceed it | | --- | --- | --- | | Maximum active streams per tenant (`max_global_streams_per_user`) | 5,000 | New streams are rejected (`stream_limit`) until existing streams age out. | | Ingestion rate (`ingestion_rate_mb`) | 4 MB per second, per tenant | Requests are rejected and can be retried (`rate_limited`). | | Ingestion burst size (`ingestion_burst_size_mb`) | 6 MB | Requests that exceed the burst allowance are rejected until the rate limit bucket refills. | | Per-stream rate limit (`per_stream_rate_limit`) | 3 MB per second | The stream is rate limited (`per_stream_rate_limit`). | | Per-stream burst limit (`per_stream_rate_limit_burst`) | 15 MB | Bursts that exceed this size are rejected. | | Maximum log line size (`max_line_size`) | 256 KB | Log lines that exceed the limit are discarded (`line_too_long`), unless you configure Loki to truncate them instead. | | Maximum labels per stream (`max_label_names_per_series`) | 15 | Streams that exceed the limit are rejected (`max_label_names_per_series`). | | Maximum length of a label name (`max_length_label_name`) | 1,024 bytes | Label names that exceed the limit are rejected (`label_name_too_long`). | | Maximum length of a label value (`max_length_label_value`) | 2,048 bytes | Label values that exceed the limit are rejected (`label_value_too_long`). | | Maximum structured metadata entries per log line (`max_structured_metadata_entries_count`) | 128 entries | Log lines with more entries are discarded. | | Maximum structured metadata size per log line (`max_structured_metadata_size`) | 64 KB | Log lines that exceed the limit are discarded. | | Maximum accepted sample age (`reject_old_samples_max_age`) | 1 week | Samples older than this age are rejected (`greater_than_max_sample_age`). | Loki rejects samples based on their age only when the `reject_old_samples` setting is `true`, which is the default. If you set `reject_old_samples` to `false`, Loki ignores `reject_old_samples_max_age`. Ingestion rate and maximum active streams aren't fixed for every deployment. Many operators raise these limits for larger deployments. For guidance on how far you can safely raise them, see [Limits we strongly recommend not exceeding](#limits-we-strongly-recommend-not-exceeding). ## Default query limits The following limits apply when you query logs in Loki. These are the default values in the Loki open source release. | Limit | Default value | What happens when you exceed it | | --- | --- | --- | | Query timeout (`query_timeout`) | 1 minute | The query is terminated before it finishes. | | Maximum log entries returned per query (`max_entries_limit_per_query`) | 5,000 | Queries that request more entries than this limit are rejected. | | Maximum unique series returned by a metric query (`max_query_series`) | 500 | The query is rejected. | | Maximum query parallelism (`max_query_parallelism`) | 32 | Additional subqueries wait in a queue until a query worker is free. | | Maximum query time range (`max_query_length`, `store.max-query-length`) | 721 hours (30 days and 1 hour) | The query is rejected. | | Maximum chunks fetched per query (`store.query-chunk-limit`) | 2,000,000 | The query is rejected. | | Maximum stream matchers per query (`max_streams_matcher_per_query`) | 1,000 | The query is rejected. | | Maximum concurrent tail requests (`max_concurrent_tail_requests`) | 10 | Additional tail requests are rejected. | ## Limits we strongly recommend not exceeding Some limits can technically be raised well beyond their defaults, but doing so puts your Loki cluster at risk. This section explains which limits fall into that category, why they exist, and what to do instead of raising them further. ### ingestion_rate_mb - **Default**: 4 MB per second - **Example large deployment value**: 100 MB per second (approximately 260 TB of data per month) - **Recommended maximum for large deployments**: up to 250 MB per second This limit controls the per-tenant ingestion rate in megabytes per second. **Why this limit matters**: Ingestion rate limits protect against cascading failures: - Sudden traffic spikes can overwhelm ingesters before your infrastructure can scale to meet demand. - Unbounded ingestion can exhaust ingester memory. - Unbounded ingestion can exhaust distributor memory. - Network and storage I/O can become bottlenecks. Rate limits provide a safety mechanism that allows the system to scale gracefully, rather than failing all at once. **Consequences of not setting rate limits properly**: - Cascading failure of distributors and ingesters, leading to an outage of the entire write path and the recent-data read path. - Ingesters overwhelmed to the point that their write-ahead logs (WALs) run out of disk space, causing slow WAL replay and difficult recovery. - Object storage overwhelmed by excessive requests. **What to do instead**: - Always set rate limits for every tenant. - Implement exponential backoff in your log shipping clients. - Monitor ingestion patterns and adjust limits proactively. - Scale infrastructure before increasing limits significantly. ### max_global_streams_per_user - **Default**: 5,000 - **Recommended maximum for large deployments**: 80,000 - **Hard limit**: approximately 200,000 (performance issues become unavoidable beyond this point) This limit controls the maximum number of active streams a single tenant can have across all ingesters. Each unique combination of labels creates a separate stream. **Why you should not exceed 200,000 streams**: Streams consume resources throughout the system: - **Ingester memory**: Each stream maintains in-memory chunks. - **Index entries**: Each stream creates entries in the index. - **Object storage operations**: Each stream generates flush operations. - **Query overhead**: More streams means more chunks to scan during queries. Beyond approximately 200,000 streams, these costs become prohibitive regardless of infrastructure size. **Consequences of exceeding**: - Ingester memory exhaustion and crashes. - Extremely slow query performance. - Index query timeouts. - Excessive object storage operations and costs. - Inability to meet performance SLAs. - Potential data loss during ingester failures. **What to do instead**: - Audit your label usage to identify high-cardinality labels. - Use static labels for source identification only. - Move high-cardinality data to structured metadata. - Calculate your expected stream count: `streams = label1_values × label2_values × ...` - Design for fewer than 80,000 streams. ### max_label_names_per_series - **Default**: 15 - **Recommended maximum**: 15 (do not exceed) This limit controls the maximum number of labels you can attach to a single log stream. **Why you should not exceed 15 labels**: You should be able to identify and categorize your logs with fewer than 15 labels. More labels increase the likelihood of higher cardinality and cause logs to fragment into too many streams and chunks. Even labels with a small, fixed set of values still require space in the index, and each additional label makes the index larger and can slow down index operations. Always use as few labels as possible. Having more than 15 labels per stream usually indicates a labeling problem, not a limit you need to raise. Loki is designed for a small set of labels that describe the log source, such as `cluster`, `namespace`, or `job`. Labels aren't meant for storing high-cardinality metadata. **Consequences of exceeding**: - Increased memory usage on ingesters. - A larger index, which leads to slower index operations and slower queries. **What to do instead**: - Use labels only for low-cardinality source identification. - Parse high-cardinality fields at query time using LogQL parsers. - Store high-cardinality metadata in [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) instead of labels. ### max_line_size - **Default**: 256 KB - **Maximum recommended**: 256 KB (do not exceed) This limit controls the maximum size, in bytes, of a single log line during ingestion. **Why you should not exceed 256 KB**: A limit on log line size is fundamental to Loki's design. Increasing this limit can have several unintended consequences that make it hard to run a stable Loki cluster: - Chunks are expected to be a few MB at most. If a log line is larger than this, it can create very large chunks, which hurts query performance because a chunk is the unit of parallelism and is expected to stay around 2 MB or smaller. - Queriers and query frontends become much more likely to run out of memory (OOM). - Increasing this limit also increases the risk of exhausting the gRPC message size limit, which can affect the performance of other tenants. **What to do instead**: - Structure your applications to produce reasonably sized log lines. - Avoid embedding large payloads (such as base64-encoded data or full HTTP bodies) in logs. - Log references to external data instead of the data itself. - Split multi-line events into separate log entries. - Set `max_line_size_truncate` to truncate large log lines instead of dropping them. Note that truncation can break JSON parsing of large log lines. ### Out-of-order ingestion window - **Default**: 1 hour (half of `max_chunk_age`, which defaults to 2 hours) - **Adjustable**: No, not on a per-tenant basis The out-of-order ingestion window controls how far back in time, relative to the newest entry already received for a stream, Loki accepts a new log entry. Loki calculates this window as `max_chunk_age / 2`, so with the default 2-hour `max_chunk_age`, Loki accepts entries up to 1 hour older than the current newest entry in each stream. This window isn't a per-tenant override. It's a cluster-wide setting, because it derives from `max_chunk_age`, which affects chunk flushing behavior across the whole deployment. **Why this limit exists**: The reasons users hit this limit vary widely: misconfiguration, re-ingestion of old data, or logs from temporarily disconnected sources such as vehicles or IoT devices. Loki needs a bounded window so that ingesters know when a chunk is complete and safe to flush. **What to do for historical data**: If you need to ingest logs older than the out-of-order window allows, you have two options: - Temporarily increase `reject_old_samples_max_age` during migrations or backfills, so Loki doesn't reject the old samples outright. - Enable `shard_streams.time_sharding_enabled` for the affected tenant. This setting splits incoming streams by injecting a synthetic `__time_shard__` label, so that no stream is ever longer than `max_chunk_age / 2`. This lets Loki accept very old logs, because the new, shorter streams are never "too far behind" by design. Time sharding has some limitations: - Newly ingested old logs aren't queryable for a few hours, until Loki flushes them to object storage. - Very old logs can still be rejected with a `greater_than_max_sample_age` error if they fall outside `reject_old_samples_max_age`. - Recording rules don't work against the old data. - Time sharding creates new time series (up to 24 per day), so check whether this affects your active streams limit. Disable time sharding after your backfill completes. ### per_stream_rate_limit - **Default**: 3 MB per second sustained, 15 MB burst - **Recommended maximum sustained rate**: 5 MB per second Individual streams have their own rate limits. This prevents a single high-volume stream from overwhelming the ingesters it is distributed to, based on the replication factor. **Why you should not exceed 5 MB per second per stream**: Streams are distributed to a fixed number of ingesters (typically 3, for replication). If a single stream sends data faster than those ingesters can process it: - Ingester memory queues grow. - Chunk flush operations lag behind incoming data. - Memory usage spikes on the affected ingesters. - The affected ingesters risk running out of memory (OOM). Unlike the global rate limit, you cannot solve per-stream limit problems by adding more ingesters, because each stream is always handled by the same set of ingesters. **Consequences of exceeding**: - Memory buildup on specific ingesters. - Increased flush latency. - Potential out-of-memory (OOM) failures on affected ingesters. - Data loss if an ingester crashes. - Uneven load distribution. **What to do instead**: - First, try adjusting automatic stream sharding by lowering `desired_rate` (not below 128 KB per second). - If that isn't enough, increase `per_stream_rate_limit` and `per_stream_rate_limit_burst`. This might require more memory headroom on your ingesters. Increase the burst limit more aggressively than the sustained rate. For example, to handle "bursty" streams, you could raise the sustained rate a little and the burst limit a lot: - `per_stream_rate_limit: 5MB/s` - `per_stream_rate_limit_burst: 30MB/s` This gives each stream a bucket that holds 30 MB. A 30 MB push drains the entire bucket, which then refills at 5 MB per second up to the 30 MB cap. You could raise the burst limit further, to 60 MB, to handle spikier streams, but avoid raising the sustained rate too far, since that keeps overall memory use predictable. Automatic stream sharding also continuously works to keep stream volumes near `desired_rate`, but it can't always react fast enough to very sudden changes. ### query_timeout - **Default**: 1 minute - **Recommended maximum**: 5 minutes (do not exceed) The query timeout controls how long a query can run before Loki terminates it. This limit exists because Loki is designed as a synchronous query engine, not for very long-running queries. **Why you should not exceed 5 minutes**: Loki was not engineered for very long-running query operations. The way Loki queries work, each querier runs a fixed number of worker routines. Query frontends and schedulers split a query by time range or by label set and enqueue each resulting subquery, and a single worker routine executes each one. All tenants share this pool of query workers. Raising the timeout for one tenant lets that tenant reserve query workers for longer, which reduces the query capacity available to every other tenant. In multi-tenant or large multi-user environments, this can make it difficult to maintain system stability. **Consequences of exceeding**: - Reduced query capacity for all tenants. - Increased query queue wait times. - Potential denial of service to other users. - Unpredictable query performance. **What to do instead**: Optimize your queries to complete faster. Use smaller time ranges, more specific labels, and filters applied as early as possible in the query. ## Understanding limit violations When you exceed a limit, Loki provides metrics and errors to help you understand what happened. ### Monitoring metrics - **`loki_discarded_samples_total`**: Counts log lines discarded, with a `reason` label indicating which limit was exceeded. - **`loki_discarded_bytes_total`**: Measures bytes discarded, by reason. - **`loki_ingester_memory_streams`**: Current number of active streams in memory, per tenant. ### Common error reasons - **`rate_limited`**: Exceeded `ingestion_rate_mb`. - **`per_stream_rate_limit`**: An individual stream exceeded its rate limit. - **`stream_limit`**: Exceeded `max_global_streams_per_user`. - **`line_too_long`**: A log line exceeded `max_line_size`. - **`max_label_names_per_series`**: Too many labels on a stream. - **`label_name_too_long`**: A label name exceeded `max_length_label_name`. - **`label_value_too_long`**: A label value exceeded `max_length_label_value`. - **`too_far_behind`**: A log entry fell outside the out-of-order ingestion window. - **`greater_than_max_sample_age`**: A log entry is older than `reject_old_samples_max_age`. ## Best practices for limits To work effectively within Loki's limits: 1. **Design for low cardinality**: Use 5 to 10 labels at most, with static values. 2. **Keep logs reasonably sized**: Target 1 to 10 KB per log line. 3. **Forward logs promptly**: Aim for sub-second delivery latency. 4. **Monitor your usage**: Set up alerts on discarded-sample metrics. 5. **Scale infrastructure first**: Increase resources before you increase limits. 6. **Use alternatives**: Use structured metadata and query-time parsing instead of more labels. 7. **Plan capacity**: Calculate your expected stream count before deployment. 8. **Test at scale**: Validate your label design with production-like data volumes. ## Next steps To change any of these limits for your own deployment, see the `limits_config` block in the [Configuration reference](https://grafana.com/docs/loki//configure/), and review the related topics below before you raise a limit. ## Related documentation - [Configure Loki](https://grafana.com/docs/loki//configure/bp-configure/) - Practical guide to configuring Loki - [Rate limits and validation](https://grafana.com/docs/loki//operations/request-validation-rate-limits/) - Detailed error handling - [Understanding cardinality](https://grafana.com/docs/loki//get-started/labels/cardinality/) - How cardinality affects performance - [Labels best practices](https://grafana.com/docs/loki//get-started/labels/bp-labels/) - Guide to label design - [Structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) - Alternative to high-cardinality labels # Storage Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki. Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. This type only requires one store, the object store, for both the index and chunks. More detailed information about TSDB can be found under the [manage section](https://grafana.com/docs/loki//operations/storage/tsdb/). ## Single Store TSDB Single Store refers to using object storage as the storage medium for both the Loki index as well as its data ("chunks"). There is one supported mode: Starting in Loki 2.8, the [TSDB index store](https://grafana.com/docs/loki//operations/storage/tsdb/) improves query performance, reduces TCO and has the same feature parity as the deprecated "boltdb-shipper". TSDB is the recommended index store for Loki 2.8 and newer. ### Supported storage backends See [Object Storage](#object-storage) for supported backends. ## Chunk storage ### File system The file system is the simplest backend for chunks, although it's also susceptible to data loss as it's unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. It is similar in concept to many Prometheus deployments where a single Prometheus is responsible for monitoring a fleet. ### Object storage #### Google Cloud Storage (GCS) GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you're already running on GCP, and is production safe. #### Amazon Simple Storage Storage (S3) S3 is AWS's hosted object store. It is a good candidate for a managed object store, especially when you're already running on AWS, and is production safe. #### Azure Blob Storage Blob Storage is Microsoft Azure's hosted object store. It is a good candidate for a managed object store, especially when you're already running on Azure, and is production safe. You can authenticate Blob Storage access by using a storage account name and key or by using a Service Principal. #### IBM Cloud Object Storage (COS) [COS](https://www.ibm.com/cloud/object-storage) is IBM Cloud hosted object store. It is a good candidate for a managed object store, especially when you're already running on IBM Cloud, and is production safe. #### Baidu Object Storage (BOS) [BOS](https://intl.cloud.baidu.com/product/bos.html) is the Baidu Cloud hosted object storage. #### Alibaba Object Storage Service (OSS) [OSS](https://www.alibabacloud.com/product/object-storage-service) is the Alibaba Cloud hosted object storage. #### Other notable mentions You may use any substitutable services, such as those that implement the S3 API like [MinIO](https://min.io/). ## Schema Config Loki aims to be backwards compatible and over the course of its development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage _schemas_ and can query across them transparently. This makes upgrading a breeze. For a more detailed reference on schema configuration, including required values and the recommended settings for new installs, refer to [Storage schema](https://grafana.com/docs/loki//operations/storage/schema/). For instance, this is what it looks like when migrating from BoltDB with v11 schema to TSDB with v13 schema starting 2023-07-01: ```yaml schema_config: configs: - from: 2019-07-01 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h - from: 2023-07-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ``` For all data ingested before 2023-07-01, Loki used BoltDB with the v11 schema, and then switched after that point to the more effective TSDB with the v13 schema. This dramatically simplifies upgrading, ensuring it's simple to take advantage of new storage optimizations. These configs should be immutable for as long as you care about retention. ## Upgrading Schemas When a new schema is released and you want to gain the advantages it provides, you can! Loki can transparently query and merge data from across schema boundaries so there is no disruption of service and upgrading is easy. First, you'll want to create a new [period_config](https://grafana.com/docs/loki//configure/#period_config) entry in your [schema_config](https://grafana.com/docs/loki//configure/#schema_config). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn't queried as if it adheres to the new schema. As an example, let's say it's 2023-07-14 and you want to start using the `v13` schema on the 20th: ```yaml schema_config: configs: - from: 2019-07-14 store: tsdb object_store: filesystem schema: v11 index: prefix: index_ period: 24h - from: 2023-07-20 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ``` It's that easy; you just created a new entry starting on the 20th. ## Retention Loki manages retention through the Compactor when using TSDB. When retention is enabled, the Compactor identifies data that falls outside of the configured retention period, removes the corresponding index entries, and deletes the underlying chunk objects asynchronously. For object storage backends (S3, GCS, Azure Blob) Loki no longer relies solely on external time to live (TTL) or bucket lifecycle rules; these may still be used as an additional safeguard, but Loki itself performs retention-driven deletion when configured. When using the filesystem chunk store, Loki does not delete data based on disk usage or free-space conditions. Deletion is determined only by the retention settings, and disk-full scenarios must be handled operationally outside of Loki. Loki also supports targeted deletion at the tenant or stream level. For more information, see the [retention configuration](https://grafana.com/docs/loki//operations/storage/retention/) documentation. ## Examples {{< admonition type="note" >}} Loki uses the Thanos-based object storage clients by default, because the `use_thanos_objstore` setting defaults to `true`. With this default, Loki reads the configuration under `storage_config.object_store` and ignores the legacy client sections such as `storage_config.aws` or `storage_config.gcs`. Each example below shows the Thanos configuration first. The legacy configuration follows it for reference, because the legacy clients are deprecated. To keep using a legacy client, you must set `use_thanos_objstore: false`. To convert an existing configuration to the new format, refer to [Migrate to Thanos storage clients](https://grafana.com/docs/loki//setup/migrate/migrate-storage-clients/). {{< /admonition >}} ### Single machine/local development (tsdb+filesystem) [The repo contains a working example](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml), you may want to checkout a tag of the repo to make sure you get a compatible example. ### GCP deployment (GCS Single Store) Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#gcs-example): ```yaml storage_config: use_thanos_objstore: true object_store: gcs: bucket_name: service_account: | { "type": "service_account", ... } tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h ``` The same deployment using the deprecated GCS client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h gcs: bucket_name: service_account: | { "type": "service_account", ... } schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h ``` `service_account` should contain JSON from either a GCP Console `client_credentials.json` file or a GCP service account key. If this value is blank, most services will fall back to GCP's Application Default Credentials (ADC) strategy. For more information about ADC, refer to [How Application Default Credentials works](https://cloud.google.com/docs/authentication/application-default-credentials). The [pre-defined `storage.objectUser` role](https://cloud.google.com/storage/docs/access-control/iam-roles) (or a custom role modeled after it) contains sufficient permissions for Loki to operate. {{< admonition type="note" >}} GCP recommends [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) instead of a service account key. {{< /admonition >}} ### AWS deployment (S3 Single Store) Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#s3-example): ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # You can either declare the access key and secret in the config or # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, # which will be picked up by the AWS SDK. access_key_id: secret_access_key: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` The Thanos-based client supports one bucket only. If you previously used `bucketnames` with several buckets, you must consolidate to a single bucket. If you don't wish to hard-code S3 credentials, you can use an EC2 instance role instead. Leave `access_key_id` and `secret_access_key` unset. The client then looks for credentials in environment variables, in the AWS credentials file, and finally in the EC2 instance metadata: ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: endpoint: s3..amazonaws.com region: ``` The same deployment using the deprecated S3 client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h aws: s3: s3://:@ bucketnames: , schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` To use an EC2 instance role with the deprecated client, change the `storage_config` section: ```yaml storage_config: use_thanos_objstore: false aws: s3: s3:// bucketnames: , ``` The role should have a policy with the following permissions attached. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "LokiStorage", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::" ] }, "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` **To setup an S3 bucket and an IAM role and policy:** This guide assumes a provisioned EKS cluster. 1. Checkout the Loki repository and navigate to [production/terraform/modules/s3](https://github.com/grafana/loki/tree/main/production/terraform/modules/s3). 2. Initialize Terraform `terraform init`. 3. Export the AWS profile and region if not done so: ```bash export AWS_PROFILE= export AWS_REGION= ``` 4. Save the OIDC provider in an environment variable: ```bash oidc_provider=$(aws eks describe-cluster --name --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") ``` See the [IAM OIDC provider guide](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for a guide for creating a provider. 5. Apply the Terraform module `terraform -var region="$AWS_REGION" -var cluster_name= -var oidc_id="$oidc_provider"` Note, the bucket name defaults to `loki-data` but can be changed via the `bucket_name` variable. ### Azure deployment (Azure Blob Storage Single Store) #### Using account name and key Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#azure-example): ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: true object_store: azure: # Your Azure storage account name account_name: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: # For the account key, see https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal # If you leave the account key unset, Loki uses an Azure managed identity instead. account_key: # Set this to use a user assigned managed identity. If you leave it empty, # Loki uses the system assigned identity. user_assigned_id: # Configure this if you use a private Azure cloud, such as Azure Stack Hub. # Loki composes the storage URL as https://account_name.endpoint_suffix/container_name/blob_name endpoint_suffix: # If `connection_string` is set, the `account_name` and `endpoint_suffix` values are not used. # Use this instead of `account_key` to authenticate with a SAS token, or to use the Azurite emulator. connection_string: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h ``` The same deployment using the deprecated Azure Blob Storage client: ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: false # required to use the deprecated client azure: # Your Azure storage account name account_name: # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal account_key: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: use_managed_identity: # Providing a user assigned ID will override use_managed_identity user_assigned_id: request_timeout: 0 # Configure this if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name endpoint_suffix: # If `connection_string` is set, the values of `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via a SAS token. Or if you use the Azurite emulator. connection_string: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h filesystem: directory: /loki/chunks ``` #### Using a service principal The [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#azure-example) has no service principal settings in `storage_config`. Instead, pass the credentials in the standard environment variables read by the [Azure Identity Client Module for Go](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity), which are `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_CLIENT_SECRET`. Leave `account_key` unset so that Loki uses those credentials. You must still set `account_name`, because Loki uses it to build the storage URL. ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: true object_store: azure: account_name: container_name: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h ``` The same deployment using the deprecated Azure Blob Storage client: ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: false # required to use the deprecated client azure: use_service_principal: true # Azure tenant ID used to authenticate through Azure OAuth tenant_id : # Azure Service Principal ID client_id: # Azure Service Principal secret key client_secret: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: request_timeout: 0 tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h filesystem: directory: /loki/chunks ``` ### IBM Deployment (COS Single Store) {{< admonition type="note" >}} The Thanos-based object store client does not support IBM COS. Because `use_thanos_objstore` defaults to `true`, you must set it to `false` to use COS. If you leave the default in place, Loki fails to start with the error `unrecognized object_store type cos`. For the list of backends that the Thanos-based client supports, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config). {{< /admonition >}} ```yaml schema_config: configs: - from: 2020-10-01 index: period: 24h prefix: loki_index_ object_store: cos schema: v13 store: tsdb storage_config: use_thanos_objstore: false # COS is not supported by the Thanos-based client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cos: bucketnames: , endpoint: api_key: region: service_instance_id: auth_endpoint: ``` ### On premise deployment (MinIO Single Store) You configure MinIO by using the S3 settings, because MinIO implements the S3 API. Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#minio-s3-compatible-example): ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true # set to false if MinIO is served over https bucket_lookup_type: path # MinIO's equivalent of s3forcepathstyle tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` The same deployment using the deprecated S3 client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client aws: # Note: use a fully qualified domain name (fqdn), like localhost. # full example: http://loki:supersecret@localhost.:9000 s3: http(s)://:@: s3forcepathstyle: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` # Usage statistics Loki sends anonymous usage statistics to Grafana Labs. This reporting is enabled by default and helps the Loki team understand aggregate usage patterns. Reports are sent to `https://stats.grafana.org/loki-usage-report` by default. ## What Loki collects Loki reports aggregate operational metadata, including: - An anonymous cluster identifier. - Build and runtime metadata, such as version, OS, architecture, and Go version. - Deployment metadata, such as the configured Loki target. - Go runtime metrics, such as memory and goroutine counts. - Aggregate product metrics, such as ingestion, query throughput, and chunk flush statistics. - Storage and compactor configuration metadata, such as schema, storage backend type, and retention settings. Loki does not report log payloads, label values, tenant identifiers, or query strings. You can view the code in [stats.go](https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go) ## Disable usage statistics reporting To disable reporting in the Loki config file, set `analytics.reporting_enabled` to `false`: ```yaml analytics: reporting_enabled: false ``` To disable reporting with a CLI flag, start Loki with: ```sh -reporting.enabled=false ``` To disable reporting with the Helm chart, set the following values: ```yaml loki: analytics: reporting_enabled: false ``` For all analytics configuration options, refer to the [analytics configuration reference](../#analytics). # Legacy configuration examples These examples configure Loki's legacy, non-default storage clients. Since Loki 4.0, Loki uses the Thanos-based storage clients by default (the `use_thanos_objstore` setting defaults to `true`). For the current default storage client configuration, refer to [Configuration examples for using Thanos-based storage clients](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/). ## 1-Local-Configuration-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by the filesystem. # The index will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default filesystem: directory: /tmp/loki/chunks ``` ## 2-S3-Cluster-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a s3-compatible API # like MinIO for storage. # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache aws: s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name s3forcepathstyle: true ``` ## 3-S3-Without-Credentials-Snippet.yaml ```yaml # If you don't wish to hard-code S3 credentials you can also configure an EC2 # instance role by changing the `storage_config` section. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: s3: s3://region/bucket_name ``` ## 4-GCS-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a GCS. # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache gcs: bucket_name: replace_by_your_bucked_name ``` ## 5-BOS-Example.yaml ```yaml # This is a partial configuration to deploy Loki backed by Baidu Object Storage (BOS). # The index will be shipped to the storage via tsdb-shipper. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: bos schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default bos: bucket_name: bucket_name_1 endpoint: bj.bcebos.com access_key_id: access_key_id secret_access_key: secret_access_key ``` ## 6-Compactor-Snippet.yaml ```yaml # This partial configuration sets the compactor to use S3 and run the compaction every 5 minutes. # Downloaded index files for compaction are stored in /loki/compactor. compactor: working_directory: /tmp/loki/compactor compaction_interval: 5m ``` ## 7-Schema-Migration-Snippet.yaml ```yaml common: path_prefix: /tmp/loki schema_config: configs: # Starting from 2018-04-15 Loki should store indexes on BoltDB with the v11 schema # using daily periodic tables and chunks on filesystem. # The index tables will be prefixed with "index_". # NOTE: `store: boltdb-shipper` is deprecated and scheduled for removal in Loki 4.0. # This entry is kept only to illustrate migrating an existing boltdb-shipper period to tsdb. - from: "2018-04-15" store: boltdb-shipper object_store: filesystem schema: v11 index: period: 24h prefix: index_ # Starting from 2023-6-15 Loki should store indexes on TSDB with the v13 schema # using daily periodic tables and chunks on AWS S3. - from: "2023-06-15" store: tsdb object_store: s3 schema: v13 index: period: 24h prefix: index_ ``` ## 8-alibaba-cloud-storage-Snippet.yaml ```yaml # This partial configuration uses Alibaba for chunk storage. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: alibabacloud schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache alibabacloud: bucket: # Alibabacloud region name. # Required when signature_version=v4. # region: endpoint: access_key_id: secret_access_key: # Specify the RAM role name of the ECS instance. ECS RAM role authentication is used only # when neither access_key_id nor secret_access_key is configured and requires signature_version=v4. # If not set, the role name will be automatically retrieved from the ECS instance metadata. # ram_role_name: # The signature version to use for authenticating against OSS. Supported values are: v1, v4. # ECS RAM role authentication requires signature_version=v4. # signature_version: ``` ## 9-S3-With-SSE-KMS-Snippet.yaml ```yaml # This partial configuration uses S3 for chunk storage and a KMS CMK for encryption. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: s3: s3://access_key:secret_access_key@region/bucket_name sse: type: SSE-KMS kms_key_id: 1234abcd-12ab-34cd-56ef-1234567890ab ``` ## 10-Expanded-S3-Snippet.yaml ```yaml # S3 configuration supports an expanded configuration. # Either an `s3` endpoint URL can be used, or an expanded configuration can be used. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: bucketnames: bucket_name1, bucket_name2 endpoint: s3.endpoint.com region: s3_region access_key_id: s3_access_key_id secret_access_key: s3_secret_access_key insecure: false http_config: idle_conn_timeout: 90s response_header_timeout: 0s insecure_skip_verify: false s3forcepathstyle: true ``` ## 11-COS-HMAC-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. HMAC will be used for authenticating with COS. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: access_key_id: secret_access_key: ``` ## 12-COS-APIKey-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. APIKey will be used for authenticating with COS. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: api_key: service_instance_id: auth_endpoint: ``` ## 13-COS-Trusted-Profile-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. # A trusted profile will be used for authenticating with COS. We can either pass # the trusted profile name or trusted profile ID along with the compute resource token file. # If we pass both trusted profile name and trusted profile ID it should be of # the same trusted profile. # In order to use trusted profile authentication we need to follow an additional step to create a trusted profile. # For more details about creating a trusted profile, see https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile&interface=ui. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: auth_endpoint: cr_token_file_path: trusted_profile_name: # You can also use trusted_profile_id instead of trusted_profile_name ``` ## 15-Memberlist-Ring-Snippet.yaml ```yaml # This partial configuration uses memberlist for the ring. common: ring: kvstore: store: memberlist replication_factor: 1 path_prefix: /loki memberlist: join_members: # You can use a headless k8s service for all distributor, ingester and querier components. - loki-gossip-ring.loki.svc.cluster.local:7946 # :7946 is the default memberlist port. ``` ## 16-Azure-Account-Name-Example.yaml ```yaml # This partial configuration uses Azure for chunk storage storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default azure: # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal account_key: account_name: container_name: use_managed_identity: false # Providing a user assigned ID will override use_managed_identity #user_assigned_id: request_timeout: 0 # Configure `endpoint_suffix` if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name #endpoint_suffix: # If `connection_string` is set, the `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via an SAS token. Or if you use the Azurite emulator. #connection_string: ``` ## 17-Azure-Service-Principal-Example.yaml ```yaml # This partial configuration uses Azure for chunk storage and a service principal for authentication storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default azure: use_service_principal: true # Azure tenant ID used to authenticate through Azure OAuth tenant_id: # Azure Service Principal ID client_id: # Azure Service Principal secret key client_secret: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: request_timeout: 0 ``` ## 18-Thanos-GCS-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by GCS, using the # Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: gcs: bucket_name: ``` ## 19-Thanos-S3-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by AWS S3, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # Leave the access key and secret unset to use an EC2 instance role, or # the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. access_key_id: secret_access_key: ``` ## 20-Thanos-Azure-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by Azure Blob Storage, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: azure schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: azure: account_name: account_key: container_name: ``` ## 21-Thanos-MinIO-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a MinIO cluster, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true bucket_lookup_type: path ``` # Query frontend example ## Before you begin This is a general-purpose example. It contains placeholder variables in the format ``. Replace them with values for your environment. ## Use case Many users start by running Grafana Loki as a single binary to simplify deployment. As usage grows and the deployment becomes more production-ready, performance bottlenecks can appear on the read path. Queries that ran quickly on small data sets may slow down or fail on larger ones. Adding more queriers can help in some cases. However, when a single query is too large for one querier to handle, you need the query frontend. ### Parallelization One of the most important functions of the query frontend is the ability to split larger queries into smaller ones, execute them in parallel, and combine the results. The `querier.split-queries-by-interval` flag (or the YAML config `limits_config.split_queries_by_interval`) controls how the frontend splits queries. For example, with this value set to `1h`, the frontend splits a 24-hour query into 24 one-hour queries, distributes them to the queriers, and collects the results. This approach has two benefits in production environments: it lets you run larger queries through aggregation, and it distributes work evenly across queriers. ## Kubernetes Deployment ### ConfigMap Use this ConfigMap to enable query parallelization and caching with the query-frontend component. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: loki-frontend namespace: data: config.yaml: | # Disable the requirement that every request to Loki has a # X-Scope-OrgID header. `fake` will be substituted in instead. auth_enabled: false server: http_listen_port: 3100 query_range: # make queries more cache-able by aligning them with their step intervals align_queries_with_step: true max_retries: 5 cache_results: true results_cache: cache: # Use the embedded cache embedded_cache: enabled: true max_size_mb: 100 ttl: 24h limits_config: max_cache_freshness_per_query: '10m' # parallelize queries in 15min intervals split_queries_by_interval: 15m frontend: log_queries_longer_than: 5s downstream_url: http://querier..svc.cluster.local:3100 compress_responses: true ``` ### Frontend Service ```yaml apiVersion: v1 kind: Service metadata: labels: name: query-frontend name: query-frontend namespace: spec: ports: - name: query-frontend-http port: 3100 protocol: TCP targetPort: 3100 selector: name: query-frontend sessionAffinity: None clusterIP: None publishNotReadyAddresses: true ``` ### Frontend Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: name: query-frontend name: query-frontend namespace: spec: minReadySeconds: 10 replicas: 2 selector: matchLabels: name: query-frontend template: metadata: labels: name: query-frontend spec: containers: - args: - -config.file=/etc/loki/config.yaml - -log.level=debug - -target=query-frontend image: grafana/loki: imagePullPolicy: Always name: query-frontend ports: - containerPort: 3100 name: http protocol: TCP resources: limits: memory: 1200Mi requests: cpu: '2' memory: 600Mi volumeMounts: - mountPath: /etc/loki name: loki-frontend restartPolicy: Always terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: loki-frontend name: loki-frontend ``` ### Grafana After you deploy these resources, point your Grafana data source to the new frontend service. The service is available within the cluster at `http://query-frontend..svc.cluster.local:3100`. ### gRPC mode (pull model) The query frontend operates in one of two modes: - **Push mode:** Specify `--frontend.downstream-url` or its YAML equivalent `frontend.downstream_url`. The frontend proxies requests over HTTP to the specified URL. - **Pull mode:** If you do not set `--frontend.downstream-url` or its YAML equivalent `frontend.downstream_url`, the query frontend operates as a pull service. The frontend creates per-tenant queues. Downstream queriers pull queries from these queues using gRPC. To use pull mode, configure queriers with `-querier.frontend-address` or its YAML equivalent `frontend_worker.frontend_address`. Set `clusterIP: None` on the query frontend pull service. This setting causes DNS to resolve to individual query frontend pod IP addresses instead of the service IP. Set `publishNotReadyAddresses: true` on the query frontend pull service. This setting prevents a race condition where a querier needs the query frontend address before the frontend is ready. # Configuration examples for using Thanos-based storage clients Use these examples as a starting point for configuring [Thanos-based object storage clients](https://grafana.com/docs/loki//configure/#thanos_object_store_config) in Grafana Loki. Each example is a complete configuration you can adapt and run, not just a snippet. ## GCS example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space object_store: gcs: bucket_name: # JSON either from a Google Developers Console client_credentials.json file, # or a Google Developers service account key. Needs to be valid JSON, not a # filesystem path. If empty, fallback to Google default logic: # 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS # environment variable. For workload identity federation, refer to # https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation # on how to generate the JSON configuration file for on-prem/non-Google cloud # platforms. # 2. A JSON file in a location known to the gcloud command-line tool: # $HOME/.config/gcloud/application_default_credentials.json. # 3. On Google Compute Engine it fetches credentials from the metadata server. service_account: |- { "type": "service_account", "project_id": "project", "private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666", "private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n", "client_email": "project@example.iam.gserviceaccount.com", "client_id": "123456789012345678901", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/project@example.iam.gserviceaccount.com" } ``` ## S3 example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # You can either declare the access key and secret in the config or # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by the AWS SDK. access_key_id: secret_access_key: ``` ## Azure example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: azure schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: azure: account_name: account_key: ${SECRET_ACCESS_KEY} # loki expands environment variables container_name: ``` ## MinIO (S3-compatible) example MinIO and other S3-compatible object stores use the same `object_store.s3` client as AWS S3. Set `bucket_lookup_type: path`, which is the equivalent of the deprecated `s3forcepathstyle` option. Set `insecure: true` if MinIO is served over plain HTTP. ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true bucket_lookup_type: path ``` ## Filesystem example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /tmp/loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true object_store: filesystem: dir: /tmp/loki/chunks ```