# Grafana Loki 文档 Grafana Loki 日志系统的说明。 # Community # Contributing to Loki For the full contribution guide, see [CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) in the root of the repository. ## Building Clone the repository and use `make` to build: ```bash git clone https://github.com/grafana/loki.git cd loki ``` | Command | Output | |---|---| | `make loki` | `./cmd/loki/loki` | | `make logcli` | `./cmd/logcli/logcli` | | `make loki-canary` | `./cmd/loki-canary/loki-canary` | | `make all` | all of the above | | `make loki-image` | Docker image for Loki | ## Running tests ```bash make test # unit tests make test-integration # integration tests (requires Docker, ~15 min) make lint # run linters ``` ## Contribute to the Helm Chart Refer to [production/helm/loki/CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/production/helm/loki/CONTRIBUTING.md) for chart-specific guidelines. The official Loki Helm chart is maintained in the [grafana/helm-charts](https://github.com/grafana/helm-charts) repository. # Loki community frequently asked questions ### Is Loki open source? What is the license? Grafana Loki is open source, licensed under the GNU Affero General Public License v3 (AGPLv3). The license change from Apache 2.0 to AGPLv3 took effect in 2021. The practical implication of AGPLv3 is that if you modify Loki and make it available over a network, you must make your modifications available under the same license. For most teams running Loki internally for log aggregation, this has no effect. If you are embedding Loki in a commercial product or offering it as a service, review the [license terms](https://github.com/grafana/loki/blob/main/LICENSE) or contact Grafana Labs about a commercial license. Other components in the Grafana observability stack (Grafana, Tempo, Mimir, Alloy) each have their own license terms, detailed in their respective repositories. ### How is the documentation organized, and where do I find docs for a specific product? The Loki documentation lives at [grafana.com/docs/loki/](https://grafana.com/docs/loki/latest/) and is organized into the following top-level sections: - **[Get started](https://grafana.com/docs/loki/latest/get-started/)** — Architecture overview, deployment modes, and label concepts. - **[Set up](https://grafana.com/docs/loki/latest/setup/)** — Installation instructions (Helm, Docker, local binary), migration guides, and upgrade steps. - **[Configure](https://grafana.com/docs/loki/latest/configure/)** — Configuration reference and example configurations for common object storage backends. - **[Send data](https://grafana.com/docs/loki/latest/send-data/)** — Guides for shipping logs to Loki using Grafana Alloy, Promtail, the Docker driver, and the HTTP API. - **[Query](https://grafana.com/docs/loki/latest/query/)** — LogQL language reference, query examples, and the query API. - **[Manage](https://grafana.com/docs/loki/latest/operations/)** — Day-two operations including storage, retention, caching, and multi-tenancy. - **[Alert](https://grafana.com/docs/loki/latest/alert/)** — Recording rules and alerting rules powered by LogQL. Documentation for other Grafana Labs products (Grafana, Mimir, Tempo, Alloy, and others) is available from the [Grafana documentation home page](https://grafana.com/docs/). ### Am I reading the right version of the docs? The Loki documentation is versioned to match each release. By default, links from grafana.com and search results take you to the latest version, but if you arrived via an older link — for example, from a blog post, a forum thread, or a search result — you may be reading documentation for an older release. To check which version you are reading, look at the version selector in the left-hand navigation panel. If it does not say **latest**, you can switch to the current version from that selector, or navigate directly using `/latest/` in the URL — for example, `grafana.com/docs/loki/latest/`. If the version selector shows **next**, you are reading unpublished documentation for the upcoming release. This content may describe features or configuration options that are not yet available in any released version of Loki. The **next** docs are useful for previewing what is coming, but should not be used as a reference for your current deployment. If a configuration option or feature described in the docs does not appear to exist in your Loki deployment, a version mismatch is the most common explanation. Check your installed Loki version (run `loki -version` or check the container image tag) and compare it against the version shown in the docs before assuming the documentation is wrong. ### How can I contribute to or report a problem with the docs? The Loki documentation is open source and lives in the [`docs/sources`](https://github.com/grafana/loki/tree/main/docs/sources) directory of the Loki GitHub repository. Community contributions are welcome. At the bottom of every documentation page you will find three options: - **Suggest an edit** — opens the page source directly in GitHub so you can submit a pull request for a correction or improvement. - **Contribute to docs** — for reporting a minor issue or mistake without making the edit yourself. - **Report a problem** — for reporting technical issues with how the documentation page renders or behaves. For questions that the documentation has not answered, the [Grafana Community Forum](https://community.grafana.com/) is the best place to ask. You can also join the `#loki` channel in the [Grafana Labs Community Slack](https://slack.grafana.com/) to discuss Loki-specific topics with maintainers and other users. ### How can I contribute to or report a problem with the code? Loki is developed in the open on GitHub at [grafana/loki](https://github.com/grafana/loki). There are several ways to contribute: - **Report a bug.** Open an [issue](https://github.com/grafana/loki/issues/new/choose) on GitHub. Include your Loki version, deployment mode, relevant configuration, and steps to reproduce the problem. - **Suggest a feature.** Feature requests are also tracked as GitHub issues. Describe the use case and the behavior you would like to see. - **Submit a pull request.** The full contribution guide is in [CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md). It covers building from source, running tests, code style, and the review process. The [Contributing to Loki](https://grafana.com/docs/loki/latest/community/contributing/) docs page provides a quick-start summary. {{< admonition type="note" >}} First-time contributors should look for issues labeled [`good first issue`](https://github.com/grafana/loki/labels/good%20first%20issue) for approachable starting points. {{< /admonition >}} # Contacting the Loki Team For questions regarding Loki: - Ask a question in the `#loki` channel of our [Slack](https://slack.grafana.com/) community - Open source Loki users are welcome to post technical questions on the Grafana Labs Community Forums under the Grafana Loki category at [community.grafana.com](https://community.grafana.com). Please be mindful that this is a community-driven support channel moderated by Grafana Labs staff where Loki maintainers and community members answer questions when bandwidth allows. Be sure to review the [Community Guidelines](https://community.grafana.com/guidelines) before posting. - Users using Loki via [Grafana Cloud](/products/cloud/) can submit support tickets via the [Grafana.com Account Portal](/login). - For questions regarding Enterprise support for Loki, you can get in touch with the Grafana Labs team [here](/contact?pg=docs). Your feedback is always welcome! To submit feedback or a report a potential bug: - [File a GitHub issue](https://github.com/grafana/loki/issues/new) for bugs, issues and feature suggestions. Grafana UI issues for the Loki data source plugin should be posted directly to the [Grafana repository](https://github.com/grafana/grafana/issues/new), not the Loki repository. - Send an email to [lokiproject@googlegroups.com](mailto:lokiproject@googlegroups.com), or visit the [google groups](https://groups.google.com/forum/#!forum/lokiproject) page. - Join us for our monthly [Loki Community Call](https://docs.google.com/document/d/1MNjiHQxwFukm2J4NJRWyRgRIiK7VpokYyATzJ5ce-O8/edit?usp=sharing) on first Thursday of every month alternating between EU (12:00 UTC) and US (17:00 UTC) timezones. Subscribe to the calendar in the doc for being up to date. # Governance This document describes the rules and governance of the project. It is meant to be followed by all the developers of the project and the Loki community. Common terminology used in this governance document are listed below: - **Team members**: Any members of the private [team mailing list][team]. - **Maintainers**: Maintainers lead an individual project or parts thereof ([`MAINTAINERS.md`][maintainers]). - **Projects**: A single repository in the Grafana GitHub organization and listed below is referred to as a project: - loki - puppet-promtail - **The Loki project**: The sum of all activities performed under this governance, concerning one or more repositories or the community. - **The Loki SIG (Special Interest Group) Operator**: The sum of all activities performed under this governance, concerning the repository subfolder `operator` or its specific community. ## Values The Loki developers and community are expected to follow the values defined in the [Code of Conduct][coc]. Furthermore, the Loki community strives for kindness, giving feedback effectively, and building a welcoming environment. The Loki developers generally decide by consensus and only resort to conflict resolution by a majority vote if consensus cannot be reached. ## Projects Each project must have a [`MAINTAINERS.md`][maintainers] file with at least one maintainer. Where a project has a release process, access and documentation should be such that more than one person can perform a release. Releases should be announced on the [announcement][announce] and [users][users] mailing lists. Any new projects should be first proposed on the [team mailing list][team] following the voting procedures listed below. ## Decision making ### Team members Team member status may be given to those who have made ongoing contributions to the Loki project for at least 3 months. This is usually in the form of code improvements and/or notable work on documentation, but organizing events or user support could also be taken into account. New members may be proposed by any existing member by email to the [team mailing list][team]. It is highly desirable to reach consensus about acceptance of a new member. However, the proposal is ultimately voted on by a formal [supermajority vote](#supermajority-vote). If the new member proposal is accepted, the proposed team member should be contacted privately via email to confirm or deny their acceptance of team membership. This email will also be CC'd to the [team mailing list][team] for record-keeping purposes. If they choose to accept, the [onboarding](#onboarding) procedure is followed. Team members may retire at any time by emailing [the team][team]. Team members can be removed by [supermajority vote](#supermajority-vote) on [the team mailing list][team]. For this vote, the member in question is not eligible to vote and does not count towards the quorum. Any removal vote can cover only one single person. Upon death of a member, they leave the team automatically. In case a member leaves, the [offboarding](#offboarding) procedure is applied. The current team members are: - Aditya C S - [adityacs](https://github.com/adityacs) - Ashwanth Goli - [ashwanthgoli](https://github.com/ashwanthgoli) ([Grafana Labs](/)) - Cyril Tovena - [cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](/)) - Danny Kopping - [dannykopping](https://github.com/dannykopping) ([Grafana Labs](/)) - David Kaltschmidt - [davkal](https://github.com/davkal) ([Grafana Labs](/)) - Dylan Guedes - [dylanguedes](https://github.com/dylanguedes) ([Grafana Labs](/)) - Edward Welch - [slim-bean](https://github.com/slim-bean) ([Grafana Labs](/)) - Goutham Veeramachaneni - [gouthamve](https://github.com/gouthamve) ([Grafana Labs](/)) - Joe Elliott - [joe-elliott](https://github.com/joe-elliott) ([Grafana Labs](/)) - Karsten Jeschkies - [jeschkies](https://github.com/jeschkies) ([Grafana Labs](/)) - Kaviraj Kanagaraj - [kavirajk](https://github.com/kavirajk) ([Grafana Labs](/)) - Li Guozhong - [liguozhong](https://github.com/liguozhong) ([Alibaba Cloud](https://alibabacloud.com/)) - Michel Hollands - [michelhollands](https://github.com/michelhollands) ([Grafana Labs](/)) - Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) - Periklis Tsirakidis - [periklis](https://github.com/periklis) ([Red Hat](https://www.redhat.com/)) - Salva Corts - [salvacorts](https://github.com/salvacorts) ([Grafana Labs](/)) - Sandeep Sukhani - [sandeepsukhani](https://github.com/sandeepsukhani) ([Grafana Labs](/)) - Susana Ferreira - [ssncferreira](https://github.com/ssncferreira) - Tom Braack - [sh0rez](https://github.com/sh0rez) ([Grafana Labs](/)) - Tom Wilkie - [tomwilkie](https://github.com/tomwilkie) ([Grafana Labs](/)) The current Loki SIG Operator team members are: - Brett Jones - [blockloop](https://github.com/blockloop/) ([InVision](https://www.invisionapp.com/)) - Cyril Tovena - [cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](/)) - Gerard Vanloo - [Red-GV](https://github.com/Red-GV) ([IBM](https://www.ibm.com)) - Periklis Tsirakidis - [periklis](https://github.com/periklis) ([Red Hat](https://www.redhat.com)) - Sashank Agrawal - [sasagarw](https://github.com/sasagarw/) ([Red Hat](https://www.redhat.com)) ### Maintainers Maintainers lead one or more project(s) or parts thereof and serve as a point of conflict resolution amongst the contributors to this project. Ideally, maintainers are also team members, but exceptions are possible for suitable maintainers that, for whatever reason, are not yet team members. Changes in maintainership have to be announced on the [developers mailing list][devs]. They are decided by [rough consensus](#consensus) and formalized by changing the [`MAINTAINERS.md`][maintainers] file of the respective repository. Maintainers are granted commit rights to all projects covered by this governance. A maintainer or committer may resign by notifying the [team mailing list][team]. A maintainer with no project activity for a year is considered to have resigned. Maintainers that wish to resign are encouraged to propose another team member to take over the project. A project may have multiple maintainers, as long as the responsibilities are clearly agreed upon between them. This includes coordinating who handles which issues and pull requests. ### Technical decisions Technical decisions that only affect a single project are made informally by the maintainer of this project, and [rough consensus](#consensus) is assumed. Technical decisions that span multiple parts of the project should be discussed and made on the [developer mailing list][devs]. Decisions are usually made by [rough consensus](#consensus). If no consensus can be reached, the matter may be resolved by [majority vote](#majority-vote). ### Governance changes Changes to this document are made by Grafana Labs. ### Other matters Any matter that needs a decision may be called to a vote by any member if they deem it necessary. For private or personnel matters, discussion and voting takes place on the [team mailing list][team], otherwise on the [developer mailing list][devs]. ## Voting The Loki project usually runs by informal consensus, however sometimes a formal decision must be made. Depending on the subject matter, as laid out [above](#decision-making), different methods of voting are used. For all votes, voting must be open for at least one week. The end date should be clearly stated in the call to vote. A vote may be called and closed early if enough votes have come in one way so that further votes cannot change the final decision. In all cases, all and only [team members](#team-members) are eligible to vote, with the sole exception of the forced removal of a team member, in which said member is not eligible to vote. Discussion and votes on personnel matters (including but not limited to team membership and maintainership) are held in private on the [team mailing list][team]. All other discussion and votes are held in public on the [developer mailing list][devs]. For public discussions, anyone interested is encouraged to participate. Formal power to object or vote is limited to [team members](#team-members). ### Consensus The default decision making mechanism for the Loki project is [rough][rough] consensus. This means that any decision on technical issues is considered supported by the [team][team] as long as nobody objects or the objection has been considered but not necessarily accommodated. Silence on any consensus decision is implicit agreement and equivalent to explicit agreement. Explicit agreement may be stated at will. Decisions may, but do not need to be called out and put up for decision on the [developers mailing list][devs] at any time and by anyone. Consensus decisions can never override or go against the spirit of an earlier explicit vote. If any [team member](#team-members) raises objections, the team members work together towards a solution that all involved can accept. This solution is again subject to rough consensus. In case no consensus can be found, but a decision one way or the other must be made, any [team member](#team-members) may call a formal [majority vote](#majority-vote). ### Majority vote Majority votes must be called explicitly in a separate thread on the appropriate mailing list. The subject must be prefixed with `[VOTE]`. In the body, the call to vote must state the proposal being voted on. It should reference any discussion leading up to this point. Votes may take the form of a single proposal, with the option to vote yes or no, or the form of multiple alternatives. A vote on a single proposal is considered successful if more vote in favor than against. If there are multiple alternatives, members may vote for one or more alternatives, or vote “no” to object to all alternatives. It is not possible to cast an “abstain” vote. A vote on multiple alternatives is considered decided in favor of one alternative if it has received the most votes in favor, and a vote from more than half of those voting. Should no alternative reach this quorum, another vote on a reduced number of options may be called separately. ### Supermajority vote Supermajority votes must be called explicitly in a separate thread on the appropriate mailing list. The subject must be prefixed with `[VOTE]`. In the body, the call to vote must state the proposal being voted on. It should reference any discussion leading up to this point. Votes may take the form of a single proposal, with the option to vote yes or no, or the form of multiple alternatives. A vote on a single proposal is considered successful if at least two thirds of those eligible to vote vote in favor. If there are multiple alternatives, members may vote for one or more alternatives, or vote “no” to object to all alternatives. A vote on multiple alternatives is considered decided in favor of one alternative if it has received the most votes in favor, and a vote from at least two thirds of those eligible to vote. Should no alternative reach this quorum, another vote on a reduced number of options may be called separately. ## On- / Offboarding ### Onboarding The new member is - added to the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. - announced on the [developers mailing list][devs] by an existing team member. Ideally, the new member replies in this thread, acknowledging team membership. - added to the projects with commit rights. - added to the [team mailing list][team]. ### Offboarding The ex-member is - removed from the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. In case of forced removal, no approval is needed. - removed from the projects. Optionally, they can retain maintainership of one or more repositories if the [team](#team-members) agrees. - removed from the team mailing list and demoted to a normal member of the other mailing lists. - not allowed to call themselves an active team member any more, nor allowed to imply this to be the case. - added to a list of previous members if they so choose. If needed, we reserve the right to publicly announce removal. [announce]: https://groups.google.com/forum/#!forum/loki-announce [coc]: https://github.com/grafana/loki/blob/main/CODE_OF_CONDUCT.md [devs]: https://groups.google.com/forum/#!forum/loki-developers [maintainers]: https://github.com/grafana/loki/blob/main/MAINTAINERS.md [rough]: https://tools.ietf.org/html/rfc7282 [team]: https://groups.google.com/forum/#!forum/loki-team [users]: https://groups.google.com/forum/#!forum/loki-users # Promtail Push API - Author: Robert Fratto (@rfratto) - Date: Feb 4 2020 - Status: DRAFT Despite being an optional piece of software, Promtail provides half the power of Loki's story: log transformations, service discovery, metrics from logs, and context switching between your existing metrics and logs. Today, Promtail can only be operated to consume logs from very specific sources: files, journal, or syslog. If users wanted to write custom tooling to ship logs, the tooling has to bypass Promtail and push directly to Loki. This can lead users to reimplement functionality Promtail already provides, including its error retries and batching code. This document proposes a Push API for Promtail. The preferred implementation is by copying the existing Loki Push API and implementing it for Promtail. By being compatible with the Loki Push API, the Promtail Push API can allow batches of logs to be processed at once for optimize performance. Matching the Promtail API also allows users to transparently switch their push URLs from their existing tooling. Finally, a series of alternative solutions are detailed. ## Configuration Promtail will have a new target called HTTPTarget, configurable in the `scrape_config` array with the following schema: ```yaml # Defines an HTTP target, which exposes an endpoint against the Promtail # HTTP server to accept log traffic. http: # Defines the base URL for the push path, adding a prefix to the # exposed endpoint. The final endpoint path is # loki/api/v1/push. If omitted, defaults to /. # # Multiple http targets with the same base_url must not exist. base_url: / # Map of labels to add to every log line passed through to the target. labels: {} ``` ### Considerations Users will be able to define multiple `http` scrape configs, but the base URL value must be different for each instance. This allows to cleanly separate pipelines through different push endpoints. Users must also be aware about problems with running Promtail with an HTTP target behind a load balancer: if payloads are load balanced between multiple Promtail instances, ordering of logs in Loki will be disrupted leading to rejected pushes. Users are recommended to do one of the following: 1. Have a dedicated Promtail instance for receiving pushes. This also applies to using the syslog target. 1. Have a separate Kubernetes service that always resolves to the same Promtail pod, bypassing the load balancing issue. ## Implementation As discussed in this document, this feature will be implemented by copying the existing [Loki Push API](/docs/loki//api/#post-lokiapiv1push) and exposing it via Promtail. ## Considered Alternatives Using the existing API was chosen for its simplicity and capabilities of being used for interesting configurations (e.g., chaining Promtails together). These other options were considered but rejected as not the best solution for the problem being solved. Note that Option 3 has value and may be implemented separately from this feature. ### Option 1: JSON / Protobuf Payload A new JSON and Protobuf payload format can be designed instead of the existing Loki push payload. Both formats would have to be exposed to support clients that either can't or won't use protobuf marshalling. The primary benefit of this approach is to allow us to tweak the payload schema independently of Loki's existing schema, but otherwise may not be very useful and is essentially just code duplication. ### Option 2: gRPC Service The [logproto.Pusher](https://github.com/grafana/loki/blob/f7ee1c753c76ef63338d53cfba782188a165144d/pkg/logproto/logproto.proto#L8-L10) service could be exposed through Promtail. This would enable clients stubs to be generated for languages that have gRPC support, and, for HTTP1 support, a [gRPC gateway](https://github.com/grpc-ecosystem/grpc-gateway) would be embedded in Promtail itself. This implementation option is similar to the original proposed solution, but uses the gRPC gateway to handle HTTP/1 traffic instead of the HTTP1 shim that Loki uses. There are some concerns with this approach: 1. The gRPC Gateway reverse proxy will need to play nice with the existing HTTP mux used in Promtail. 1. We couldn't control the HTTP and Protobuf formats separately as Loki can. 1. Log lines will be double-encoded thanks to the reverse proxy. 1. A small overhead of using a reverse proxy in-process will be introduced. 1. This breaks our normal pattern of writing our own shim functions; may add some cognitive overhead of having to deal with the gRPC gateway as an outlier in the code. ### Option 3: Plaintext Payload Prometheus' [Push Gateway API](https://github.com/prometheus/pushgateway#command-line) is cleverly designed and we should consider implementing our API in the same format: users would push to `http://promtail-url/push/label1/value1?timestamp=now` with a plaintext POST body. For example: ``` curl -X POST http://promtail.default/push/foo/bar/fizz/buzz -d “hello, world!” ``` This approach may be slightly faster when compared to a non-plaintext payload as no unmarshaling needs to be performed. This URL path and timestamp still needs to be parsed, but this will generally be faster than the reflection requirements imposed by JSON. However, note that this API limits Promtail to accepting one line at a time and may cause performance issues when trying to handle large volumes of traffic. As an alternative, this API could also be implemented by external tooling and be built on top of any of the other implementation options. An [example implementation](https://github.com/grafana/loki/pull/1270) was created and has received positive support for its simplicity and ease of integration. # Write-Ahead Logs Author: Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) Date: 30/09/2020 ## Impetus Loki already takes numerous steps to ensure the persistence of log data, most notably the use of a configurable replication factor (redundancy) in the ingesters. However, this still leaves much to be desired in persistence guarantees, especially for single binary deployments. This proposal outlines a write ahead log (WAL) in order to complement existing measures by allowing storage/replay of incoming writes via local disk on the ingester components. ## Strategy We suggest a two pass WAL implementation which includes an initial recording of accepted writes (`segments`) and a subsequent checkpointing (`checkpoints`) which coalesces the first pass into more efficient representations to speed up replaying. ### Segments Segments are the first pass and most basic WAL. They store individual records of incoming writes that have been accepted and can be used to reconstruct the in memory state of an ingester without any external input. Each segment is some multiple of 32kB and upon filling one segment, a new segment is created. Initially Loki will try 256kb segment sizes, readjusting as necessary. They are sequentially named on disk and are automatically created when a target size is hit as follows: ``` data └── wal ├── 000000 ├── 000001 └── 000002 ``` ### Truncation In order to prevent unbounded growth and remove operations which have been flushed to storage from the WAL, it is regularly truncated and all but the last segment (which is currently active) are deleted at a configurable interval (`ingester.checkpoint-duration`). This is where checkpoints come into the picture. ### Checkpoints Before truncating the WAL, we advance the WAL segments by one in order to ensure we don't delete the currently writing segment. The directory will look like: ``` data └── wal ├── 000000 ├── 000001 ├── 000002 <- likely not full, no matter └── 000003 <- newly written, empty ``` Each in memory stream is iterated across an interval, calculated by `checkpoint_duration / in_memory_streams` and written to the checkpoint. After the checkpoint completes, it is moved from its temp directory to the `ingester.wal-dir`, taking the name of the last segment before it started (`checkpoint.000002`) and then all applicable segments (`00000`, `00001`, `00002`) and any previous checkpoint are deleted. Afterwards, it will look like: ``` data └── wal ├── checkpoint.000002 <- completed checkpoint └── 000003 <- currently active wal segment ``` #### Queueing Checkpoint operations It’s possible that one checkpoint operation will start at the same time another is running. In this case, the existing checkpoint operation should disregard its internal ticker and flush its series as fast as possible. Afterwards, the next checkpoint operation can begin. This will likely create a localized spike in IOPS before the amortization of the following checkpoint operation takes over and is another important reason to run the WAL on an isolated disk in order to mitigate noisy neighbor problems. After we've written/moved the current checkpoint, we reap the old one. ### WAL Record Types #### Streams A `Stream` record type is written when an ingester receives a push for a series it doesn't yet have in memory. At a high level, this will contain ```golang type SeriesRecord struct { UserID string Labels labels.Labels Fingerprint uint64 // label fingerprint } ``` #### Logs A `Logs` record type is written when an ingester receives a push, containing the fingerprint of the series it refers to and a list of `(timestamp, log_line)` tuples, _after_ a `Stream` record type is written, if applicable. ```golang type LogsRecord struct { UserID string Fingperprint uint64 // label fingerprint for the series these logs refer to Entries []logproto.Entry } ``` ### Restoration Replaying a WAL is done by loading any available checkpoints into memory and then replaying any operations from successively named segments on top (`checkpoint.000003` -> `000004` -> `000005`, etc). It's likely some of these operations will fail because they're already included in the checkpoint (due to delay introduced in our amortizations), but this is ok -- we won't _lose_ any data, only try to write some data twice, which will be ignored. ### Deployment Introduction of the WAL requires that ingesters have persistent disks which are reconnected across restarts (this is a good fit for StatefulSets in Kubernetes). Additionally, it's recommended that the WAL uses an independent disk such that it's isolated from being affected by or causing noisy neighbor problems, especially during any IOPS spike(s). ### Implementation goals - Use underlying prometheus wal pkg when possible for consistency and to mitigate undifferentiated heavy lifting. Interfaces handle page alignment and use []byte. - Ensure this package handles arbitrarily long records (log lines in Loki’s case). - Ensure our in memory representations can be efficiently moved to/from `[]byte` in order to generate conversions for fast/efficient loading from checkpoints. - Ensure chunks which have already been flushed to storage are kept around for `ingester.retain-period`, even after a WAL replay. ### Alternatives #### Use the Cortex WAL Since we're not checkpointing from the WAL records but instead doing a memory dump, this isn't bottlenecked by throughput but rather memory size. Therefore we can start with checkpointing by duration rather than accounting for throughput as well. This makes the proposed solution nearly identical to the Cortex WAL approach. The one caveat is that wal segments will accrue between checkpoint operations and may constitute a large amount of data (log throughput varies). We may eventually consider other routes to handle this if duration based checkpointing proves insufficient. #### Don't build checkpoints from memory, instead write new WAL elements Instead of building checkpoints from memory, this would build the same efficiencies into two distinct WAL Record types: `Blocks` and `FlushedChunks`. The former is a record type which will contain an entire compressed block after it's cut and the latter will contain an entire chunk + the sequence of blocks it holds when it's flushed. This may offer good enough amortization of writes because block cuts are assumed to be evenly distributed and chunk flushes have the same property and use jitter for synchronization. This could be used to drop WAL records which have already elapsed the `ingester.retain-period`, allowing for faster WAL replays and more efficient loading. ```golang type FlushRecord struct { Fingerprint uint64 // labels FlushedAt uint64 // timestamp when it was flushed, can be used with `ingester.retain-period` to either keep or discard records on replay LastEntry logproto.Entry // last entry included in the flushed chunk } ``` It would also allow building checkpoints without relying on an ingester's internal state, but would likely require multiple WALs, partitioned by record type in order to be able to iterate all `FlushedChunks` -> `Blocks` -> `Series` -> `Samples` such that we could no-op the later (lesser priority) types that are superseded by the former types. The benefits do not seem worth the cost here, especially considering the simpler suggested alternative and the extensibility costs if we need to add new record types if/when the ingester changes internally. # Ordering Constraint Removal Author: Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) Date: 28/01/2021 ## Problem Loki imposes an ordering constraint on ingested data; that is to say incoming data must have monotonically increasing timestamps, partitioned by stream. This has historical inertia from our parent project, Cortex, but presents unintended consequences specific to log ingestion. In contrast to metric scraping, Loki has reasonable use cases where the ordering constraint poses a problem, including: - Ingesting logs from a cloud function without feeling pressured to add high cardinality labels like invocation_id to avoid out of order errors. - Ingesting logs from other agents/mechanisms that don’t take into account Loki’s ordering constraint. For instance, fluent{d,bit} variants may batch and retry writes independently of other batches, causing unpredictable log loss via out of order errors. Many of these illustrate the adversity between _ordering_ and _cardinality_. In addition to enabling some previously difficult/impossible use cases, removing the ordering constraint lets us avoid potential conflict between these two concepts and helps incentivize good practice in the form of fewer useful labels. ### Requirements - Enable out of order writes - Maintain query interface parity #### Bonuses - Optimize for in order writes ## Alternatives - Implement order-agnostic blocks + increase memory usage by compression_ratio: Deemed unacceptable due to TCO (total cost of ownership). - Implement order-agnostic blocks + scale horizontally (reduce per-ingester streams): Deemed unacceptable due to TCO and increasing ring pressure. - Implement order-agnostic blocks + flush chunks more frequently: Deemed unacceptable due to negatively increasing the index and the number of chunks requiring merge during reads. ## Design ### Background I suggest allowing a stream's head block to accept unordered writes and later re-order cut blocks similar to merge-sort before flushing them to storage. Currently, writes are accepted in monotonically increasing timestamp order to a _headBlock_, which is occasionally "cut" into a compressed, immutable _block_. In turn, these _blocks_ are combined into a _chunk_ and persisted to storage. ``` Figure 1 Data while being buffered in Ingester | Chunk in storage | Blocks Head | --------------------------------------------------------------------- | | ts0 ts1 ts2 ts3 ts4 ts5 ts6 ts7 ts8 ts9 | -------------- ---------------- | | --------- --------- --------- --------- --------- | | blocks |-- | head block | | | |block 0| |block 1| |block 2| |block 4| |block 5| | |(compressed)| | |(uncompressed)| | | | | | | | | | | | | | | | | ------> | | | | --------- --------- --------- --------- --------- | | | | | | | | | -------------- | ---------------- | --------------------------------------------------------------------- | | | -------------- | ``` Historically because of Loki's ordering constraint, these blocks maintain a monotonically increasing timestamp (abbreviated `ts`) order where ``` Figure 2 start end ts0 ts1 ts2 ts3 -------------- -------------- | | | | | | --------> | | | | | | | | | | -------------- -------------- ``` This allows us two optimizations: 1) We can store much more data in memory because each block is compressed after being cut from a head block. 2) We can query the block's metadata, such as `ts0` and `ts1` and skip querying it in the case of i.e. the timestamps are outside a request's bounds. ### Unordered Head Blocks The head block's internal structure will be replaced with a tree structure, enabling logarithmic inserts/lookups and `n log(n)` scans. _Cutting_ a block from the head block will iterate through this tree, creating a sorted block identical to the ones currently in use. However, because we'll be accepting arbitrarily-ordered writes, there will no longer be any guaranteed inter-block order. In contrast to figure 2, blocks may have overlapping data: ``` Figure 3 start end ts1 ts3 ts0 ts2 -------------- -------------- | | | | | | --------> | | | | | | | | | | -------------- -------------- ``` Thus _all_ blocks must have their metadata checked against a query. In this example, a query for the bounds `[ts1,ts2]` would need to decompress and scan the `[ts1, ts2]` range across both of them, but a query against `[ts3, ts4]` would only decompress and scan _one_ block. ``` Figure 4 chunk1 ------------------- chunk2 --------------------- query range requiring both ---------- query range requiring chunk2 only ----------- ts0 ts1 ts2 ts3 ts4 (not in any block) ------------------------------ | | | | | | | | | | | | | | | | ------------------------------ ``` The performance losses against the current approach includes: 1) Appending a log line is now performed in logarithmic time instead of amortized (due to array resizing) constant time. 2) Blocks may contain overlapping data (although ordering is still guaranteed within each block). 3) Head block scans are now `O(n log(n))` instead of `O(n)` ### Flushing and Chunk Creation Loki regularly combines multiple blocks into a chunk and "flushes" it to storage. In order to ensure that reads over flushed chunks remain as performant as possible, we will re-order a possibly-overlapping set of blocks into a set of blocks that maintain monotonically increasing order between them. From the perspective of the rest of Loki’s components (queriers/rulers fetching chunks from storage), nothing has changed. {{< admonition type="note" >}} **In the case that data for a stream is ingested in order, this is effectively a no-op, making it well optimized for in-order writes (which is both the requirement and default in Loki currently). Thus, this should have little performance impact on ordered data while enabling Loki to ingest unordered data.** {{< /admonition >}} #### Chunk Durations When `--validation.reject-old-samples` is enabled, Loki accepts incoming timestamps within the range ``` [now() - `--validation.reject-old-samples.max-age`, now() + `--validation.create-grace-period`] ``` For most of our clusters, this would mean the range of acceptable data is one week long. In contrast, our max chunk age is `2h`. Allowing unordered writes would mean that ingesters would willingly receive data for 168h, or up to 84 distinct chunk lengths. This presents a problem: a malicious user could be writing to many (84 in this case) distinct chunks simultaneously, flooding Loki with underutilized chunks which bloat the index. In order to mitigate this, there are a few options (not mutually exclusive): 1) Lower the valid acceptance range 2) Create an _active_ validity window, such as `[most_recent_sample-max_chunk_age, now() + creation_grace_period]`. The first option is simple, already available, and likely somewhat reasonable. The second is simple to implement and an effective way to ensure Loki can ingest unordered logs but maintain a sliding validity window. I expect this to cover nearly all reasonable use cases and effectively mitigate bad actors. #### Chunk Synchronization We also cut chunks according to the `sync_period`. The first timestamp ingested past this bound will trigger a cut. This process aids in increasing chunk determinism and therefore our deduplication ratio in object storage because chunks are [content addressed](https://en.wikipedia.org/wiki/Content-addressable_storage). With the removal of our ordering constraint, it's possible that in some cases the synchronization method will not be as effective, such as during concurrent writes to the same stream across this bound. {{< admonition type="note" >}} **It's important to mention that this is possible today with the current ordering constraint, but we'll be increasing the likelihood by removing it.** {{< /admonition >}} ``` Figure 5 Concurrent Writes over threshold ^ ^ | | | | -----------------|----------------- | v Sync Marker ``` To mitigate this problem and preserve the benefits of chunk deduplication, we'll need to make chunk synchronization less susceptible to non-determinism during concurrent writes. To do this, we can move the synchronization trigger from the `Append` code path to the asynchronous `FlushLoop`. Note, the semantics for _when_ a chunk is cut will not change: that is, on the first timestamp crossing the synchronization bound. However, _cutting_ the chunks for synchronization on the flush path mitigates the likelihood of _different_ chunks being cut. In order to cut multiple chunks with different hashes, appends would then need to cross this boundary at the same time the flush loop checks the stream, which should be very unlikely. ### Future Opportunities This ends the initial design portion of this document. Below, I'll describe some possible changes we can address in the future, should they become warranted. #### Variance Budget The intended approach of a "sliding validity" window for each stream is simple and effective at preventing misuse and bad actors from writing across the entire acceptable range for incoming timestamps. However, we may in the future wish to take a more sophisticated approach, introducing per tenant "variance" budgets, likely derived from the stream limit. This ingester limit could, for example use an incremental (online) standard deviation/variance algorithm such as [Welford's](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance), which would allow writing to larger ranges than option (2) in the _Chunk Durations_ section. #### LSM Tree Much of the proposed approach mirrors an [LSM-Tree](http://www.benstopford.com/2015/02/14/log-structured-merge-trees/) (Log Structured Merge Tree), albeit in memory instead of using disk. What a weird choice -- LSM Trees are designed to effectively use disk, so why not go that route? We currently have no wish to add extra disk dependencies to Loki where we can avoid it, but below I will outline what an LSM-Tree approach would look like. Ultimately, using disk would enable buffering more data in the ingester before flushing, **Allowing us to** - Flush more efficiently utilized chunks (in some cases) - Keep open a wider validity window for incoming logs **At the cost of** - being susceptible to disk-related complexity and problems ##### MemTable (head block) Writes in an LSM-Tree are first accepted to an in-memory structure called a _memtable_ (generally a balancing tree such as red-black) until the memtable hits a preconfigured size. In Loki, this corresponds to the stream’s head block, which is uncompressed. ##### SSTables (blocks) Once a Memtable (head block) in an LSM-Tree hits a predefined size, it is flushed to disk as an immutable sorted structure called an SSTable (sorted strings table). In Loki, we can use either the pre-existing MemChunk format, which is ordered, compact, and contains a block index within it, or the pre-existing block format directly. These are stored on disk to lessen memory pressure and loaded for queries when necessary. ##### Block Index Incoming reads in an LSM-Tree may need access to the SSTable entries in addition to the currently active memtable (head block). In order to improve this, we may cache the metadata including block offsets, start and end timestamps within an SSTable (block || MemChunk) in memory to mitigate lookups, seeking, and loading unnecessary data from disk. ##### Compaction (flushing) Compaction in an LSM-Tree combines and reorders multiple SSTables (blocks || MemChunks). This is mainly covered in the _Flushing_ section of the in-memory approach, but _compaction_ is equivalent to _flushing_ for our case. That is, merge multiple SSTables on disk together in an algorithm reminiscent of merge sort and flush them to storage in our ordered chunk format. # Labels Author: Ed Welch Date: February 2019 This is the official version of this doc as of 2019/04/03, the original discussion was had via a [Google doc](https://docs.google.com/document/d/16y_XFux4h2oQkJdfQgMjqu3PUxMBAq71FoKC_SkHzvk/edit?usp=sharing), which is being kept for posterity but will not be updated moving forward. ## Problem Statement We should be able to filter logs by labels extracted from log content. Keeping in mind: Loki is not a log search tool and we need to discourage the use of log labels as an attempt to recreate log search functionality. Having a label on “order number” would be bad, however, having a label on “orderType=plant” and then filtering the results on a time window with an order number would be fine. (think: grep “plant” | grep “12324134” ) Loki as a grep replacement, log tailing or log scrolling tool is highly desirable, log labels will be useful in reducing query results and improving query performance, combined with logQL to narrow down results. ## Use Cases As defined for prometheus “Use labels to differentiate the characteristics of the thing that is being measured” There are common cases where someone would want to search for all logs which had a level of “Error” or for a certain HTTP path (possibly too high cardinality), or of a certain order or event type. Examples: - Log levels. - HTTP Status codes. - Event type. ## Challenges - Logs are often unstructured data, it can be very difficult to extract reliable data from some unstructured formats, often requiring the use of complicated regular expressions. - Easy to abuse. Easy to create a Label with high cardinality, even possibly by accident with a rogue regular expression. - Where do we extract metrics and labels at the client (Promtail or other?) or Loki? Extraction at the server (Loki) side has some pros/cons. Can we do both? At least with labels we could define a set of expected labels and if Loki doesn’t receive them they could be extracted. - Server side extraction would improve interoperability at the expense of increase server workload and cost. - Are there discoverability questions/concerns with metrics exposed via Loki vs the agent? Maybe this is better/easier to manage? - Potentially more difficult to manage configuration with the server side having to match configs to incoming log streams ## Existing Solutions There already exist solutions for extracting processing and extracting metrics from unstructured log data, however, they will not quite work for extracting labels without some work and neither support easy inclusion as a library. It’s worth noting and understanding how they work to try to get the best features in our solution. mtail https://github.com/google/mtail 1721 github stars, huge number of commits, releases and contributors, google project All go, uses go RE2 regular expressions which is going to be more performant than grok_exporter below which uses a full regex implementation allowing backtracking and lookahead required to be compliant with Grok but which are also slower. grok_exporter https://github.com/fstab/grok_exporter 278 github stars, mature/active project If you are familiar with Grok this would be more comfortable, many people use ELK stacks and would likely be familiar with or already have Grok strings for their logs, making it easy to use grok_exporter to extract metrics. One caveat is the dependency on the oniguruma C library which parses the regular expressions. ## Implementation ### Details As mentioned previously in the challenges for working with unstructured data, there isn’t a good one size fits all solution for extracting structured data. The Docker log format is an example where multiple levels of processing may be required, where the docker log is json, however, it also contains the log message field which itself could be embedded json, or a log message which needs regex parsing. A pipelined approach should allow for handling these more challenging scenarios There are 2 interfaces within Promtail already that should support constructing a pipeline: ```go type EntryMiddleware interface { Wrap(next EntryHandler) EntryHandler } ``` ```go type EntryHandler interface { Handle(labels model.LabelSet, time time.Time, entry string) error } ```` Essentially every entry in the pipeline will Wrap the log line with another EntryHandler which can add to the LabelSet, set the timestamp, and mutate(or not) the log line) before it gets handed to the next stage in the pipeline. ### Example ```json { "log": "level=info msg=\”some log message\”\n", "stream": "stderr", "time": "2012-11-01T22:08:41+00:00" } ``` This is a docker format log file which is JSON but also contains a log message which has some key-value pairs. Our pipelined config might look like this: ```yaml scrape_configs: - job_name: system pipeline_stages: - json: timestamp: source: time format: RFC3339 labels: stream: source: json_key_name.json_sub_key_name output: log - regex: expr: '.*level=(?P[a-zA-Z]+).*' labels: level: - regex: expr: '.*msg=(?P[a-zA-Z]+).*' output: message ``` Looking at this a little closer: ```yaml - json: timestamp: source: time format: TODO ① labels: stream: source: json_key_name.json_sub_key_name ② output: log ③ ``` ① The format key will likely be a format string for Go’s time.Parse or a format string for strptime, this still needs to be decided, but the idea would be to specify a format string used to extract the timestamp data, for the regex parser there would also need to be a expr key used to extract the timestamp. ② One of the json elements was “stream” so we extract that as a label, if the json value matches the desired label name it should only be required to specify the label name as a key, if some mapping is required you can optionally provide a “source” key to specify where to find the label in the document. (Note the use of `json_key_name.json_sub_key_name` is just an example here and doesn't match our example log) ③ Tell the pipeline which element from the json to send to the next stage. ```yaml - regex: expr: '.*level=(?P[a-zA-Z]+).*' ① labels: level: ② ``` ① Define the Go RE2 regex, making sure to use a named capture group. ② Extract labels using the named capture group names. Notice there was not an output section defined here, omitting the output key should instruct the parser to return the incoming log message to the next stage with no changes. ```yaml - regex: expr: '.*msg=(?P[a-zA-Z]+).*' output: message ① ``` ① Send the log message as the output to the last stage in the pipeline, this will be what you want Loki to store as the log message. There is an alternative configuration that could be used here to accomplish the same result: ```yaml scrape_configs: - job_name: system pipeline_stages: - json: timestamp: source: time format: FIXME labels: stream: output: log - regex: expr: '.*level=(?P[a-zA-Z]+).*msg=(?P[a-zA-Z]+).*' labels: level: ① log: source: message ② output: message ``` ① Similar to the json parser, if your log label matches the regex named group, you need only specify the label name as a yaml key ② If you had a use case for specifying a different label name from the regex group name you can optionally provide the `source` key with the value matching the named capture group. You can define a more complicated regular expression with multiple capture groups to extract many labels and/or the output log message in one entry parser. This has the advantage of being more performant, however, the regular expression will also get much more complicated. Note the regex for `message` is incomplete and would do a terrible job of matching any standard log message which might contain spaces or non alpha characters. ### Concerns - Debugging, especially if a pipeline stage is mutating the log entry. - Clashing labels and how to handle this (two stages try to set the same label) - Performance vs ease of writing/use, if every label is extracted one at a time and there are a lot of labels and a long line, it would force reading the line many times, however contrast this to a really long complicated regex which only has to read the line once but is difficult to write and/or change and maintain ### Further improvements There are some basic building blocks for our pipeline which will use the EntryMiddleware interface, the two most commonly used will likely be: - Regex Parser - JSON Parser However we don’t want to ask people to copy and paste basic configs over and over for very common use cases, so it would make sense to add some additional parsers which would really be supersets of the base parsers above. For example, the config above might be simplified to: ```yaml scrape_configs: - job_name: system pipeline_stages: - docker: ``` or ```yaml scrape_configs: - job_name: system pipeline_stages: - cri: ``` Which could still easily be extended to extract additional labels: ```yaml scrape_configs: - job_name: system pipeline_stages: - docker: - regex: expr: '.*level=(?P[a-zA-Z]+).*' labels: level: ``` ### Auto Detection? An even further simplification would be to attempt to autodetect the log format, a PR for this work has been submitted, then the config could be as simple as: ```yaml scrape_configs: - job_name: system pipeline_stages: - auto: ``` This certainly has some advantages for people first adopting and testing Loki, allowing them to point it at their logs and at least get the timestamp and log message extracted properly for the common formats like Docker and CRI. There are also some challenges with auto detection and edge cases, though most people are going to want to augment the basic config with additional labels, so maybe it makes sense to default to auto but suggest when people start writing configs they chose the correct parser? ## Other Thoughts and Considerations - We should have a standalone client in some fashion which allows for testing of log parsing at the command line, allowing users to validate regular expressions or configurations to see what information is extracted. - Other input formats to the pipeline which are not reading from log files, such as containerd grpc api, or from stdin or unix pipes, etc. - It would be nice if at some point we could support loading code into the pipeline stages for even more advanced/powerful parsing capabilities. # Releasing Loki Build Image The [`loki-build-image`](https://github.com/grafana/loki/blob/main/loki-build-image) is the Docker image used to run tests and build Grafana Loki binaries in CI. The build and publish process of the image is triggered upon a merge to `main` if any changes were made in the folder `./loki-build-image/`. **To build and use the `loki-build-image`:** 1. Create a branch with the desired changes to the `./loki-build-image/Dockerfile`. 1. Update the `BUILD_IMAGE_VERSION` variable in the `Makefile`. 1. Commit your changes. 1. Run `make build-image-push` to build and publish the new version of the build image. 1. Run `make release-workflows` to update the Github workflows. 1. Commit your changes. 1. Push your changes to the remote branch. 1. Open a PR against the `main` branch. # Backport commits Any PRs or commits not on the release branch that you want to include in the release must be backported to the release branch. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 2. If the release branch already has all the code changes on it, skip this step. ## Steps 1. Pick a PR that you want to backport to `release-VERSION_PREFIX` branch. 1. Add two labels to the PR. First, one of the `product-approved`, `type/doc` or `type/bug` appropriately. This is to make sure the PRs that are backported are done with right intention. Second `backport release-VERSION_PREFIX` label. Now CI should automatically create backport PR to the correct release branch. Example [PR](https://github.com/grafana/loki/pull/10333) {{< admonition type="note" >}} CI automation can fail sometimes if there are some merge conflicts in cherry picking the commits. In those cases, the original PR where you added the label should have additional comment explaining how to backport it manually. {{< /admonition >}} {{< admonition type="note" >}} The CI job that helps with backporting PR is `.github/workflows/backport.yml`. Useful for debugging purposes. {{< /admonition >}} 1. Repeat the above steps for any PRs that need to be backported. ## Backporting Release PRs If backporting a release PR, make sure you remove any `autorelease: pending` or `autorelease: tagged` labels before merging the backport PR. By default our backport action brings over all labels, but these labels are reserved for the release workflow and will cause future pipelines to fail if left of backport PRs. # Create Release Branch A single release branch is created for every `major` or `minor` release (not for patch releases). That release branch is then used for all the Stable Releases, and all Patch Releases for that `major` and `minor` versions of the Grafana Loki. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 1. Announce about the upcoming release in `#loki-releases` internal slack channel. 1. Skip this announcement for a patch release. Create an issue to communicate beginning of the release process with the community. Example issue [here](https://github.com/grafana/loki/issues/10468). ## Steps 1. Determine which commit should be used as a base for the release branch. Usually this is a `kxx` weekly release branch. 1. Create and push the release branch from the selected base commit: The name of the release branch should be `release-VERSION_PREFIX`, such as `release-2.9.x`. {{< admonition type="note" >}} Branches are only made for VERSION_PREFIX; do not create branches for the full VERSION such as `release-2.9.1`. {{< /admonition >}} {{< admonition type="note" >}} Don't create any other branches that are prefixed with `release` when creating PRs or those branches will collide with our automated release build publish rules. {{< /admonition >}} 1. Create a label to make backporting PRs to this branch easy. The name of the label should be `backport release-VERSION_PREFIX`, such as `backport release-2.9.x`. {{< admonition type="note" >}} Note there is space in the label name. The label name must follow this naming convention to trigger CI related jobs. {{< /admonition >}} # Document metrics and configurations changes Any metrics and configurations that are removed or modified need to be documented in the upgrade guide. Configurations whose default values changed should also be documented in the upgrade guide. ## Before you begin All the steps are performed on `release-VERSION_PREFIX` branch. ## Steps 1. Check which configs changed, including whose default values changed. ``` $ OLD_VERSION=X.Y.Z ./tools/diff-config.sh ``` 1. Record configurations that are modified (either renamed or had its default value changed) in the [upgrade guide](../prepare-upgrade-guide/). 1. Check if any metrics have changed. ``` $ OLD_VERSION=X.Y.Z ./tools/diff-metrics.sh ``` 1. Record metrics whose names have been modified in the [upgrade guide](../prepare-upgrade-guide/). # Prepare Major Release A major release follows the same process as [minor and patch releases](../prepare-release/), but requires a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don't do major releases very often, so it is not economical to keep those workflows running all the time. To create a major release workflow, follow the steps below. 1. Edit `./github/release-workflows.jsonnet` 1. Add a new workflow for the major release. For example, the 3.0 release looked like the following: ```jsonnet 'major-release-pr.yml': std.manifestYamlDoc( lokiRelease.releasePRWorkflow( branches=['release-*.0.0'], buildImage=buildImage, checkTemplate=checkTemplate, distRunsOn='ubuntu-x64', distOptionalTargets=['dist/loki-linux-riscv64'], golangCiLintVersion=golangCiLintVersion, imageBuildTimeoutMin=imageBuildTimeoutMin, imageJobs=imageJobs, imagePrefix=imagePrefix, releaseLibRef=releaseLibRef, releaseRepo='grafana/loki', skipArm=false, skipValidation=false, // Instead of hard-coding the release version, push an empty to the target branch `release-X.0.0` // git commit --allow-empty -m "chore: Release X.0.0-rcY" -m "Release-As: X.0.0-rcY" // https://github.com/googleapis/release-please#how-do-i-change-the-version-number // releaseAs='x.0.0', ) + { name: 'Prepare Major Release PR', }, false, false ), ``` 1. Make sure the `branches` field is set to the release branch you want to release from. 1. Make sure the `releaseAs` field is set to the version you want to release. 1. Run `make release-workflows` to generate the new workflow. Merge this change to both the main and release branch. You may also want to disable the patch release workflow as part of this PR, see below. ## Race Condition with Patch Release Once this new workflow is created there is a bit of a race condition with the patch workflow. This can be resolved in 1 of 2 ways: 1. Disable the patch release workflow until after the major release is complete. 1. Watch all actions for the release branch (ie. ) and cancel and patch release runs. # Merge Release PR To release Loki, merge the release PR. This PR will have the title `chore(): release `. Here's what the 3.0 release PR looked like. ![3.0 release PR](.3.0-release-pr.png) 3.0 was the first major release using this new process, and enforcing conventional commits was relatively new. Going forward the release notes in the release PR will be much more thorough, as we're now enforcing that every PR has a conventional commit message. To test the artifacts before releasing, you can download the artifacts from the link in the footer of the PR description. Merging the PR will: 1. Fetch the built artifacts. 1. Create a draft GitHub release with the release notes in the PR description. 1. Upload fetched binaries to the draft release. 1. Publish fetched images to Docker Hub as multi-arch images. 1. Publish the draft release and create the GitHub tag. 1. (Optionally) Mark the release as the latest if it represents the newest version of Loki. ## Troubleshooting / Retrying Release If something goes wrong with the release automation, you may need to rerun the job. This may require updating the release code in the `grafana/loki` repo via a PR, or it may include updating the code in `grafana/loki-release` that the release pipeline fetches. In either case, if you need to re-release a merge release PR, you'll need to remove the `autorelease: tagged` label from that PR and add the `autorelease: pending` label. The automation relies on these labels to determine which merged PRs have and have not yet been released. After fixing the labels, re-run the release workflow, and it should correctly get passed the `should-release` step. The release process is idempotent, so if a draft release has already been created via a previously failed release, the process will continue, re-upload the binaries, and re-publish the images. Once the release is finished, you may need to manually flip the label back to `autorelease: tagged`. # Patch Go version Update vulnerable Go version to non-vulnerable Go version to build Grafana Loki binaries. ## Steps 1. Find Go version to which you need to update. Example `1.20.5` to `1.20.6` 1. Update Go version (`GO_VERSION`) in the `Makefile`. 1. Run `make release-workflows` to update the version in generated release workflows. 1. Run `make update-go-version` to update all relevant files. 1. Open a pull request against the target branch (`main` or `release-MAJOR.MINOR.x`). # Patch vulnerabilities This step patches vulnerabilities in Grafana Loki binaries and Docker images. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). Vulnerabilities can be from two main sources. 1. Grafana Loki source code. 1. Grafana Loki dependencies. Grafana Loki dependencies can be 1. Go dependencies in `go.mod` 1. The Go compiler itself 1. Grafana Loki Docker dependencies, for example, the base images Before start patching vulnerabilities, know what are you patching. It can be one or more from sources mentioned above. Use `#security-go`, `#security` slack channels to clarify. ## Steps 1. Patch Grafana Loki source code. Means, there are vulnerabilities in Grafana Loki source code itself. 1. Patch it on `main` branch 1. [Backport](../backport-commits/) to `release-$VERSION_PREFIX` branch. 1. Patch Go dependencies. 1. Pick all the Go dependencies that need to be patched. 1. Check if [dependabot already patched the dependency](https://github.com/grafana/loki/pulls?q=is%3Apr+label%3Adependencies+is%3Aclosed) or [have a PR opened to patch](https://github.com/grafana/loki/pulls?q=is%3Apr+is%3Aopen+label%3Adependencies) . If not, manually upgrade the package on the `main` branch as follows. ```shell go get -u -v @ go mod tidy go mod vendor ``` 1. [Backport](../backport-commits/) it to `release-$VERSION_PREFIX` branch. 1. Repeat for each Go dependency 1. [Patch Go compiler](../patch-go-version/). 1. Patch Grafana Loki Docker dependencies, for example: Alpine Linux base images). 1. Update Docker image version. [Example PR](https://github.com/grafana/loki/pull/10573). 1. [Backport](../backport-commits/) to `release-$VERSION_PREFIX` branch # Prepare Release Releasing Grafana Loki consists of merging a long-running release PR. Two workflows keep these PRs up to date, one for patch releases (which runs on commits to branches matching `release-[0-9]+.[0-9]+.x`) and one for minor releases (which runs on commits to branches matching `k[0-9]+`). These pipelines use release please to do the following: 1. Run tests and linting 1. Build binaries and images for the proposed release version 1. Generate release notes based on conventional commits since the last release 1. Create or update the long-running release PR for that release, indicating which commit will be released if the PR is merged with a link to the built artifacts that will be published. ## Major releases Major releases follow the same process as minor and patch releases, but require a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don't do major releases very often, so it is not economical to keep those workflows running all the time.To create a major release workflow, follow the steps in the [major release workflow](../major-release/) document. # Prepare Upgrade guide The upgrade guide records changes that require user attention or interaction to upgrade to specific Loki version from previous versions. ## Before you begin The upgrade guide in Grafana Loki works as follows. We have a `setup/upgrade/_index.md` file that records upgrade information for all Loki releases. ## Steps 1. Make sure the upgrade guide is up to date on the `release-VERSION_PREFIX` branch under the `Main/Unreleased` section. 1. On the `release-VERSION_PREFIX` branch promote `Main/Unreleased` to `VERSION`. Example [PR](https://github.com/grafana/loki/pull/10470). 1. On the `main` branch remove entries from `Main/Unreleased` that are already part of `VERSION (YYY-MM-DD)`. # Update version numbers Upgrade the Loki version to the new release version in documents, examples, jsonnets, etc. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 2. Skip this step if you are doing a patch release on old release branch. Example: Latest release branch is `release-2.9.x` but you are releasing patch version in `release-2.8.x` (or older) ## Steps 1. Upgrade the versions in the `release-VERSION_PREFIX` branch. Example commands: ``` LOKI_NEW_VERSION=$VERSION ./tools/release_update_tags.sh ``` {{< admonition type="note" >}} Do not upgrade the version numbers in the `operator/` directory as @periklis and team have a different process to upgrade the Operator version. {{< /admonition >}} # Version Grafana Loki uses Semantic Versioning. The next version can be determined by looking at the current version and incrementing it. When the release is a new `major` version (for example, after 2.9.1 we decide to go 3.0.0 instead of 2.9.2) this should not be done. You need to set two environmental values `VERSION` and `VERSION_PREFIX` to do the release process. ## Version To determine the `VERSION` for a Stable Release or Patch Release, use the Semantic Version `A.B.C`. - Examples - For example, `2.9.0` is the Stable Release `VERSION` for the v2.9.0 release. - For example, `2.9.1` is the first Patch Release `VERSION` for the v2.9.0 release. ## Version Prefix To determine the `VERSION PREFIX`, use only the major and minor version `A.B.x`. - Examples - `2.9.x` # Configure # Configuration best practices Grafana Loki is under active development, and the Loki team is constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki. ## Configure authentication Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Loki (for example, nginx). Refer to [Manage authentication](https://grafana.com/docs/loki//operations/authentication/) for a list of open-source reverse proxies you can use. ## Configure caching Loki can cache data at many levels, which can drastically improve performance. Details of this will be in a future post. ## Time ordering of logs Loki [accepts out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) _by default_. This section identifies best practices when Loki is _not_ configured to accept out-of-order writes. One issue many people have with Loki is their client receiving errors for out of order log entries. This happens because of this hard and fast rule within Loki: - For any single log stream, logs must always be sent in increasing time order. If a log is received with a timestamp older than the most recent log received for that stream, that log will be dropped. There are a few things to dissect from that statement. The first is this restriction is per stream. Let’s look at an example: ```bash {job="syslog"} 00:00:00 i'm a syslog! {job="syslog"} 00:00:01 i'm a syslog! ``` If Loki received these two lines which are for the same stream, everything would be fine. But what about this case: ```bash {job="syslog"} 00:00:00 i'm a syslog! {job="syslog"} 00:00:02 i'm a syslog! {job="syslog"} 00:00:01 i'm a syslog! <- Rejected out of order! ``` What can you do about this? What if this was because the sources of these logs were different systems? You can solve this with an additional label which is unique per system: ```bash {job="syslog", instance="host1"} 00:00:00 i'm a syslog! {job="syslog", instance="host1"} 00:00:02 i'm a syslog! {job="syslog", instance="host2"} 00:00:01 i'm a syslog! <- Accepted, this is a new stream! {job="syslog", instance="host1"} 00:00:03 i'm a syslog! <- Accepted, still in order for stream 1 {job="syslog", instance="host2"} 00:00:02 i'm a syslog! <- Accepted, still in order for stream 2 ``` But what if the application itself generated logs that were out of order? Well, I'm afraid this is a problem. If you are extracting the timestamp from the log line with something like the [Alloy `stage.timestamp` block](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagetimestamp), you could instead _not_ do this and let your log shipping agent assign a timestamp to the log lines. Or you can hopefully fix it in the application itself. It's also worth noting that the batching nature of the Loki push API can lead to some instances of out of order errors being received which are really false positives. (Perhaps a batch partially succeeded and was present; or anything that previously succeeded would return an out of order entry; or anything new would be accepted.) ## Consider the `snappy` compression algorithm The default `chunk_encoding` is `gzip`, which compresses well but is comparatively slow, and can cause slow query responses under load. `Snappy` is a recommended override for `chunk_encoding` in higher-throughput environments. It performs much better than `gzip` for speed, but it is not as efficient in storage. Whether that tradeoff is acceptable depends on your workload. `LZ4` is a good compromise of speed and compression performance. While compression is slightly slower than `snappy`, the compression ratio is higher, resulting in smaller chunks in object storage. ## Use `chunk_target_size` Using `chunk_target_size` instructs Loki to try to fill all chunks to a target _compressed_ size of 1.5MB. These larger chunks are more efficient for Loki to process. Other configuration variables affect how full a chunk can get. Loki has a default `max_chunk_age` of 2h and `chunk_idle_period` of 30m to limit the amount of memory used as well as the exposure of lost logs if the process crashes. Depending on the compression used (for example, `snappy`, which has less compressibility but faster performance than the `gzip` default), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled. Lots of small, unfilled chunks negatively affect Loki. The team is always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks. If you have an application that can log fast enough to fill these chunks quickly (much less than `max_chunk_age`), then it becomes more reasonable to use dynamic labels to break that up into separate streams. ## Use `-print-config-stderr` or `-log-config-reverse-order` Loki has flags which will dump the entire config object to stderr or the log file when it starts. `-print-config-stderr` works well when invoking Loki from the command line, as you can get a quick output of the entire Loki configuration. `-log-config-reverse-order` is the flag Grafana runs Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana's Explore. ## Recommended production limits ```yaml limits_config: # Rate limits ingestion_rate_strategy: global ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 per_stream_rate_limit: 3MB per_stream_rate_limit_burst: 15MB # Stream limits max_global_streams_per_user: 10000 max_streams_per_user: 0 # Validation max_line_size: 256KB max_line_size_truncate: false max_label_name_length: 1024 max_label_value_length: 2048 max_label_names_per_series: 15 # Time constraints reject_old_samples: true reject_old_samples_max_age: 168h # 7 days creation_grace_period: 10m ``` ## Recommended Ingester configuration ```yaml ingester: # Chunk settings chunk_idle_period: 30m chunk_target_size: 1572864 # 1.5 MB chunk_encoding: snappy max_chunk_age: 2h # WAL settings wal: enabled: true dir: /loki/wal checkpoint_duration: 5m flush_on_shutdown: true ``` # Understanding Loki configuration limits Loki implements configuration limits to maintain system stability, ensure fair resource distribution, and protect performance across all users. You can adjust many limits for specific use cases. Other limits are hard boundaries that you should not exceed, to prevent performance degradation and system instability. This topic explains the critical configuration limits in Loki, why they exist, and the consequences of exceeding them. ## Why limits exist Configuration limits in Loki serve multiple essential purposes: * **System stability:** Limits prevent individual queries or ingestion patterns from exhausting system resources. Without limits, a single tenant or misconfigured application could consume all available memory, CPU, or I/O (input/output) capacity. This can cause cascading failures across the entire system. * **Performance protection:** Loki is optimized for specific usage patterns. Limits ensure that workloads stay within the parameters where Loki performs efficiently. Exceeding these limits often results in exponential performance degradation rather than linear scaling. * **Fair resource allocation:** In multi-tenant environments, limits ensure that no single tenant can monopolize shared resources. Query workers, ingester memory, and storage I/O must be distributed fairly across all users. * **Operational predictability:** Limits allow operators to plan capacity and scale infrastructure appropriately. When operators understand the boundaries of expected workloads, they can provision resources that meet service level agreement (SLA) requirements. Exceeding a limit doesn't just affect the tenant who exceeded it. It can also impact overall system performance and stability, often affecting all tenants in a shared environment. ## Default ingest limits The following limits apply when you push logs to Loki. These are the default values in the Loki open source release. If a request exceeds one of these limits, Loki rejects the affected log lines or streams and returns an error. | Limit | Default value | What happens when you exceed it | | --- | --- | --- | | Maximum active streams per tenant (`max_global_streams_per_user`) | 5,000 | New streams are rejected (`stream_limit`) until existing streams age out. | | Ingestion rate (`ingestion_rate_mb`) | 4 MB per second, per tenant | Requests are rejected and can be retried (`rate_limited`). | | Ingestion burst size (`ingestion_burst_size_mb`) | 6 MB | Requests that exceed the burst allowance are rejected until the rate limit bucket refills. | | Per-stream rate limit (`per_stream_rate_limit`) | 3 MB per second | The stream is rate limited (`per_stream_rate_limit`). | | Per-stream burst limit (`per_stream_rate_limit_burst`) | 15 MB | Bursts that exceed this size are rejected. | | Maximum log line size (`max_line_size`) | 256 KB | Log lines that exceed the limit are discarded (`line_too_long`), unless you configure Loki to truncate them instead. | | Maximum labels per stream (`max_label_names_per_series`) | 15 | Streams that exceed the limit are rejected (`max_label_names_per_series`). | | Maximum length of a label name (`max_length_label_name`) | 1,024 bytes | Label names that exceed the limit are rejected (`label_name_too_long`). | | Maximum length of a label value (`max_length_label_value`) | 2,048 bytes | Label values that exceed the limit are rejected (`label_value_too_long`). | | Maximum structured metadata entries per log line (`max_structured_metadata_entries_count`) | 128 entries | Log lines with more entries are discarded. | | Maximum structured metadata size per log line (`max_structured_metadata_size`) | 64 KB | Log lines that exceed the limit are discarded. | | Maximum accepted sample age (`reject_old_samples_max_age`) | 1 week | Samples older than this age are rejected (`greater_than_max_sample_age`). | Loki rejects samples based on their age only when the `reject_old_samples` setting is `true`, which is the default. If you set `reject_old_samples` to `false`, Loki ignores `reject_old_samples_max_age`. Ingestion rate and maximum active streams aren't fixed for every deployment. Many operators raise these limits for larger deployments. For guidance on how far you can safely raise them, see [Limits we strongly recommend not exceeding](#limits-we-strongly-recommend-not-exceeding). ## Default query limits The following limits apply when you query logs in Loki. These are the default values in the Loki open source release. | Limit | Default value | What happens when you exceed it | | --- | --- | --- | | Query timeout (`query_timeout`) | 1 minute | The query is terminated before it finishes. | | Maximum log entries returned per query (`max_entries_limit_per_query`) | 5,000 | Queries that request more entries than this limit are rejected. | | Maximum unique series returned by a metric query (`max_query_series`) | 500 | The query is rejected. | | Maximum query parallelism (`max_query_parallelism`) | 32 | Additional subqueries wait in a queue until a query worker is free. | | Maximum query time range (`max_query_length`, `store.max-query-length`) | 721 hours (30 days and 1 hour) | The query is rejected. | | Maximum chunks fetched per query (`store.query-chunk-limit`) | 2,000,000 | The query is rejected. | | Maximum stream matchers per query (`max_streams_matcher_per_query`) | 1,000 | The query is rejected. | | Maximum concurrent tail requests (`max_concurrent_tail_requests`) | 10 | Additional tail requests are rejected. | ## Limits we strongly recommend not exceeding Some limits can technically be raised well beyond their defaults, but doing so puts your Loki cluster at risk. This section explains which limits fall into that category, why they exist, and what to do instead of raising them further. ### ingestion_rate_mb - **Default**: 4 MB per second - **Example large deployment value**: 100 MB per second (approximately 260 TB of data per month) - **Recommended maximum for large deployments**: up to 250 MB per second This limit controls the per-tenant ingestion rate in megabytes per second. **Why this limit matters**: Ingestion rate limits protect against cascading failures: - Sudden traffic spikes can overwhelm ingesters before your infrastructure can scale to meet demand. - Unbounded ingestion can exhaust ingester memory. - Unbounded ingestion can exhaust distributor memory. - Network and storage I/O can become bottlenecks. Rate limits provide a safety mechanism that allows the system to scale gracefully, rather than failing all at once. **Consequences of not setting rate limits properly**: - Cascading failure of distributors and ingesters, leading to an outage of the entire write path and the recent-data read path. - Ingesters overwhelmed to the point that their write-ahead logs (WALs) run out of disk space, causing slow WAL replay and difficult recovery. - Object storage overwhelmed by excessive requests. **What to do instead**: - Always set rate limits for every tenant. - Implement exponential backoff in your log shipping clients. - Monitor ingestion patterns and adjust limits proactively. - Scale infrastructure before increasing limits significantly. ### max_global_streams_per_user - **Default**: 5,000 - **Recommended maximum for large deployments**: 80,000 - **Hard limit**: approximately 200,000 (performance issues become unavoidable beyond this point) This limit controls the maximum number of active streams a single tenant can have across all ingesters. Each unique combination of labels creates a separate stream. **Why you should not exceed 200,000 streams**: Streams consume resources throughout the system: - **Ingester memory**: Each stream maintains in-memory chunks. - **Index entries**: Each stream creates entries in the index. - **Object storage operations**: Each stream generates flush operations. - **Query overhead**: More streams means more chunks to scan during queries. Beyond approximately 200,000 streams, these costs become prohibitive regardless of infrastructure size. **Consequences of exceeding**: - Ingester memory exhaustion and crashes. - Extremely slow query performance. - Index query timeouts. - Excessive object storage operations and costs. - Inability to meet performance SLAs. - Potential data loss during ingester failures. **What to do instead**: - Audit your label usage to identify high-cardinality labels. - Use static labels for source identification only. - Move high-cardinality data to structured metadata. - Calculate your expected stream count: `streams = label1_values × label2_values × ...` - Design for fewer than 80,000 streams. ### max_label_names_per_series - **Default**: 15 - **Recommended maximum**: 15 (do not exceed) This limit controls the maximum number of labels you can attach to a single log stream. **Why you should not exceed 15 labels**: You should be able to identify and categorize your logs with fewer than 15 labels. More labels increase the likelihood of higher cardinality and cause logs to fragment into too many streams and chunks. Even labels with a small, fixed set of values still require space in the index, and each additional label makes the index larger and can slow down index operations. Always use as few labels as possible. Having more than 15 labels per stream usually indicates a labeling problem, not a limit you need to raise. Loki is designed for a small set of labels that describe the log source, such as `cluster`, `namespace`, or `job`. Labels aren't meant for storing high-cardinality metadata. **Consequences of exceeding**: - Increased memory usage on ingesters. - A larger index, which leads to slower index operations and slower queries. **What to do instead**: - Use labels only for low-cardinality source identification. - Parse high-cardinality fields at query time using LogQL parsers. - Store high-cardinality metadata in [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) instead of labels. ### max_line_size - **Default**: 256 KB - **Maximum recommended**: 256 KB (do not exceed) This limit controls the maximum size, in bytes, of a single log line during ingestion. **Why you should not exceed 256 KB**: A limit on log line size is fundamental to Loki's design. Increasing this limit can have several unintended consequences that make it hard to run a stable Loki cluster: - Chunks are expected to be a few MB at most. If a log line is larger than this, it can create very large chunks, which hurts query performance because a chunk is the unit of parallelism and is expected to stay around 2 MB or smaller. - Queriers and query frontends become much more likely to run out of memory (OOM). - Increasing this limit also increases the risk of exhausting the gRPC message size limit, which can affect the performance of other tenants. **What to do instead**: - Structure your applications to produce reasonably sized log lines. - Avoid embedding large payloads (such as base64-encoded data or full HTTP bodies) in logs. - Log references to external data instead of the data itself. - Split multi-line events into separate log entries. - Set `max_line_size_truncate` to truncate large log lines instead of dropping them. Note that truncation can break JSON parsing of large log lines. ### Out-of-order ingestion window - **Default**: 1 hour (half of `max_chunk_age`, which defaults to 2 hours) - **Adjustable**: No, not on a per-tenant basis The out-of-order ingestion window controls how far back in time, relative to the newest entry already received for a stream, Loki accepts a new log entry. Loki calculates this window as `max_chunk_age / 2`, so with the default 2-hour `max_chunk_age`, Loki accepts entries up to 1 hour older than the current newest entry in each stream. This window isn't a per-tenant override. It's a cluster-wide setting, because it derives from `max_chunk_age`, which affects chunk flushing behavior across the whole deployment. **Why this limit exists**: The reasons users hit this limit vary widely: misconfiguration, re-ingestion of old data, or logs from temporarily disconnected sources such as vehicles or IoT devices. Loki needs a bounded window so that ingesters know when a chunk is complete and safe to flush. **What to do for historical data**: If you need to ingest logs older than the out-of-order window allows, you have two options: - Temporarily increase `reject_old_samples_max_age` during migrations or backfills, so Loki doesn't reject the old samples outright. - Enable `shard_streams.time_sharding_enabled` for the affected tenant. This setting splits incoming streams by injecting a synthetic `__time_shard__` label, so that no stream is ever longer than `max_chunk_age / 2`. This lets Loki accept very old logs, because the new, shorter streams are never "too far behind" by design. Time sharding has some limitations: - Newly ingested old logs aren't queryable for a few hours, until Loki flushes them to object storage. - Very old logs can still be rejected with a `greater_than_max_sample_age` error if they fall outside `reject_old_samples_max_age`. - Recording rules don't work against the old data. - Time sharding creates new time series (up to 24 per day), so check whether this affects your active streams limit. Disable time sharding after your backfill completes. ### per_stream_rate_limit - **Default**: 3 MB per second sustained, 15 MB burst - **Recommended maximum sustained rate**: 5 MB per second Individual streams have their own rate limits. This prevents a single high-volume stream from overwhelming the ingesters it is distributed to, based on the replication factor. **Why you should not exceed 5 MB per second per stream**: Streams are distributed to a fixed number of ingesters (typically 3, for replication). If a single stream sends data faster than those ingesters can process it: - Ingester memory queues grow. - Chunk flush operations lag behind incoming data. - Memory usage spikes on the affected ingesters. - The affected ingesters risk running out of memory (OOM). Unlike the global rate limit, you cannot solve per-stream limit problems by adding more ingesters, because each stream is always handled by the same set of ingesters. **Consequences of exceeding**: - Memory buildup on specific ingesters. - Increased flush latency. - Potential out-of-memory (OOM) failures on affected ingesters. - Data loss if an ingester crashes. - Uneven load distribution. **What to do instead**: - First, try adjusting automatic stream sharding by lowering `desired_rate` (not below 128 KB per second). - If that isn't enough, increase `per_stream_rate_limit` and `per_stream_rate_limit_burst`. This might require more memory headroom on your ingesters. Increase the burst limit more aggressively than the sustained rate. For example, to handle "bursty" streams, you could raise the sustained rate a little and the burst limit a lot: - `per_stream_rate_limit: 5MB/s` - `per_stream_rate_limit_burst: 30MB/s` This gives each stream a bucket that holds 30 MB. A 30 MB push drains the entire bucket, which then refills at 5 MB per second up to the 30 MB cap. You could raise the burst limit further, to 60 MB, to handle spikier streams, but avoid raising the sustained rate too far, since that keeps overall memory use predictable. Automatic stream sharding also continuously works to keep stream volumes near `desired_rate`, but it can't always react fast enough to very sudden changes. ### query_timeout - **Default**: 1 minute - **Recommended maximum**: 5 minutes (do not exceed) The query timeout controls how long a query can run before Loki terminates it. This limit exists because Loki is designed as a synchronous query engine, not for very long-running queries. **Why you should not exceed 5 minutes**: Loki was not engineered for very long-running query operations. The way Loki queries work, each querier runs a fixed number of worker routines. Query frontends and schedulers split a query by time range or by label set and enqueue each resulting subquery, and a single worker routine executes each one. All tenants share this pool of query workers. Raising the timeout for one tenant lets that tenant reserve query workers for longer, which reduces the query capacity available to every other tenant. In multi-tenant or large multi-user environments, this can make it difficult to maintain system stability. **Consequences of exceeding**: - Reduced query capacity for all tenants. - Increased query queue wait times. - Potential denial of service to other users. - Unpredictable query performance. **What to do instead**: Optimize your queries to complete faster. Use smaller time ranges, more specific labels, and filters applied as early as possible in the query. ## Understanding limit violations When you exceed a limit, Loki provides metrics and errors to help you understand what happened. ### Monitoring metrics - **`loki_discarded_samples_total`**: Counts log lines discarded, with a `reason` label indicating which limit was exceeded. - **`loki_discarded_bytes_total`**: Measures bytes discarded, by reason. - **`loki_ingester_memory_streams`**: Current number of active streams in memory, per tenant. ### Common error reasons - **`rate_limited`**: Exceeded `ingestion_rate_mb`. - **`per_stream_rate_limit`**: An individual stream exceeded its rate limit. - **`stream_limit`**: Exceeded `max_global_streams_per_user`. - **`line_too_long`**: A log line exceeded `max_line_size`. - **`max_label_names_per_series`**: Too many labels on a stream. - **`label_name_too_long`**: A label name exceeded `max_length_label_name`. - **`label_value_too_long`**: A label value exceeded `max_length_label_value`. - **`too_far_behind`**: A log entry fell outside the out-of-order ingestion window. - **`greater_than_max_sample_age`**: A log entry is older than `reject_old_samples_max_age`. ## Best practices for limits To work effectively within Loki's limits: 1. **Design for low cardinality**: Use 5 to 10 labels at most, with static values. 2. **Keep logs reasonably sized**: Target 1 to 10 KB per log line. 3. **Forward logs promptly**: Aim for sub-second delivery latency. 4. **Monitor your usage**: Set up alerts on discarded-sample metrics. 5. **Scale infrastructure first**: Increase resources before you increase limits. 6. **Use alternatives**: Use structured metadata and query-time parsing instead of more labels. 7. **Plan capacity**: Calculate your expected stream count before deployment. 8. **Test at scale**: Validate your label design with production-like data volumes. ## Next steps To change any of these limits for your own deployment, see the `limits_config` block in the [Configuration reference](https://grafana.com/docs/loki//configure/), and review the related topics below before you raise a limit. ## Related documentation - [Configure Loki](https://grafana.com/docs/loki//configure/bp-configure/) - Practical guide to configuring Loki - [Rate limits and validation](https://grafana.com/docs/loki//operations/request-validation-rate-limits/) - Detailed error handling - [Understanding cardinality](https://grafana.com/docs/loki//get-started/labels/cardinality/) - How cardinality affects performance - [Labels best practices](https://grafana.com/docs/loki//get-started/labels/bp-labels/) - Guide to label design - [Structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) - Alternative to high-cardinality labels # Storage Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki. Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. This type only requires one store, the object store, for both the index and chunks. More detailed information about TSDB can be found under the [manage section](https://grafana.com/docs/loki//operations/storage/tsdb/). ## Single Store TSDB Single Store refers to using object storage as the storage medium for both the Loki index as well as its data ("chunks"). There is one supported mode: Starting in Loki 2.8, the [TSDB index store](https://grafana.com/docs/loki//operations/storage/tsdb/) improves query performance, reduces TCO and has the same feature parity as the deprecated "boltdb-shipper". TSDB is the recommended index store for Loki 2.8 and newer. ### Supported storage backends See [Object Storage](#object-storage) for supported backends. ## Chunk storage ### File system The file system is the simplest backend for chunks, although it's also susceptible to data loss as it's unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. It is similar in concept to many Prometheus deployments where a single Prometheus is responsible for monitoring a fleet. ### Object storage #### Google Cloud Storage (GCS) GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you're already running on GCP, and is production safe. #### Amazon Simple Storage Storage (S3) S3 is AWS's hosted object store. It is a good candidate for a managed object store, especially when you're already running on AWS, and is production safe. #### Azure Blob Storage Blob Storage is Microsoft Azure's hosted object store. It is a good candidate for a managed object store, especially when you're already running on Azure, and is production safe. You can authenticate Blob Storage access by using a storage account name and key or by using a Service Principal. #### IBM Cloud Object Storage (COS) [COS](https://www.ibm.com/cloud/object-storage) is IBM Cloud hosted object store. It is a good candidate for a managed object store, especially when you're already running on IBM Cloud, and is production safe. #### Baidu Object Storage (BOS) [BOS](https://intl.cloud.baidu.com/product/bos.html) is the Baidu Cloud hosted object storage. #### Alibaba Object Storage Service (OSS) [OSS](https://www.alibabacloud.com/product/object-storage-service) is the Alibaba Cloud hosted object storage. #### Other notable mentions You may use any substitutable services, such as those that implement the S3 API like [MinIO](https://min.io/). ## Schema Config Loki aims to be backwards compatible and over the course of its development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage _schemas_ and can query across them transparently. This makes upgrading a breeze. For a more detailed reference on schema configuration, including required values and the recommended settings for new installs, refer to [Storage schema](https://grafana.com/docs/loki//operations/storage/schema/). For instance, this is what it looks like when migrating from BoltDB with v11 schema to TSDB with v13 schema starting 2023-07-01: ```yaml schema_config: configs: - from: 2019-07-01 store: boltdb-shipper object_store: filesystem schema: v11 index: prefix: index_ period: 24h - from: 2023-07-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ``` For all data ingested before 2023-07-01, Loki used BoltDB with the v11 schema, and then switched after that point to the more effective TSDB with the v13 schema. This dramatically simplifies upgrading, ensuring it's simple to take advantage of new storage optimizations. These configs should be immutable for as long as you care about retention. ## Upgrading Schemas When a new schema is released and you want to gain the advantages it provides, you can! Loki can transparently query and merge data from across schema boundaries so there is no disruption of service and upgrading is easy. First, you'll want to create a new [period_config](https://grafana.com/docs/loki//configure/#period_config) entry in your [schema_config](https://grafana.com/docs/loki//configure/#schema_config). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn't queried as if it adheres to the new schema. As an example, let's say it's 2023-07-14 and you want to start using the `v13` schema on the 20th: ```yaml schema_config: configs: - from: 2019-07-14 store: tsdb object_store: filesystem schema: v11 index: prefix: index_ period: 24h - from: 2023-07-20 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ``` It's that easy; you just created a new entry starting on the 20th. ## Retention Loki manages retention through the Compactor when using TSDB. When retention is enabled, the Compactor identifies data that falls outside of the configured retention period, removes the corresponding index entries, and deletes the underlying chunk objects asynchronously. For object storage backends (S3, GCS, Azure Blob) Loki no longer relies solely on external time to live (TTL) or bucket lifecycle rules; these may still be used as an additional safeguard, but Loki itself performs retention-driven deletion when configured. When using the filesystem chunk store, Loki does not delete data based on disk usage or free-space conditions. Deletion is determined only by the retention settings, and disk-full scenarios must be handled operationally outside of Loki. Loki also supports targeted deletion at the tenant or stream level. For more information, see the [retention configuration](https://grafana.com/docs/loki//operations/storage/retention/) documentation. ## Examples {{< admonition type="note" >}} Loki uses the Thanos-based object storage clients by default, because the `use_thanos_objstore` setting defaults to `true`. With this default, Loki reads the configuration under `storage_config.object_store` and ignores the legacy client sections such as `storage_config.aws` or `storage_config.gcs`. Each example below shows the Thanos configuration first. The legacy configuration follows it for reference, because the legacy clients are deprecated. To keep using a legacy client, you must set `use_thanos_objstore: false`. To convert an existing configuration to the new format, refer to [Migrate to Thanos storage clients](https://grafana.com/docs/loki//setup/migrate/migrate-storage-clients/). {{< /admonition >}} ### Single machine/local development (tsdb+filesystem) [The repo contains a working example](https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml), you may want to checkout a tag of the repo to make sure you get a compatible example. ### GCP deployment (GCS Single Store) Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#gcs-example): ```yaml storage_config: use_thanos_objstore: true object_store: gcs: bucket_name: service_account: | { "type": "service_account", ... } tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h ``` The same deployment using the deprecated GCS client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h gcs: bucket_name: service_account: | { "type": "service_account", ... } schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h ``` `service_account` should contain JSON from either a GCP Console `client_credentials.json` file or a GCP service account key. If this value is blank, most services will fall back to GCP's Application Default Credentials (ADC) strategy. For more information about ADC, refer to [How Application Default Credentials works](https://cloud.google.com/docs/authentication/application-default-credentials). The [pre-defined `storage.objectUser` role](https://cloud.google.com/storage/docs/access-control/iam-roles) (or a custom role modeled after it) contains sufficient permissions for Loki to operate. {{< admonition type="note" >}} GCP recommends [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) instead of a service account key. {{< /admonition >}} ### AWS deployment (S3 Single Store) Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#s3-example): ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # You can either declare the access key and secret in the config or # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, # which will be picked up by the AWS SDK. access_key_id: secret_access_key: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` The Thanos-based client supports one bucket only. If you previously used `bucketnames` with several buckets, you must consolidate to a single bucket. If you don't wish to hard-code S3 credentials, you can use an EC2 instance role instead. Leave `access_key_id` and `secret_access_key` unset. The client then looks for credentials in environment variables, in the AWS credentials file, and finally in the EC2 instance metadata: ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: endpoint: s3..amazonaws.com region: ``` The same deployment using the deprecated S3 client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h aws: s3: s3://:@ bucketnames: , schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` To use an EC2 instance role with the deprecated client, change the `storage_config` section: ```yaml storage_config: use_thanos_objstore: false aws: s3: s3:// bucketnames: , ``` The role should have a policy with the following permissions attached. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "LokiStorage", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::" ] }, "Action": [ "s3:ListBucket", "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` **To setup an S3 bucket and an IAM role and policy:** This guide assumes a provisioned EKS cluster. 1. Checkout the Loki repository and navigate to [production/terraform/modules/s3](https://github.com/grafana/loki/tree/main/production/terraform/modules/s3). 2. Initialize Terraform `terraform init`. 3. Export the AWS profile and region if not done so: ```bash export AWS_PROFILE= export AWS_REGION= ``` 4. Save the OIDC provider in an environment variable: ```bash oidc_provider=$(aws eks describe-cluster --name --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///") ``` See the [IAM OIDC provider guide](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for a guide for creating a provider. 5. Apply the Terraform module `terraform -var region="$AWS_REGION" -var cluster_name= -var oidc_id="$oidc_provider"` Note, the bucket name defaults to `loki-data` but can be changed via the `bucket_name` variable. ### Azure deployment (Azure Blob Storage Single Store) #### Using account name and key Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#azure-example): ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: true object_store: azure: # Your Azure storage account name account_name: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: # For the account key, see https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal # If you leave the account key unset, Loki uses an Azure managed identity instead. account_key: # Set this to use a user assigned managed identity. If you leave it empty, # Loki uses the system assigned identity. user_assigned_id: # Configure this if you use a private Azure cloud, such as Azure Stack Hub. # Loki composes the storage URL as https://account_name.endpoint_suffix/container_name/blob_name endpoint_suffix: # If `connection_string` is set, the `account_name` and `endpoint_suffix` values are not used. # Use this instead of `account_key` to authenticate with a SAS token, or to use the Azurite emulator. connection_string: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h ``` The same deployment using the deprecated Azure Blob Storage client: ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: false # required to use the deprecated client azure: # Your Azure storage account name account_name: # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal account_key: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: use_managed_identity: # Providing a user assigned ID will override use_managed_identity user_assigned_id: request_timeout: 0 # Configure this if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name endpoint_suffix: # If `connection_string` is set, the values of `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via a SAS token. Or if you use the Azurite emulator. connection_string: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h filesystem: directory: /loki/chunks ``` #### Using a service principal The [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#azure-example) has no service principal settings in `storage_config`. Instead, pass the credentials in the standard environment variables read by the [Azure Identity Client Module for Go](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity), which are `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_CLIENT_SECRET`. Leave `account_key` unset so that Loki uses those credentials. You must still set `account_name`, because Loki uses it to build the storage URL. ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: true object_store: azure: account_name: container_name: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h ``` The same deployment using the deprecated Azure Blob Storage client: ```yaml schema_config: configs: - from: "2020-12-11" index: period: 24h prefix: index_ object_store: azure schema: v13 store: tsdb storage_config: use_thanos_objstore: false # required to use the deprecated client azure: use_service_principal: true # Azure tenant ID used to authenticate through Azure OAuth tenant_id : # Azure Service Principal ID client_id: # Azure Service Principal secret key client_secret: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: request_timeout: 0 tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h filesystem: directory: /loki/chunks ``` ### IBM Deployment (COS Single Store) {{< admonition type="note" >}} The Thanos-based object store client does not support IBM COS. Because `use_thanos_objstore` defaults to `true`, you must set it to `false` to use COS. If you leave the default in place, Loki fails to start with the error `unrecognized object_store type cos`. For the list of backends that the Thanos-based client supports, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config). {{< /admonition >}} ```yaml schema_config: configs: - from: 2020-10-01 index: period: 24h prefix: loki_index_ object_store: cos schema: v13 store: tsdb storage_config: use_thanos_objstore: false # COS is not supported by the Thanos-based client tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cos: bucketnames: , endpoint: api_key: region: service_instance_id: auth_endpoint: ``` ### On premise deployment (MinIO Single Store) You configure MinIO by using the S3 settings, because MinIO implements the S3 API. Configuration using the [Thanos-based object store client](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/#minio-s3-compatible-example): ```yaml storage_config: use_thanos_objstore: true object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true # set to false if MinIO is served over https bucket_lookup_type: path # MinIO's equivalent of s3forcepathstyle tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` The same deployment using the deprecated S3 client: ```yaml storage_config: use_thanos_objstore: false # required to use the deprecated client aws: # Note: use a fully qualified domain name (fqdn), like localhost. # full example: http://loki:supersecret@localhost.:9000 s3: http(s)://:@: s3forcepathstyle: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h ``` # Usage statistics Loki sends anonymous usage statistics to Grafana Labs. This reporting is enabled by default and helps the Loki team understand aggregate usage patterns. Reports are sent to `https://stats.grafana.org/loki-usage-report` by default. ## What Loki collects Loki reports aggregate operational metadata, including: - An anonymous cluster identifier. - Build and runtime metadata, such as version, OS, architecture, and Go version. - Deployment metadata, such as the configured Loki target. - Go runtime metrics, such as memory and goroutine counts. - Aggregate product metrics, such as ingestion, query throughput, and chunk flush statistics. - Storage and compactor configuration metadata, such as schema, storage backend type, and retention settings. Loki does not report log payloads, label values, tenant identifiers, or query strings. You can view the code in [stats.go](https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go) ## Disable usage statistics reporting To disable reporting in the Loki config file, set `analytics.reporting_enabled` to `false`: ```yaml analytics: reporting_enabled: false ``` To disable reporting with a CLI flag, start Loki with: ```sh -reporting.enabled=false ``` To disable reporting with the Helm chart, set the following values: ```yaml loki: analytics: reporting_enabled: false ``` For all analytics configuration options, refer to the [analytics configuration reference](../#analytics). # Legacy configuration examples These examples configure Loki's legacy, non-default storage clients. Since Loki 4.0, Loki uses the Thanos-based storage clients by default (the `use_thanos_objstore` setting defaults to `true`). For the current default storage client configuration, refer to [Configuration examples for using Thanos-based storage clients](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/). ## 1-Local-Configuration-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by the filesystem. # The index will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default filesystem: directory: /tmp/loki/chunks ``` ## 2-S3-Cluster-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a s3-compatible API # like MinIO for storage. # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache aws: s3: s3://access_key:secret_access_key@custom_endpoint/bucket_name s3forcepathstyle: true ``` ## 3-S3-Without-Credentials-Snippet.yaml ```yaml # If you don't wish to hard-code S3 credentials you can also configure an EC2 # instance role by changing the `storage_config` section. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: s3: s3://region/bucket_name ``` ## 4-GCS-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a GCS. # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache gcs: bucket_name: replace_by_your_bucked_name ``` ## 5-BOS-Example.yaml ```yaml # This is a partial configuration to deploy Loki backed by Baidu Object Storage (BOS). # The index will be shipped to the storage via tsdb-shipper. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: bos schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default bos: bucket_name: bucket_name_1 endpoint: bj.bcebos.com access_key_id: access_key_id secret_access_key: secret_access_key ``` ## 6-Compactor-Snippet.yaml ```yaml # This partial configuration sets the compactor to use S3 and run the compaction every 5 minutes. # Downloaded index files for compaction are stored in /loki/compactor. compactor: working_directory: /tmp/loki/compactor compaction_interval: 5m ``` ## 7-Schema-Migration-Snippet.yaml ```yaml common: path_prefix: /tmp/loki schema_config: configs: # Starting from 2018-04-15 Loki should store indexes on BoltDB with the v11 schema # using daily periodic tables and chunks on filesystem. # The index tables will be prefixed with "index_". # NOTE: `store: boltdb-shipper` is deprecated and scheduled for removal in Loki 4.0. # This entry is kept only to illustrate migrating an existing boltdb-shipper period to tsdb. - from: "2018-04-15" store: boltdb-shipper object_store: filesystem schema: v11 index: period: 24h prefix: index_ # Starting from 2023-6-15 Loki should store indexes on TSDB with the v13 schema # using daily periodic tables and chunks on AWS S3. - from: "2023-06-15" store: tsdb object_store: s3 schema: v13 index: period: 24h prefix: index_ ``` ## 8-alibaba-cloud-storage-Snippet.yaml ```yaml # This partial configuration uses Alibaba for chunk storage. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: alibabacloud schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache alibabacloud: bucket: # Alibabacloud region name. # Required when signature_version=v4. # region: endpoint: access_key_id: secret_access_key: # Specify the RAM role name of the ECS instance. ECS RAM role authentication is used only # when neither access_key_id nor secret_access_key is configured and requires signature_version=v4. # If not set, the role name will be automatically retrieved from the ECS instance metadata. # ram_role_name: # The signature version to use for authenticating against OSS. Supported values are: v1, v4. # ECS RAM role authentication requires signature_version=v4. # signature_version: ``` ## 9-S3-With-SSE-KMS-Snippet.yaml ```yaml # This partial configuration uses S3 for chunk storage and a KMS CMK for encryption. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: s3: s3://access_key:secret_access_key@region/bucket_name sse: type: SSE-KMS kms_key_id: 1234abcd-12ab-34cd-56ef-1234567890ab ``` ## 10-Expanded-S3-Snippet.yaml ```yaml # S3 configuration supports an expanded configuration. # Either an `s3` endpoint URL can be used, or an expanded configuration can be used. storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default aws: bucketnames: bucket_name1, bucket_name2 endpoint: s3.endpoint.com region: s3_region access_key_id: s3_access_key_id secret_access_key: s3_secret_access_key insecure: false http_config: idle_conn_timeout: 90s response_header_timeout: 0s insecure_skip_verify: false s3forcepathstyle: true ``` ## 11-COS-HMAC-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. HMAC will be used for authenticating with COS. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: access_key_id: secret_access_key: ``` ## 12-COS-APIKey-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. APIKey will be used for authenticating with COS. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: api_key: service_instance_id: auth_endpoint: ``` ## 13-COS-Trusted-Profile-Example.yaml ```yaml # This partial configuration uses IBM Cloud Object Storage (COS) for chunk storage. # A trusted profile will be used for authenticating with COS. We can either pass # the trusted profile name or trusted profile ID along with the compute resource token file. # If we pass both trusted profile name and trusted profile ID it should be of # the same trusted profile. # In order to use trusted profile authentication we need to follow an additional step to create a trusted profile. # For more details about creating a trusted profile, see https://cloud.ibm.com/docs/account?topic=account-create-trusted-profile&interface=ui. common: path_prefix: /tmp/loki schema_config: configs: - from: 2020-10-01 store: tsdb object_store: cos schema: v13 index: period: 24h prefix: index_ storage_config: use_thanos_objstore: false # COS is not yet supported by Thanos storage client cos: bucketnames: endpoint: region: auth_endpoint: cr_token_file_path: trusted_profile_name: # You can also use trusted_profile_id instead of trusted_profile_name ``` ## 15-Memberlist-Ring-Snippet.yaml ```yaml # This partial configuration uses memberlist for the ring. common: ring: kvstore: store: memberlist replication_factor: 1 path_prefix: /loki memberlist: join_members: # You can use a headless k8s service for all distributor, ingester and querier components. - loki-gossip-ring.loki.svc.cluster.local:7946 # :7946 is the default memberlist port. ``` ## 16-Azure-Account-Name-Example.yaml ```yaml # This partial configuration uses Azure for chunk storage storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default azure: # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal account_key: account_name: container_name: use_managed_identity: false # Providing a user assigned ID will override use_managed_identity #user_assigned_id: request_timeout: 0 # Configure `endpoint_suffix` if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name #endpoint_suffix: # If `connection_string` is set, the `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via an SAS token. Or if you use the Azurite emulator. #connection_string: ``` ## 17-Azure-Service-Principal-Example.yaml ```yaml # This partial configuration uses Azure for chunk storage and a service principal for authentication storage_config: use_thanos_objstore: false # legacy storage client; see the Thanos storage examples for the current default azure: use_service_principal: true # Azure tenant ID used to authenticate through Azure OAuth tenant_id: # Azure Service Principal ID client_id: # Azure Service Principal secret key client_secret: # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers container_name: request_timeout: 0 ``` ## 18-Thanos-GCS-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by GCS, using the # Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: gcs: bucket_name: ``` ## 19-Thanos-S3-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by AWS S3, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # Leave the access key and secret unset to use an EC2 instance role, or # the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. access_key_id: secret_access_key: ``` ## 20-Thanos-Azure-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by Azure Blob Storage, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: azure schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: azure: account_name: account_key: container_name: ``` ## 21-Thanos-MinIO-Example.yaml ```yaml # This is a complete configuration to deploy Loki backed by a MinIO cluster, # using the Thanos-based object store client (storage_config.object_store). # Index files will be written locally at /loki/index and, eventually, will be shipped to the storage via tsdb-shipper. auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-05-15 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true bucket_lookup_type: path ``` # Query frontend example ## Before you begin This is a general-purpose example. It contains placeholder variables in the format ``. Replace them with values for your environment. ## Use case Many users start by running Grafana Loki as a single binary to simplify deployment. As usage grows and the deployment becomes more production-ready, performance bottlenecks can appear on the read path. Queries that ran quickly on small data sets may slow down or fail on larger ones. Adding more queriers can help in some cases. However, when a single query is too large for one querier to handle, you need the query frontend. ### Parallelization One of the most important functions of the query frontend is the ability to split larger queries into smaller ones, execute them in parallel, and combine the results. The `querier.split-queries-by-interval` flag (or the YAML config `limits_config.split_queries_by_interval`) controls how the frontend splits queries. For example, with this value set to `1h`, the frontend splits a 24-hour query into 24 one-hour queries, distributes them to the queriers, and collects the results. This approach has two benefits in production environments: it lets you run larger queries through aggregation, and it distributes work evenly across queriers. ## Kubernetes Deployment ### ConfigMap Use this ConfigMap to enable query parallelization and caching with the query-frontend component. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: loki-frontend namespace: data: config.yaml: | # Disable the requirement that every request to Loki has a # X-Scope-OrgID header. `fake` will be substituted in instead. auth_enabled: false server: http_listen_port: 3100 query_range: # make queries more cache-able by aligning them with their step intervals align_queries_with_step: true max_retries: 5 cache_results: true results_cache: cache: # Use the embedded cache embedded_cache: enabled: true max_size_mb: 100 ttl: 24h limits_config: max_cache_freshness_per_query: '10m' # parallelize queries in 15min intervals split_queries_by_interval: 15m frontend: log_queries_longer_than: 5s downstream_url: http://querier..svc.cluster.local:3100 compress_responses: true ``` ### Frontend Service ```yaml apiVersion: v1 kind: Service metadata: labels: name: query-frontend name: query-frontend namespace: spec: ports: - name: query-frontend-http port: 3100 protocol: TCP targetPort: 3100 selector: name: query-frontend sessionAffinity: None clusterIP: None publishNotReadyAddresses: true ``` ### Frontend Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: name: query-frontend name: query-frontend namespace: spec: minReadySeconds: 10 replicas: 2 selector: matchLabels: name: query-frontend template: metadata: labels: name: query-frontend spec: containers: - args: - -config.file=/etc/loki/config.yaml - -log.level=debug - -target=query-frontend image: grafana/loki: imagePullPolicy: Always name: query-frontend ports: - containerPort: 3100 name: http protocol: TCP resources: limits: memory: 1200Mi requests: cpu: '2' memory: 600Mi volumeMounts: - mountPath: /etc/loki name: loki-frontend restartPolicy: Always terminationGracePeriodSeconds: 30 volumes: - configMap: defaultMode: 420 name: loki-frontend name: loki-frontend ``` ### Grafana After you deploy these resources, point your Grafana data source to the new frontend service. The service is available within the cluster at `http://query-frontend..svc.cluster.local:3100`. ### gRPC mode (pull model) The query frontend operates in one of two modes: - **Push mode:** Specify `--frontend.downstream-url` or its YAML equivalent `frontend.downstream_url`. The frontend proxies requests over HTTP to the specified URL. - **Pull mode:** If you do not set `--frontend.downstream-url` or its YAML equivalent `frontend.downstream_url`, the query frontend operates as a pull service. The frontend creates per-tenant queues. Downstream queriers pull queries from these queues using gRPC. To use pull mode, configure queriers with `-querier.frontend-address` or its YAML equivalent `frontend_worker.frontend_address`. Set `clusterIP: None` on the query frontend pull service. This setting causes DNS to resolve to individual query frontend pod IP addresses instead of the service IP. Set `publishNotReadyAddresses: true` on the query frontend pull service. This setting prevents a race condition where a querier needs the query frontend address before the frontend is ready. # Configuration examples for using Thanos-based storage clients Use these examples as a starting point for configuring [Thanos-based object storage clients](https://grafana.com/docs/loki//configure/#thanos_object_store_config) in Grafana Loki. Each example is a complete configuration you can adapt and run, not just a snippet. ## GCS example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: gcs schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space object_store: gcs: bucket_name: # JSON either from a Google Developers Console client_credentials.json file, # or a Google Developers service account key. Needs to be valid JSON, not a # filesystem path. If empty, fallback to Google default logic: # 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS # environment variable. For workload identity federation, refer to # https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation # on how to generate the JSON configuration file for on-prem/non-Google cloud # platforms. # 2. A JSON file in a location known to the gcloud command-line tool: # $HOME/.config/gcloud/application_default_credentials.json. # 3. On Google Compute Engine it fetches credentials from the metadata server. service_account: |- { "type": "service_account", "project_id": "project", "private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666", "private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n", "client_email": "project@example.iam.gserviceaccount.com", "client_id": "123456789012345678901", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/project@example.iam.gserviceaccount.com" } ``` ## S3 example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: s3: bucket_name: # The endpoint is required. For AWS, use the regional S3 endpoint. endpoint: s3..amazonaws.com region: # You can either declare the access key and secret in the config or # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by the AWS SDK. access_key_id: secret_access_key: ``` ## Azure example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: azure schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: azure: account_name: account_key: ${SECRET_ACCESS_KEY} # loki expands environment variables container_name: ``` ## MinIO (S3-compatible) example MinIO and other S3-compatible object stores use the same `object_store.s3` client as AWS S3. Set `bucket_lookup_type: path`, which is the equivalent of the deprecated `s3forcepathstyle` option. Set `insecure: true` if MinIO is served over plain HTTP. ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h object_store: s3: bucket_name: # Use a fully qualified domain name (fqdn), like localhost, without a scheme. endpoint: : access_key_id: secret_access_key: insecure: true bucket_lookup_type: path ``` ## Filesystem example ```yaml auth_enabled: false server: http_listen_port: 3100 common: ring: instance_addr: 127.0.0.1 kvstore: store: inmemory replication_factor: 1 path_prefix: /tmp/loki schema_config: configs: - from: 2020-07-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h storage_config: use_thanos_objstore: true object_store: filesystem: dir: /tmp/loki/chunks ``` # Get Started # Loki architecture Grafana Loki has a microservices-based architecture and is designed to run as a horizontally scalable, distributed system. The system has multiple components that can run separately and in parallel. The Grafana Loki design compiles the code for all components into a single binary or Docker image. The `-target` command-line flag controls which component(s) that binary will behave as. To get started easily, run Grafana Loki in "single binary" mode with all components running simultaneously in one process, or in "simple scalable deployment" mode, which groups components into read, write, and backend parts. Grafana Loki is designed to easily redeploy a cluster under a different mode as your needs change, with no configuration changes or minimal configuration changes. For more information, refer to [Deployment modes](../deployment-modes/) and [Components](../components/). ![Loki components](../loki_architecture_components.svg "Loki components") ## Storage Loki stores all data in a single object storage backend, such as Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), Azure Blob Storage, among others. This mode uses an adapter called **index shipper** (or short **shipper**) to store index (TSDB) files the same way we store chunk files in object storage. This mode of operation became generally available with Loki 2.0 and is fast, cost-effective, and simple. It is where all current and future development lies. Prior to 2.0, Loki had different storage backends for indexes and chunks. For more information, refer to [Legacy storage](../../operations/storage/legacy-storage/). ### Data format Grafana Loki has two main file types: **index** and **chunks**. - The [**index**](#index-format) is a table of contents of where to find logs for a specific set of labels. - The [**chunk**](#chunk-format) is a container for log entries for a specific set of labels. ![Loki data format: chunks and indexes](../chunks_diagram.png) The diagram above shows the high-level overview of the data that is stored in the chunk and data that is stored in the index. #### Index format There is one index format that is currently supported as single store with index shipper: - [TSDB](../../operations/storage/tsdb/) Time Series Database (or short TSDB) is an [index format](https://github.com/prometheus/prometheus/blob/main/tsdb/docs/format/index.md) originally developed by the maintainers of [Prometheus](https://github.com/prometheus/prometheus) for time series (metric) data. It is extensible and has many advantages over the deprecated BoltDB index. New storage features in Loki are solely available when using TSDB. #### Chunk format A chunk is a container for log lines of a stream (unique set of labels) of a specific time range. The following ASCII diagram describes the chunk format in detail. ``` ---------------------------------------------------------------------------- | | | | | MagicNumber(4b) | version(1b) | encoding (1b) | | | | | ---------------------------------------------------------------------------- | #structuredMetadata (uvarint) | ---------------------------------------------------------------------------- | len(label-1) (uvarint) | label-1 (bytes) | ---------------------------------------------------------------------------- | len(label-2) (uvarint) | label-2 (bytes) | ---------------------------------------------------------------------------- | len(label-n) (uvarint) | label-n (bytes) | ---------------------------------------------------------------------------- | checksum(from #structuredMetadata) | ---------------------------------------------------------------------------- | block-1 bytes | checksum (4b) | ---------------------------------------------------------------------------- | block-2 bytes | checksum (4b) | ---------------------------------------------------------------------------- | block-n bytes | checksum (4b) | ---------------------------------------------------------------------------- | #blocks (uvarint) | ---------------------------------------------------------------------------- | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | ---------------------------------------------------------------------------- | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | ---------------------------------------------------------------------------- | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | ---------------------------------------------------------------------------- | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | ---------------------------------------------------------------------------- | checksum(from #blocks) | ---------------------------------------------------------------------------- | #structuredMetadata len (uvarint) | #structuredMetadata offset (uvarint) | ---------------------------------------------------------------------------- | #blocks len (uvarint) | #blocks offset (uvarint) | ---------------------------------------------------------------------------- ``` `mint` and `maxt` describe the minimum and maximum Unix nanosecond timestamp, respectively. The `structuredMetadata` section stores non-repeated strings. It is used to store label names and label values from [structured metadata](../labels/structured-metadata/). Note that the labels strings and lengths within the `structuredMetadata` section are stored compressed. #### Block format A block is comprised of a series of entries, each of which is an individual log line. Note that the bytes of a block are stored compressed. The following is their form when uncompressed: ``` ----------------------------------------------------------------------------------------------------------------------------------------------- | ts (varint) | len (uvarint) | log-1 bytes | len(from #symbols) | #symbols (uvarint) | symbol-1 (uvarint) | symbol-n*2 (uvarint) | ----------------------------------------------------------------------------------------------------------------------------------------------- | ts (varint) | len (uvarint) | log-2 bytes | len(from #symbols) | #symbols (uvarint) | symbol-1 (uvarint) | symbol-n*2 (uvarint) | ----------------------------------------------------------------------------------------------------------------------------------------------- | ts (varint) | len (uvarint) | log-3 bytes | len(from #symbols) | #symbols (uvarint) | symbol-1 (uvarint) | symbol-n*2 (uvarint) | ----------------------------------------------------------------------------------------------------------------------------------------------- | ts (varint) | len (uvarint) | log-n bytes | len(from #symbols) | #symbols (uvarint) | symbol-1 (uvarint) | symbol-n*2 (uvarint) | ----------------------------------------------------------------------------------------------------------------------------------------------- ``` `ts` is the Unix nanosecond timestamp of the logs, while `len` is the length in bytes of the log entry. Symbols store references to the actual strings containing label names and values in the `structuredMetadata` section of the chunk. ## Write path On a high level, the write path in Loki works as follows: 1. The distributor receives an HTTP POST request with streams and log lines. 1. The distributor hashes each stream contained in the request so it can determine the ingester instance to which it needs to be sent based on the information from the consistent hash ring. 1. The distributor sends each stream to the appropriate ingester and its replicas (based on the configured replication factor). 1. The ingester receives the stream with log lines and creates a chunk or appends to an existing chunk for the stream's data. A chunk is unique per tenant and per label set. 1. The ingester acknowledges the write. 1. The distributor waits for a majority (quorum) of the ingesters to acknowledge their writes. 1. The distributor responds with a success (2xx status code) in case it received at least a quorum of acknowledged writes. or with an error (4xx or 5xx status code) in case write operations failed. Refer to [Components](../components/) for a more detailed description of the components involved in the write path. ### Kafka-based write path (experimental) {{< admonition type="warning" >}} Kafka-based ingestion is an [experimental feature](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided. {{< /admonition >}} Loki can optionally use Apache Kafka, or a Kafka-protocol-compatible system, as a durable buffer on the write path. When you enable Kafka-based ingestion, the write path changes: the distributor writes log streams as Kafka records to a configured topic, instead of, or in addition to, pushing them directly to ingesters over gRPC. Each ingester consumes from exactly one Kafka partition, determined by its position in the partition ring. This decouples ingestion durability from ingester availability, because Kafka buffers writes while an ingester is restarting or unavailable. Enabling Kafka-based ingestion primarily changes the write path. The read path still works the same way, but the querier must be configured to look up ingesters using the partition ring instead of the classic hash ring. For full details, including configuration, monitoring, and migration guidance, refer to [Kafka-based ingestion](https://grafana.com/docs/loki//operations/kafka/). ## Read path On a high level, the read path in Loki works as follows: 1. The query frontend receives an HTTP GET request with a LogQL query. 1. The query frontend splits the query into sub-queries and passes them to the query scheduler. 1. The querier pulls sub-queries from the scheduler. 1. The querier passes the query to all ingesters for in-memory data. 1. The ingesters return in-memory data matching the query, if any. 1. The querier lazily loads data from the backing store and runs the query against it if ingesters returned no or insufficient data. 1. The querier iterates over all received data and deduplicates, returning the result of the sub-query to the query frontend. 1. The query frontend waits for all sub-queries of a query to be finished and returned by the queriers. 1. The query frontend merges the individual results into a final result and return it to the client. Refer to [Components](../components/) for a more detailed description of the components involved in the read path. ## Multi-tenancy All data, both in memory and in long-term storage, may be partitioned by a tenant ID, pulled from the `X-Scope-OrgID` HTTP header in the request when Grafana Loki is running in multi-tenant mode. When Loki is **not** in multi-tenant mode, the header is ignored and the tenant ID is set to `fake`, which will appear in the index and in stored chunks. # Loki components {{< youtube id="_hv4i84Z68s" >}} Loki is a modular system that contains many components that can either be run together (in "single binary" mode with target `all`), in logical groups (in "simple scalable deployment" mode with targets `read`, `write`, `backend`), or individually (in "microservice" mode). For more information see [Deployment modes](https://grafana.com/docs/loki//get-started/deployment-modes/). | Component | _individual_ | `all` | `read` | `write` | `backend` | |----------------------------------------------------|--------------| - | - | - | - | | [Distributor](#distributor) | x | x | | x | | | [Ingester](#ingester) | x | x | | x | | | [Query Frontend](#query-frontend) | x | x | x | | | | [Query Scheduler](#query-scheduler) | x | x | | | x | | [Querier](#querier) | x | x | x | | | | [Index Gateway](#index-gateway) | x | | | | x | | [Compactor](#compactor) | x | x | | | x | | [Ruler](#ruler) | x | x | | | x | | [Pattern ingester](#pattern-ingester) | x | x | | x | | | [Bloom Planner (Deprecated)](#bloom-planner) | x | | | | x | | [Bloom Builder (Deprecated)](#bloom-builder) | x | | | | x | | [Bloom Gateway (Deprecated)](#bloom-gateway) | x | | | | x | This page describes the responsibilities of each of these components. {{< admonition type="note" >}} Kafka is infrastructure that Loki connects to, not a Loki component binary, so it doesn't have its own row in the table above. When [Kafka-based ingestion](#kafka-based-ingestion-experimental) is enabled, the Distributor and Ingester rows in the table above have optional Kafka write and read behavior. Refer to [Kafka-based ingestion](#kafka-based-ingestion-experimental) for details. {{< /admonition >}} ## Distributor The **distributor** service is responsible for handling incoming push requests from clients. It's the first step in the write path for log data. Once the distributor receives a set of streams in an HTTP request, each stream is validated for correctness and to ensure that it is within the configured tenant (or global) limits. Each valid stream is then sent to `n` [ingesters](#ingester) in parallel, where `n` is the [replication factor](#replication-factor) for data. The distributor determines the ingesters to which it sends a stream to using [consistent hashing](#hashing). A load balancer must sit in front of the distributor to properly balance incoming traffic to them. In Kubernetes, the service load balancer provides this service. The distributor is a stateless component. This makes it easy to scale and offload as much work as possible from the ingesters, which are the most critical component on the write path. The ability to independently scale these validation operations means that Loki can also protect itself against denial of service attacks that could otherwise overload the ingesters. It also allows us to fan-out writes according to the [replication factor](#replication-factor). ### Validation The first step the distributor takes is to ensure that all incoming data is according to specification. This includes things like checking that the labels are valid Prometheus labels as well as ensuring the timestamps aren't too old or too new or the log lines aren't too long. ### Preprocessing Currently, the only way the distributor mutates incoming data is by normalizing labels. What this means is making `{foo="bar", bazz="buzz"}` equivalent to `{bazz="buzz", foo="bar"}`, or in other words, sorting the labels. This allows Loki to cache and hash them deterministically. ### Rate limiting The distributor can also rate-limit incoming logs based on the maximum data ingest rate per tenant. It does this by checking a per-tenant limit and dividing it by the current number of distributors. This allows the rate limit to be specified per tenant at the cluster level and enables us to scale the distributors up or down and have the per-distributor limit adjust accordingly. For instance, say we have 10 distributors and tenant A has a 10MB rate limit. Each distributor will allow up to 1MB/s before limiting. Now, say another large tenant joins the cluster and we need to spin up 10 more distributors. The now 20 distributors will adjust their rate limits for tenant A to `(10MB / 20 distributors) = 500KB/s`. This is how global limits allow much simpler and safer operation of the Loki cluster. {{< admonition type="note" >}} The distributor uses the `ring` component under the hood to register itself amongst its peers and get the total number of active distributors. This is a different "key" than the ingesters use in the ring and comes from the distributor's own [ring configuration](https://grafana.com/docs/loki//configure/#distributor). {{< /admonition >}} ### Forwarding Once the distributor has performed all of its validation duties, it forwards data to the ingester component which is ultimately responsible for acknowledging the write operation. #### Replication factor In order to mitigate the chance of _losing_ data on any single ingester, the distributor will forward writes to a _replication factor_ of them. Generally, the replication factor is `3`. Replication allows for ingester restarts and rollouts without failing writes and adds additional protection from data loss for some scenarios. Loosely, for each label set (called a _stream_) that is pushed to a distributor, it will hash the labels and use the resulting value to look up `replication_factor` ingesters in the `ring` (which is a subcomponent that exposes a [distributed hash table](https://en.wikipedia.org/wiki/Distributed_hash_table)). It will then try to write the same data to all of them. This will generate an error if less than a _quorum_ of writes succeeds. A quorum is defined as `floor( replication_factor / 2 ) + 1`. So, for our `replication_factor` of `3`, we require that two writes succeed. If less than two writes succeed, the distributor returns an error and the write operation will be retried. {{< admonition type="caution" >}} If a write is acknowledged by 2 out of 3 ingesters, we can tolerate the loss of one ingester but not two, as this would result in data loss. {{< /admonition >}} The replication factor is not the only thing that prevents data loss, though, and its main purpose is to allow writes to continue uninterrupted during rollouts and restarts. The [ingester component](#ingester) now includes a [write ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) (WAL) which persists incoming writes to disk to ensures they are not lost as long as the disk isn't corrupted. The complementary nature of the replication factor and WAL ensures data isn't lost unless there are significant failures in both mechanisms (that is, multiple ingesters die and lose/corrupt their disks). ### Hashing Distributors use consistent hashing in conjunction with a configurable replication factor to determine which instances of the ingester service should receive a given stream. A stream is a set of logs associated to a tenant and a unique label set. The stream is hashed using both the tenant ID and the label set and then the hash is used to find the ingesters to send the stream to. A hash ring, maintained by peer-to-peer communication using the [Memberlist](https://github.com/hashicorp/memberlist) protocol, or stored in a Key-Value store such as [Consul](https://www.consul.io) is used to achieve consistent hashing; all [ingesters](#ingester) register themselves into the hash ring with a set of tokens they own. Each token is a random unsigned 32-bit number. Along with a set of tokens, ingesters register their state into the hash ring. The state `JOINING`, and `ACTIVE` may all receive write requests, while `ACTIVE` and `LEAVING` ingesters may receive read requests. When doing a hash lookup, distributors only use tokens for ingesters who are in the appropriate state for the request. To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the hash of the stream. When the replication factor is larger than 1, the next subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result. The effect of this hash setup is that each token that an ingester owns is responsible for a range of hashes. If there are three tokens with values 0, 25, and 50, then a hash of 3 would be given to the ingester that owns the token 25; the ingester owning token 25 is responsible for the hash range of 1-25. ### Quorum consistency Since all distributors share access to the same hash ring, write requests can be sent to any distributor. To ensure consistent query results, Loki uses [Dynamo-style](https://www.cs.princeton.edu/courses/archive/fall15/cos518/studpres/dynamo.pdf) quorum consistency on reads and writes. This means that the distributor will wait for a positive response of at least one half plus one of the ingesters to send the sample to before responding to the client that initiated the send. {{< admonition type="note" >}} When [Kafka-based ingestion](#kafka-based-ingestion-experimental) is enabled, the distributor can also, or instead, write streams to Kafka. Refer to [Kafka-based ingestion](#kafka-based-ingestion-experimental) for details. {{< /admonition >}} ## Ingester The **ingester** service is responsible for persisting data and shipping it to long-term storage (Amazon Simple Storage Service, Google Cloud Storage, Azure Blob Storage, etc.) on the write path, and returning recently ingested, in-memory log data for queries on the read path. Ingesters contain a _lifecycler_ which manages the lifecycle of an ingester in the hash ring. Each ingester has a state of either `PENDING`, `JOINING`, `ACTIVE`, `LEAVING`, or `UNHEALTHY`: 1. `PENDING` is an Ingester's state when it is waiting for a [handoff](#handoff) from another ingester that is `LEAVING`. This only applies for legacy deployment modes. {{< admonition type="note" >}} Handoff is a deprecated behavior mainly used in stateless deployments of ingesters, which is discouraged. Instead, it's recommended using a stateful deployment model together with the [write ahead log](https://grafana.com/docs/loki//operations/storage/wal/). {{< /admonition >}} 1. `JOINING` is an Ingester's state when it is currently inserting its tokens into the ring and initializing itself. It may receive write requests for tokens it owns. 1. `ACTIVE` is an Ingester's state when it is fully initialized. It may receive both write and read requests for tokens it owns. 1. `LEAVING` is an Ingester's state when it is shutting down. It may receive read requests for data it still has in memory. 1. `UNHEALTHY` is an Ingester's state when it has failed to heartbeat. `UNHEALTHY` is set by the distributor when it periodically checks the ring. Each log stream that an ingester receives is built up into a set of many "chunks" in memory and flushed to the backing storage backend at a configurable interval. Chunks are compressed and marked as read-only when: 1. The current chunk has reached capacity (a configurable value). 1. Too much time has passed without the current chunk being updated 1. A flush occurs. Whenever a chunk is compressed and marked as read-only, a writable chunk takes its place. If an ingester process crashes or exits abruptly, all the data that has not yet been flushed will be lost. Loki is usually configured to replicate multiple replicas (usually 3) of each log to mitigate this risk. When a flush occurs to a persistent storage provider, the chunk is hashed based on its tenant, labels, and contents. This means that multiple ingesters with the same copy of data will not write the same data to the backing store twice, but if any write failed to one of the replicas, multiple differing chunk objects will be created in the backing store. See [Querier](#querier) for how data is deduplicated. ### Timestamp Ordering Loki is configured to [accept out-of-order writes](https://grafana.com/docs/loki//configure/#accept-out-of-order-writes) by default. When not configured to accept out-of-order writes, the ingester validates that ingested log lines are in order. When an ingester receives a log line that doesn't follow the expected order, the line is rejected and an error is returned to the user. The ingester validates that log lines are received in timestamp-ascending order. Each log has a timestamp that occurs at a later time than the log before it. When the ingester receives a log that does not follow this order, the log line is rejected and an error is returned. Logs from each unique set of labels are built up into "chunks" in memory and then flushed to the backing storage backend. If an ingester process crashes or exits abruptly, all the data that has not yet been flushed could be lost. Loki is usually configured with a [Write Ahead Log](https://grafana.com/docs/loki//operations/storage/wal/) which can be _replayed_ on restart as well as with a `replication_factor` (usually 3) of each log to mitigate this risk. When not configured to accept out-of-order writes, all lines pushed to Loki for a given stream (unique combination of labels) must have a newer timestamp than the line received before it. There are, however, two cases for handling logs for the same stream with identical nanosecond timestamps: 1. If the incoming line exactly matches the previously received line (matching both the previous timestamp and log text), the incoming line will be treated as an exact duplicate and ignored. 2. If the incoming line has the same timestamp as the previous line but different content, the log line is accepted. This means it is possible to have two different log lines for the same timestamp. ### Handoff {{< admonition type="warning" >}} Handoff is deprecated behavior mainly used in stateless deployments of ingesters, which is discouraged. Instead, it's recommended using a stateful deployment model together with the [write ahead log](https://grafana.com/docs/loki/latest/operations/storage/wal/). {{< /admonition >}} By default, when an ingester is shutting down and tries to leave the hash ring, it will wait to see if a new ingester tries to enter before flushing and will try to initiate a handoff. The handoff will transfer all of the tokens and in-memory chunks owned by the leaving ingester to the new ingester. Before joining the hash ring, ingesters will wait in `PENDING` state for a handoff to occur. After a configurable timeout, ingesters in the `PENDING` state that have not received a transfer will join the ring normally, inserting a new set of tokens. This process is used to avoid flushing all chunks when shutting down, which is a slow process. ### Filesystem support Ingesters can flush chunks to the filesystem when Loki uses TSDB as the index store. TSDB ships its index files to object storage, including a filesystem object store, so [queriers](#querier) running in a separate process can read the same data, as long as every process can reach the same directory. This differs from the deprecated, non-shipped BoltDB index store, which only allows one process to hold a lock on the database file at a time. {{< admonition type="note" >}} When [Kafka-based ingestion](#kafka-based-ingestion-experimental) is enabled, ingesters consume log streams from a Kafka partition instead of, or in addition to, receiving them directly from the distributor over gRPC. Refer to [Kafka-based ingestion](#kafka-based-ingestion-experimental) for details. {{< /admonition >}} ## Query frontend The **query frontend** is an **optional service** providing the querier's API endpoints and can be used to accelerate the read path. When the query frontend is in place, incoming query requests should be directed to the query frontend instead of the queriers. The querier service will be still required within the cluster, in order to execute the actual queries. The query frontend internally performs some query adjustments and holds queries in an internal queue. In this setup, queriers act as workers which pull jobs from the queue, execute them, and return them to the query frontend for aggregation. Queriers need to be configured with the query frontend address (via the `-querier.frontend-address` CLI flag) in order to allow them to connect to the query frontends. Query frontends are **stateless**. However, due to how the internal queue works, it's recommended to run a few query frontend replicas to reap the benefit of fair scheduling. Two replicas should suffice in most cases. ### Queueing If no separate [query scheduler](#query-scheduler) component is used, the query frontend will also perform basic query queueing. - Ensure that large queries, that could cause an out-of-memory (OOM) error in the querier, will be retried on failure. This allows administrators to under-provision memory for queries, or optimistically run more small queries in parallel, which helps to reduce the total cost of ownership (TCO). - Prevent multiple large requests from being convoyed on a single querier by distributing them across all queriers using a first-in/first-out queue (FIFO). - Prevent a single tenant from denial-of-service-ing (DOSing) other tenants by fairly scheduling queries between tenants. ### Splitting The query frontend splits larger queries into multiple smaller queries, executing these queries in parallel on downstream queriers and stitching the results back together again. This prevents large (multi-day, etc) queries from causing out of memory issues in a single querier and helps to execute them faster. ### Caching #### Metric queries The query frontend supports caching metric query results and reuses them on subsequent queries. If the cached results are incomplete, the query frontend calculates the required sub-queries and executes them in parallel on downstream queriers. The query frontend can optionally align queries with their step parameter to improve the cacheability of the query results. The result cache is compatible with any Loki caching backend (currently Memcached, Redis, and an in-memory cache). #### Log queries The query frontend also supports caching of log queries in form of a negative cache. This means that instead of caching the log results for quantized time ranges, Loki only caches empty results for quantized time ranges. This is more efficient than caching actual results because log queries are limited (usually 1000 results) and if you have a query over a long time range that matches only a few lines, and you only cache actual results, you'd still need to process a lot of data in addition to the data from the results cache in order to verify that nothing else matches. #### Index stats queries The query frontend caches index stats query results similar to the [metric query](#metric-queries) results. This cache is only applicable when using single store TSDB. #### Log volume queries The query frontend caches log volume query results similar to the [metric query](#metric-queries) results. This cache is only applicable when using single store TSDB. ## Query scheduler The **query scheduler** is an **optional service** providing more [advanced queuing functionality](https://grafana.com/docs/loki//operations/query-fairness/) than the [query frontend](#query-frontend). When using this component in the Loki deployment, query frontend pushes split up queries to the query scheduler which enqueues them in an internal in-memory queue. There is a queue for each tenant to guarantee the query fairness across all tenants. The queriers that connect to the query scheduler act as workers that pull their jobs from the queue, execute them, and return them to the query frontend for aggregation. Queriers therefore need to be configured with the query scheduler address (via the `-querier.scheduler-address` CLI flag, or the `scheduler_address` field in the [`frontend_worker`](https://grafana.com/docs/loki//configure/#frontend_worker) YAML block) in order to allow them to connect to the query scheduler. As an alternative to a static address, query schedulers can register themselves in a hash ring so that queriers and query frontends discover them automatically. For more information, refer to [Scale Loki](https://grafana.com/docs/loki//operations/scalability/#scheduler-discovery-using-a-ring). Query schedulers are **stateless**. However, due to the in-memory queue, it's recommended to run more than one replica to keep the benefit of high availability. Two replicas should suffice in most cases. ## Querier The **querier** service is responsible for executing [Log Query Language (LogQL)](https://grafana.com/docs/loki//query/) queries. The querier can handle HTTP requests from the client directly (in "single binary" mode, or as part of the read path in "simple scalable deployment") or pull subqueries from the query frontend or query scheduler (in "microservice" mode). It fetches log data from both the ingesters and from long-term storage. Queriers query all ingesters for in-memory data before falling back to running the same query against the backend store. Because of the replication factor, it is possible that the querier may receive duplicate data. To resolve this, the querier internally **deduplicates** data that has the same nanosecond timestamp, label set, and log message. ## Index Gateway The **index gateway** service is responsible for handling and serving metadata queries. Metadata queries are queries that look up data from the index. The index gateway is only used by [single store TSDB](https://grafana.com/docs/loki//operations/storage/tsdb/). The query frontend queries the index gateway for the log volume of queries so it can make a decision on how to shard the queries. The queriers query the index gateway for chunk references for a given query so they know which chunks to fetch and query. The index gateway can run in `simple` or `ring` mode. In `simple` mode, each index gateway instance serves all indexes from all tenants. In `ring` mode, index gateways use a consistent hash ring to distribute and shard the indexes per tenant amongst available instances. ## Compactor The **compactor** service is used by [single store TSDB](https://grafana.com/docs/loki//operations/storage/tsdb/), to compact the multiple index files produced by the ingesters and shipped to object storage into single index files per day and tenant. This makes index lookups more efficient. To do so, the compactor downloads the files from object storage in a regular interval, merges them into a single one, uploads the newly created index, and cleans up the old files. Additionally, the compactor is also responsible for [log retention](https://grafana.com/docs/loki//operations/storage/retention/) and [log deletion](https://grafana.com/docs/loki//operations/storage/logs-deletion/). In a Loki deployment, the compactor service is usually run as a single instance. ## Ruler The **ruler** service manages and evaluates rule and/or alert expressions provided in a rule configuration. The rule configuration is stored in object storage (or alternatively on local file system) and can be managed via the ruler API or directly by uploading the files to object storage. Alternatively, the ruler can also delegate rule evaluation to the query frontend. This mode is called remote rule evaluation and is used to gain the advantages of query splitting, query sharding, and caching from the query frontend. When running multiple rulers, they use a consistent hash ring to distribute rule groups amongst available ruler instances. ## Pattern ingester The optional **pattern ingester** component receives log data from the ingesters and scans the logs to detect and aggregate patterns. This can be useful for understanding the structure of your logs at scale. The pattern ingester is used by the pattern feature in Logs Drilldown, which lets you detect similar log lines and add or exclude them from your search. The ingester uses a drain algorithm to identify related logs that share the same pattern, and maintain their counts over time. Patterns consist of a number, a string, and a Loki series identifier. The pattern ingester exposes a query API, so you can fetch detected patterns. This API is used by the Patterns tab in the Grafana Logs Drilldown plugin. This component is disabled by default and must be enabled in your [Loki config file](https://grafana.com/docs/loki/latest/configure/#supported-contents-and-default-values-of-lokiyaml). ## Kafka-based ingestion (experimental) {{< admonition type="warning" >}} This feature is an [experimental feature](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided. {{< /admonition >}} Loki can optionally use Apache Kafka, or a Kafka-protocol-compatible system, as a durable buffer on the write path, instead of, or in addition to, the direct gRPC path between the distributor and the ingester. When you enable Kafka-based ingestion: - The [distributor](#distributor) writes each log stream as one or more records to a Kafka topic. - Each [ingester](#ingester) consumes records from exactly one Kafka partition of that topic. - Ingesters use the **partition ring**, a hash ring similar to the one ingesters use for gRPC writes, to coordinate which ingester owns which partition. Unlike the main hash ring, the partition ring assigns one partition to one ingester at a time, rather than distributing hash ranges across replicas. Because Kafka durably stores records until an ingester consumes them, ingesters can restart or roll out without losing or blocking writes: on restart, an ingester resumes consuming from the last offset it committed to Kafka. Enabling Kafka-based ingestion primarily changes the write path. The read path still works the same way, but the querier must be configured to look up ingesters using the partition ring instead of the classic hash ring. For details, refer to [Kafka-based ingestion](https://grafana.com/docs/loki//operations/kafka/#read-path). For configuration, monitoring, and migration guidance, refer to [Kafka-based ingestion](https://grafana.com/docs/loki//operations/kafka/). ## Bloom Planner {{< admonition type="warning" >}} This feature is deprecated and will be removed in a future release. Engineering and on-call support is not available. No SLA is provided. {{< /admonition >}} The Bloom Planner service is responsible for planning the tasks for blooms creation. It runs as a singleton and provides a queue from which tasks are pulled by the Bloom Builders. The planning runs periodically and takes into account what blooms have already been built for a given day and tenant and what series need to be newly added. This service is also used to apply blooms retention. ## Bloom Builder {{< admonition type="warning" >}} This feature is deprecated and will be removed in a future release. Engineering and on-call support is not available. No SLA is provided. {{< /admonition >}} The Bloom Builder service is responsible for processing the tasks created by the Bloom Planner. The Bloom Builder creates bloom blocks from structured metadata of log entries. The resulting blooms are grouped in bloom blocks spanning multiple series and chunks from a given day. This component also builds metadata files to track which blocks are available for each series and TSDB index file. The service is stateless and horizontally scalable. ## Bloom Gateway {{< admonition type="warning" >}} This feature is deprecated and will be removed in a future release. Engineering and on-call support is not available. No SLA is provided. {{< /admonition >}} The Bloom Gateway service is responsible for handling and serving chunks filtering requests. The index gateway queries the Bloom Gateway when computing chunk references, or when computing shards for a given query. The gateway service takes a list of chunks and a filtering expression and matches them against the blooms, filtering out any chunks that do not match the given label filter expression. The service is horizontally scalable. When running multiple instances, the client (Index Gateway) shards requests across instances based on the hash of the bloom blocks that are referenced. # Loki deployment modes Loki is a distributed system consisting of many microservices. It also has a unique build model where all of those microservices exist within the same binary. You can configure the behavior of the single binary with the `-target` command-line flag to specify which microservices will run on startup. You can further configure each of the components in the `loki.yaml` file. Because Loki decouples the data it stores from the software which ingests and queries it, you can easily redeploy a cluster under a different mode as your needs change, with minimal or no configuration changes. ## Comparison of deployment modes | | [Monolithic](#monolithic-mode) | [HA Monolithic](#ha-monolithic-mode) | [Microservices](#microservices-mode) | |---|---|---|---| | Durability | ✅ | ✅ | ✅ | | High availability | ❌ | ✅ | ✅ | | Separation of execution paths | ❌ | ❌ | ✅ | | Operational complexity | 🟩 low | 🟧 medium| 🟥 high | | Scalability | 🟥 low | 🟧 medium | 🟩 high | ## Monolithic mode Sometimes also referred to as **single binary** deployment. The simplest mode of operation is the monolithic deployment mode. You enable monolithic mode by setting the `-target=all` command line parameter. This mode runs all of Loki’s microservice components inside a single process as a single binary or Docker image. ![monolithic mode diagram](../monolithic-mode.png "Monolithic mode") Monolithic mode is useful for getting started quickly to experiment with Loki, as well as for small read/write volumes of up to approximately 20GB per day. Query parallelization is limited by the number of instances and the setting `max_query_parallelism` which is defined in the `loki.yaml` file. ## HA Monolithic mode Sometimes also referred to as **HA single binary** deployment. This mode replaces the deprecated Simple Scalable Deployment (SSD), which will be removed in Loki 4.0. HA monolithic mode sits between [monolithic mode](#monolithic-mode) and [microservices mode](#microservices-mode): it provides high availability and moderate horizontal scalability without the operational complexity of managing individual microservices. If you need fine-grained, per-component scaling, use microservices mode instead. You can horizontally scale a monolithic mode deployment to more instances by using a shared object store, and by configuring the [`ring` section](https://grafana.com/docs/loki//configure/#common) of the `loki.yaml` file to share cluster state between all instances. ### Configuration requirements The following requirements must be met to run Loki in HA monolithic mode: 1. **Run every instance with `-target=all`.** All replicas use the single-binary target so that every process serves every component. 1. **Shared object storage.** All instances must read from and write to the same S3-compatible bucket (or equivalent) for chunk data and, if rulers are used, a separate bucket for ruler data. Local disk is only used for ephemeral caches and the write-ahead log (WAL). 1. **Memberlist for ring state.** Configure the `memberlist` block so that every instance knows the addresses of all other instances. Set `common.ring.kvstore.store: memberlist` so that all internal rings (ingester, distributor, and so on) use the shared memberlist cluster. 1. **Replication factor of 3.** Set `common.replication_factor: 3` and run **at least three** instances so that the cluster can tolerate the loss of one instance without data loss. 1. **Dedicated compactor instance.** Designate one instance as the main compactor by setting `compactor.horizontal_scaling_mode: main` on one instance and `worker` on all others. The address of the main compactor needs to be set in `common.compactor_grpc_address`. 1. **Load balancer in front of all instances.** Route all inbound push and query traffic to every replica in round-robin order. See the [Loki HA Monolithic example](https://github.com/grafana/loki/tree/main/examples/ha-monolithic) for a more complete configuration. ## Microservices mode Sometimes also referred to as **distributed** deployment. The microservices deployment mode runs components of Loki as distinct processes. The microservices deployment is also referred to as a Distributed deployment. Each process is invoked specifying its `target`. For release 3.3 the components are: - Bloom Builder (deprecated) - Bloom Gateway (deprecated) - Bloom Planner (deprecated) - Compactor - Distributor - Index Gateway - Ingester - Overrides Exporter - Querier - Query Frontend - Query Scheduler - Ruler - Table Manager (deprecated) {{< admonition type="tip" >}} You can see the complete list of targets for your version of Loki by running Loki with the flag `-list-targets`, for example: ```bash docker run docker.io/grafana/loki:3.2.1 -config.file=/etc/loki/local-config.yaml -list-targets ``` {{< /admonition >}} ![Microservices mode diagram](../microservices-mode.png "Microservices mode") Running components as individual microservices provides more granularity, letting you scale each component as individual microservices, to better match your specific use case. Microservices mode deployments can be more efficient Loki installations. However, they are also the most complex to set up and maintain. Microservices mode is only recommended for very large Loki clusters or for operators who need more precise control over scaling and cluster operations. Microservices mode is designed for Kubernetes deployments. A [community-supported Helm chart](https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed) is available for deploying Loki in microservices mode. # Consistent hash rings [Consistent hash rings](https://en.wikipedia.org/wiki/Consistent_hashing) are incorporated into Loki cluster architectures to - aid in the sharding of log lines - implement high availability - ease the horizontal scale up and scale down of clusters. There is less of a performance hit for operations that must rebalance data. Hash rings connect instances of a single type of component when - there are a set of Loki instances in monolithic deployment mode - there are multiple read components or multiple write components in simple scalable deployment mode - there are multiple instances of one type of component in microservices mode Not all Loki components are connected by hash rings. These components need to be connected into a hash ring: - distributors - ingesters - query schedulers - compactors - rulers These components can optionally be connected into a hash ring: - index gateway In an architecture that has three distributors and three ingesters defined, the hash rings for these components connect the instances of same-type components. ![Distributor and ingester rings](../ring-overview.png "Distributor and ingester rings") Each node in the ring represents an instance of a component. Each node has a key-value store that holds communication information for each of the nodes in that ring. Nodes update the key-value store periodically to keep the contents consistent across all nodes. For each node, the key-value store holds: - an ID of the component node - component address, used by other nodes as a communication channel - an indication of the component node's health ## Configuring rings Define [ring configuration](https://grafana.com/docs/loki//configure/#common) within the `common.ring` block. Use the `memberlist` key-value store type unless there is a compelling reason to use a different key-value store type. `memberlist` uses a [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol) to propagate information to all the nodes to guarantee the eventual consistency of the key-value store contents. There are additional configuration options for distributor rings, ingester rings, and ruler rings. These options are for advanced, specialized use only. These options are defined within the `distributor.ring` block for distributors, the `ingester.lifecycler.ring` block for ingesters, and the `ruler.ring` block for rulers. ## About the distributor ring Distributors use the information in their key-value store to keep a count of the quantity of distributors in the distributor ring. The count further informs cluster limits. ## About the ingester ring Ingester ring information in the key-value stores is used by distributors. The information lets the distributors shard log lines, determining which ingester or set of ingesters a distributor sends log lines to. ## About the query scheduler ring Query schedulers use the information in their key-value store for service discovery of the schedulers. This allows queriers to connect to all available schedulers, and it allows schedulers to connect to all available query frontends, effectively creating a single queue that aids in balancing the query load. ## About the compactor ring Compactors use the information in the key-value store to identify a single compactor instance that will be responsible for compaction. The compactor is only enabled on the responsible instance, despite the compactor target being on multiple instances. ## About the ruler ring The ruler ring is used to determine which rulers evaluate which rule groups. ## About the index gateway ring The index gateway ring is used to determine which gateway is responsible for which tenant's indexes when queried by rulers or queriers. # Loki overview Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by [Prometheus](https://prometheus.io/). Loki differs from Prometheus by focusing on logs instead of metrics, and collecting logs via push, instead of pull. Loki is designed to be very cost effective and highly scalable. Unlike other logging systems, Loki does not index the contents of the logs, but only indexes metadata about your logs as a set of labels for each log stream. A log stream is a set of logs which share the same labels. Labels help Loki to find a log stream within your data store, so having a quality set of labels is key to efficient query execution. Log data is then compressed and stored in chunks in an object store such as Amazon Simple Storage Service (S3) or Google Cloud Storage (GCS), or even, for development or proof of concept, on the filesystem. A small index and highly compressed chunks simplify the operation and significantly lower the cost of Loki. {{< figure src="../loki-overview-2.png" caption="**Loki logging stack**" >}} A typical Loki-based logging stack consists of 3 components: - **Agent** - An agent or client, for example [Grafana Alloy](https://grafana.com/docs/alloy/latest/). The agent scrapes logs, turns the logs into streams by adding labels, and pushes the streams to Loki through an HTTP API. - **Loki** - The main server, responsible for ingesting and storing logs and processing queries. It can be deployed in three different configurations, for more information see [deployment modes](../deployment-modes/). - **[Grafana](https://github.com/grafana/grafana)** for querying and displaying log data. You can also query logs from the command line, using [LogCLI](../../query/logcli/) or using the Loki API directly. ## Loki features - **Scalability** - Loki is designed for scalability, and can scale from as small as running on a Raspberry Pi to ingesting petabytes a day. Loki can run as a single binary for simple setups, in [HA monolithic mode](../deployment-modes/#ha-monolithic-mode) for moderate horizontal scalability without added operational complexity, or as fine-grained microservices designed to run natively within Kubernetes for the largest, highest-scale installations. {{< admonition type="note" >}} Simple Scalable Deployment (SSD) mode, which decoupled requests into separate read and write paths, is deprecated and will be removed in Loki 4.0. The new HA monolithic mode will be the recommended replacement for most SSD use cases. See [deployment modes](../deployment-modes/) for details. {{< /admonition >}} - **Multi-tenancy** - Loki allows multiple tenants to share a single Loki instance. With multi-tenancy, the data and requests of each tenant is completely isolated from the others. Multi-tenancy is [configured](../../operations/multi-tenancy/) by assigning a tenant ID in the agent. - **Third-party integrations** - Several third-party agents (clients) have support for Loki, via plugins. This lets you keep your existing observability setup while also shipping logs to Loki. - **Efficient storage** - Loki stores log data in highly compressed chunks. Similarly, the Loki index, because it indexes only the set of labels, is significantly smaller than other log aggregation tools. By leveraging object storage as the only data storage mechanism, Loki inherits the reliability and stability of the underlying object store. It also capitalizes on both the cost efficiency and operational simplicity of object storage over other storage mechanisms like locally attached solid state drives (SSD) and hard disk drives (HDD). The compressed chunks, smaller index, and use of low-cost object storage, make Loki less expensive to operate. - **LogQL, the Loki query language** - [LogQL](../../query/) is the query language for Loki. Users who are already familiar with the Prometheus query language, [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/), will find LogQL familiar and flexible for generating queries against the logs. The language also facilitates the generation of metrics from log data, a powerful feature that goes well beyond log aggregation. - **Alerting** - Loki includes a component called the [ruler](../../alert/), which can continually evaluate queries against your logs, and perform an action based on the result. This allows you to monitor your logs for anomalies or events. Loki integrates with [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/), or the [alert manager](/docs/grafana/latest/alerting) within Grafana. - **Grafana integration** - Loki integrates with Grafana, Mimir, and Tempo, providing a complete observability stack, and seamless correlation between logs, metrics and traces. ## Frequently asked questions {{< qa-list >}} {{< qa question="What is a good open source tool for aggregating and querying logs?" >}} Grafana Loki is a horizontally scalable, highly available log aggregation system built for the cloud-native era. Instead of expensively indexing every line of your logs, Loki indexes just a small set of labels and stores the rest in cheap object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage, dramatically lowering your costs while scaling effortlessly from a Raspberry Pi to petabytes a day. Paired with the powerful LogQL query language and deeply integrated with Grafana, Loki brings your logs, metrics, and traces together in one seamless observability experience. {{< /qa >}} {{< qa question="What are the best open source log aggregation tools for Kubernetes?" >}} The strongest open source option for most cloud-native teams, Grafana Loki is the best choice. Purpose-built for Kubernetes, Loki indexes only a small set of labels, rather than the full contents of every log line, then stores compressed logs in cheap object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. The result is dramatically lower cost and effortless scaling—from a single binary up to petabytes a day. With the Prometheus-inspired LogQL query language, built-in alerting, and seamless integration with Grafana, Mimir, and Tempo, Loki delivers cost-efficient, Kubernetes-native log aggregation inside a complete observability stack. {{< /qa >}} {{< qa question="What is Grafana Loki and how does it work?" >}} Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Unlike traditional logging tools that index the full contents of every log line, Loki indexes only a small set of labels (metadata) for each log stream, while the log data itself is compressed and stored in chunks in low-cost object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage, keeping the index small and making Loki cheaper to operate and easier to scale. In practice, an agent like Grafana Alloy scrapes your logs, attaches labels to turn them into streams, and pushes them to Loki over HTTP; Loki then ingests and stores those streams and serves queries written in LogQL, its Prometheus-inspired query language. You can explore the results in Grafana, alert on log patterns using the built-in ruler, and correlate your logs with metrics and traces for a complete observability stack. {{< /qa >}} {{< qa question="How does Grafana Loki store and query logs?" >}} Grafana Loki separates a small index from the bulk log data to keep storage cheap and scalable. Rather than indexing the full contents of your logs, Loki groups incoming logs into streams identified by a set of labels and indexes only that metadata; the log lines themselves are compressed and written as chunks to low-cost object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. When you run a query, Loki uses the labels to quickly locate the relevant streams, then fetches and decompresses only the matching chunks, which is why a well-chosen, low-cardinality set of labels is key to fast queries. You express those queries in LogQL, Loki's Prometheus-inspired query language, using label matchers to select streams, line and label filters to narrow results, and pipeline stages that can even generate metrics from your log data. {{< /qa >}} {{< /qa-list >}} # Label best practices Grafana Loki is under active development, and we are constantly working to improve performance. But here are some of the most current best practices for labels that will give you the best experience with Loki. ## Static labels are good Use labels for things like regions, clusters, servers, applications, namespaces, and environments. They will be fixed for a given system/app and have bounded values. Use static labels to make it easier to query your logs in a logical sense (for example, show me all the logs for a given application and specific environment, or show me all the logs for all the apps on a specific host). ## Use dynamic labels sparingly Too many label value combinations leads to too many streams. The penalties for that in Loki are a large index and small chunks in the store, which in turn can actually reduce performance. To avoid those issues, don't add a label for something until you know you need it! Use filter expressions (`|= "text"`, `|~ "regex"`, …) and brute force those logs. It works -- and it's fast. If you often parse a label from a log line at query time, the label has a high cardinality, and extracting that label is expensive in terms of performance; consider extracting the label on the client side attaching it as [structured metadata](../structured-metadata/) to log lines . From early on, we have set a label dynamically using client pipelines for `level`. This seemed intuitive for us as we often wanted to only show logs for `level="error"`; however, we are re-evaluating this now as writing a query. `{app="loki"} |= "level=error"` is proving to be just as fast for many of our applications as `{app="loki",level="error"}`. This may seem surprising, but if applications have medium to low volume, that label causes one application's logs to be split into up to five streams, which means 5x chunks being stored. And loading chunks has an overhead associated with it. Imagine now if that query were `{app="loki",level!="debug"}`. That would have to load **way** more chunks than `{app="loki"} != "level=debug"`. Above, we mentioned not to add labels until you _need_ them, so when would you _need_ labels?? A little farther down is a section on `chunk_target_size`. If you set this to 1MB (which is reasonable), this will try to cut chunks at 1MB compressed size, which is about 5MB-ish of uncompressed logs (might be as much as 10MB depending on compression). If your logs have sufficient volume to write 5MB in less time than `max_chunk_age`, or **many** chunks in that timeframe, you might want to consider splitting it into separate streams with a dynamic label. What you want to avoid is splitting a log file into streams, which result in chunks getting flushed because the stream is idle or hits the max age before being full. As of [Loki 1.4.0](/blog/2020/04/01/loki-v1.4.0-released-with-query-statistics-and-up-to-300x-regex-optimization/), there is a metric which can help you understand why chunks are flushed `sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster="dev"}[1m]))`. It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a `level` dynamic label, just `|= "level=debug"` instead. Here are some best practices for using dynamic labels with Loki: - Ensure the labels have low cardinality, ideally limited to tens of values. - Use labels with long-lived values, such as the initial segment of an HTTP path: `/load`, `/save`, `/update`. - Do not extract ephemeral values like a trace ID or an order ID into a label; the values should be static, not dynamic. - Only add labels that users will frequently use in their queries. - Don’t increase the size of the index and fragment your log streams if nobody is actually using these labels. This will degrade performance. ## Label values must always be bounded If you are dynamically setting labels, never use a label which can have unbounded or infinite values. This will always result in big problems for Loki. Try to keep values bounded to as small a set as possible. We don't have perfect guidance as to what Loki can handle, but think single digits, or maybe 10’s of values for a dynamic label. This is less critical for static labels. For example, if you have 1,000 hosts in your environment it's going to be just fine to have a host label with 1,000 values. As a general rule, you should try to keep any single tenant in Loki to less than **100,000 active streams**, and less than a million streams in a 24-hour period. These values are for HUGE tenants, sending more than **10 TB** a day. If your tenant is 10x smaller, you should have at least 10x fewer labels. ## Be aware of dynamic labels applied by clients Loki has several client options: [Grafana Alloy](https://grafana.com/docs/alloy/latest/) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd](https://grafana.com/docs/loki//send-data/fluentd/), [Fluent Bit](https://grafana.com/docs/loki//send-data/fluentbit/), a [Docker plugin](https://grafana.com/docs/loki//send-data/docker-driver/), and more. Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied. Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality. Series information can be queried through the [Series API](https://grafana.com/docs/loki//reference/loki-http-api/), or you can use [logcli](../../../query/). In Loki 1.6.0 and newer the logcli series command added the `--analyze-labels` flag specifically for debugging high cardinality labels: ``` Total Streams: 25017 Unique Labels: 8 Label Name Unique Values Found In Streams requestId 24653 24979 logStream 1194 25016 logGroup 140 25016 accountId 13 25016 logger 1 25017 source 1 25016 transport 1 25017 format 1 25017 ``` In this example you can see the `requestId` label had a 24653 different values out of 24979 streams it was found in, this is bad!! This is a perfect example of something which should not be a label, `requestId` should be removed as a label and instead filter expressions should be used to query logs for a specific `requestId`. For example if `requestId` is found in the log line as a key=value pair you could write a query like this: `{logGroup="group1"} |= "requestId=32422355"` # Cardinality The cardinality of a data attribute is the number of distinct values that the attribute can have. For example, a boolean column in a database, which can only have a value of either `true` or `false` has a cardinality of 2. High cardinality refers to a column or row in a database that can have many possible values. For an online shopping system, fields like `userId`, `shoppingCartId`, and `orderId` are often high-cardinality columns that can have hundreds of thousands of distinct values. Other examples of high cardinality attributes include the following: - Timestamp - IP addresses - Kubernetes pod names - User ID - Customer ID - Trace ID When we talk about _cardinality_ in Loki we are referring to the combination of labels and values and the number of log streams they create. In Loki, the fewer labels you use, the better. This is why Loki has a default limit of 15 index labels. High cardinality can result from using labels with a large range of possible values, **or** combining many labels, even if they have a small and finite set of values, such as combining `status_code` and `action`. A typical set of status codes (200, 404, 500) and actions (GET, POST, PUT, PATCH, DELETE) would create 15 unique streams. But, adding just one more label like `endpoint` (/cart, /products, /customers) would triple this to 45 unique streams. To see an example of series labels and cardinality, refer to the [LogCLI tutorial](https://grafana.com/docs/loki//query/logcli/logcli-tutorial/#checking-series-cardinality). As you can see, the cardinality for individual labels can be quite high, even before you begin combining labels for a particular log stream, which increases the cardinality even further. To view the cardinality of your current labels, you can use [logcli](https://grafana.com/docs/loki//query/logcli/getting-started/). `logcli series '{}' --since=1h --analyze-labels` ## Impact of high cardinality in Loki High cardinality causes Loki to create many streams, especially when labels have many unique values, and when those values are short-lived (for example, active for seconds or minutes). This causes Loki to build a huge index, and to flush thousands of tiny chunks to the object store. Loki was not designed or built to support high cardinality label values. In fact, it was built for exactly the opposite. It was built for very long-lived streams and very low cardinality in the labels. In Loki, the fewer labels you use, the better. High cardinality can lead to significant performance degradation. ## Avoiding high cardinality To avoid high cardinality in Loki, you should: - Avoid assigning labels with unbounded values, for example timestamp, trace ID, order ID. - Prefer static labels that describe the origin or context of the log message, for example, application, namespace, environment. - Don't assign "dynamic" labels, which are values from the log message itself, unless it is low-cardinality, or a long-lived value. - Use structured metadata to store frequently-searched, high-cardinality metadata fields, such as customer IDs or transaction IDs, without impacting Loki's index. {{< admonition type="note" >}} [Structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) is a feature in Loki and Cloud Logs that allows customers to store metadata that is too high cardinality for log lines, without needing to embed that information in log lines themselves. It is a great home for metadata which is not easily embeddable in a log line, but is too high cardinality to be used effectively as a label. [Query acceleration with Blooms](https://grafana.com/docs/loki//operations/bloom-filters/) also utilizes structured metadata. {{< /admonition >}} # Modify default OpenTelemetry labels When Grafana Loki first started supporting OpenTelemetry, we selected a set of [default resource attributes to promote to labels](https://grafana.com/docs/loki//get-started/labels/#default-labels-for-opentelemetry). We included some labels such as k8s.pod.name to be consistent with the scrape configs we have been suggesting for many years for Grafana Alloy. However, Loki has evolved a lot over the years and with the introduction of Structured Metadata we no longer recommend indexing a few of the labels we included in the defaults as they often have very high [cardinality](https://grafana.com/docs/loki//get-started/labels/cardinality/) and Structured Metadata is a much better place for them. We no longer recommend the following as default labels: - `k8s.pod.name` - `service.instance.id` Because removing these resource attributes from labels would be a breaking change for existing users, they have not yet been deprecated. If you are a new user of Grafana Loki, we recommend that you modify your Grafana Alloy or OpenTelemetry Collector configuration to convert these resource attributes from index labels to structured metadata. ## Alloy configuration file example If you are using [Grafana Alloy](https://grafana.com/docs/alloy/latest/) to collect logs, the following example shows how to update your `alloy.config` file to demote `k8s.pod.name` from an index label to structured metadata. ```alloy // discovery.kubernetes allows you to find scrape targets from Kubernetes resources. // It watches cluster state and ensures targets are continually synced with what is currently running in your cluster. discovery.kubernetes "pod" { role = "pod" } // discovery.relabel rewrites the label set of the input targets by applying one or more relabeling rules. // If no rules are defined, then the input targets are exported as-is. discovery.relabel "pod_logs" { targets = discovery.kubernetes.pod.targets // Label creation - "namespace" field from "__meta_kubernetes_namespace" rule { source_labels = ["__meta_kubernetes_namespace"] action = "replace" target_label = "namespace" } // Label creation - "pod" field from "__meta_kubernetes_pod_name" //rule { // source_labels = ["__meta_kubernetes_pod_name"] // action = "replace" // target_label = "pod" //} // Label creation - "container" field from "__meta_kubernetes_pod_container_name" rule { source_labels = ["__meta_kubernetes_pod_container_name"] action = "replace" target_label = "container" } // Label creation - "app" field from "__meta_kubernetes_pod_label_app_kubernetes_io_name" rule { source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"] action = "replace" target_label = "app" } // Label creation - "job" field from "__meta_kubernetes_namespace" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_namespace/__meta_kubernetes_pod_container_name rule { source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "job" separator = "/" replacement = "$1" } // Label creation - "container" field from "__meta_kubernetes_pod_uid" and "__meta_kubernetes_pod_container_name" // Concatenate values __meta_kubernetes_pod_uid/__meta_kubernetes_pod_container_name.log rule { source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"] action = "replace" target_label = "__path__" separator = "/" replacement = "/var/log/pods/*$1/*.log" } // Label creation - "container_runtime" field from "__meta_kubernetes_pod_container_id" rule { source_labels = ["__meta_kubernetes_pod_container_id"] action = "replace" target_label = "container_runtime" regex = "^(\\S+):\\/\\/.+$" replacement = "$1" } } // loki.source.kubernetes tails logs from Kubernetes containers using the Kubernetes API. loki.source.kubernetes "pod_logs" { targets = discovery.relabel.pod_logs.output forward_to = [loki.process.pod_logs.receiver] } // loki.process receives log entries from other Loki components, applies one or more processing stages, // and forwards the results to the list of receivers in the component's arguments. loki.process "pod_logs" { stage.static_labels { values = { cluster = "", } } forward_to = [loki.write..receiver] } ``` ## Kubernetes Monitoring Helm chart configuration example If you are using the [Kubernetes Monitoring Helm chart](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/) to collect Kubernetes logs to send to Loki, the following example shows how to update the `values.yaml` file for your Helm chart to demote `k8s.pod.name` and `service.instance.id` from index labels to structured metadata when gathering Pod logs via the Kubernetes API (the `podLogsViaKubernetesApi` feature). ```yaml # Enable Pod log collection for the cluster. Collects logs from all Pods in both the meta and loki namespaces. podLogsViaKubernetesApi: enabled: true # collector must match a name defined under your own top-level collectors: map. collector: alloy-singleton namespaces: - meta - loki # This is the section that specifies to send k8s.pod.name and service.instance.id to structured metadata # instead of promoting them to indexed labels. Format is : . structuredMetadata: k8s.pod.name: k8s.pod.name service.instance.id: service.instance.id ``` {{< admonition type="note" >}} Chart version 4.x replaced the old `labelsToKeep` filter list with explicit `labels:` and `annotations:` promotion maps: instead of filtering down a larger generated label set, you opt in to the specific Pod labels and annotations you want copied to log labels. By default, `podLogsViaKubernetesApi` already keeps `k8s.pod.name` and `service.instance.id` out of the indexed labels and sends them as structured metadata; the configuration above shows how to set this explicitly. For the full set of available `labels:`, `annotations:`, and `structuredMetadata:` options, refer to the [feature-pod-logs-via-kubernetes-api documentation](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/charts/feature-pod-logs-via-kubernetes-api). {{< /admonition >}} ## Loki configuration example If you are running Loki in a Docker container or on-premises, here is an example of how to modify your Loki `values.yaml` file to demote `k8s.pod.name` and `service.instance.id` from index labels to structured metadata. ```yaml distributor: otlp_config: # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name] ``` Kubernetes example `values.yaml`: ```yaml loki: distributor: otlp_config: # List of default otlp resource attributes to be picked as index labels - EDIT TO REMOVE k8s.pod.name AND service.instance.id FROM THE LIST # CLI flag: -distributor.otlp.default_resource_attributes_as_index_labels default_resource_attributes_as_index_labels: [service.name service.namespace deployment.environment deployment.environment.name cloud.region cloud.availability_zone k8s.cluster.name k8s.namespace.name k8s.container.name container.name k8s.replicaset.name k8s.deployment.name k8s.statefulset.name k8s.daemonset.name k8s.cronjob.name k8s.job.name] ``` # What is structured metadata {{< admonition type="warning" >}} Structured metadata was added to chunk format V4, which is used if the schema version is greater than or equal to `13`. Structured metadata also requires the `tsdb` index type; it is not supported by the deprecated `boltdb-shipper` store. See [Storage schema](https://grafana.com/docs/loki//operations/storage/schema/) for more details about schema versions and index types. {{< /admonition >}} Selecting proper, low cardinality labels is critical to operating and querying Loki effectively. Some metadata, especially infrastructure related metadata, can be difficult to embed in log lines, and is too high cardinality to effectively store as indexed labels (and therefore reducing performance of the index). Structured metadata is a way to attach metadata to logs without indexing them or including them in the log line content itself. Examples of useful metadata are kubernetes pod names, process ID's, or any other label that is often used in queries but has high cardinality and is expensive to extract at query time. Structured metadata can also be used to query commonly needed metadata from log lines without needing to apply a parser at query time. Large json blobs or a poorly written query using complex regex patterns, for example, come with a high performance cost. Examples of useful metadata include container_IDs or user IDs. ## When to use structured metadata You should only use structured metadata in the following situations: - If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data. - If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names. - If you are using [Logs Drilldown](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/) to visualize and explore your Loki logs. You must enable `pattern_ingester.enabled`, `allow_structured_metadata`, `volume_enabled`, and `discover_log_levels` in your Loki configuration. For the full setup, refer to [Access or install Grafana Logs Drilldown](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/access/). - If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using [Bloom filters](https://grafana.com/docs/loki//operations/bloom-filters/) (deprecated), starting in [Loki 3.3](https://grafana.com/docs/loki//release-notes/v3-3/) Bloom filters now utilize structured metadata. ## Enable or disable structured metadata You enable structured metadata in the Loki config.yaml file. ```yaml limits_config: allow_structured_metadata: true ``` If you are also using [Logs Drilldown](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/), you need additional settings. The following example enables structured metadata along with the other settings required by Logs Drilldown: ```yaml pattern_ingester: enabled: true limits_config: allow_structured_metadata: true volume_enabled: true discover_log_levels: true ``` You can disable Structured Metadata by setting `allow_structured_metadata: false` in the `limits_config` section or set the command line argument `-validation.allow-structured-metadata=false`. Note structured metadata is required to support ingesting OTLP data. ## Attaching structured metadata to log lines You have the option to attach structured metadata to log lines in the push payload along with each log line and the timestamp. For more information on how to push logs to Loki via the HTTP endpoint, refer to the [HTTP API documentation](https://grafana.com/docs/loki//reference/api/#ingest-logs). Alternatively, you can use Grafana Alloy to extract and attach structured metadata to your log lines. See the [Alloy `stage.structured_metadata` block](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagestructured_metadata) for more information. With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash](https://grafana.com/docs/loki//send-data/logstash/). {{< admonition type="warning" >}} Structured metadata size is taken into account while asserting ingestion rate limiting. Along with that, there are separate limits on how much structured metadata can be attached per log line. ```yaml # Maximum size accepted for structured metadata per log line. # CLI flag: -limits.max-structured-metadata-size [max_structured_metadata_size: | default = 64KB] # Maximum number of structured metadata entries per log line. # CLI flag: -limits.max-structured-metadata-entries-count [max_structured_metadata_entries_count: | default = 128] ``` Log lines that exceed either limit are rejected by the distributor with an HTTP 400 response, and the discarded entries are counted in the `loki_discarded_samples_total` metric with reason `structured_metadata_too_large` or `structured_metadata_too_many`. A large OpenTelemetry log attribute such as `exception.stacktrace` can be the cause — refer to [Handle large attributes such as stack traces](https://grafana.com/docs/loki//send-data/otel/#handle-large-attributes-such-as-stack-traces) for mitigations. {{< /admonition >}} ## Querying structured metadata Structured metadata is extracted automatically for each returned log line and added to the labels returned for the query. You can use labels of structured metadata to filter log line using a [label filter expression](https://grafana.com/docs/loki//query/log_queries/#label-filter-expression). For example, if you have a label `pod` attached to some of your log lines as structured metadata, you can filter log lines using: ```logql {job="example"} | pod="myservice-abc1234-56789" ``` Of course, you can filter by multiple labels of structured metadata at the same time: ```logql {job="example"} | pod="myservice-abc1234-56789" | trace_id="0242ac120002" ``` Note that since structured metadata is extracted automatically to the results labels, some metric queries might return an error like `maximum of series (50000) reached for a single query`. You can use the [Keep](https://grafana.com/docs/loki//query/log_queries/#keep-labels-expression) and [Drop](https://grafana.com/docs/loki//query/log_queries/#drop-labels-expression) stages to filter out labels that you don't need. For example: ```logql count_over_time({job="example"} | trace_id="0242ac120002" | keep job [5m]) ``` # Quickstart to run Loki locally # Quickstart to run Loki locally If you want to experiment with Loki, you can run Loki locally using the Docker Compose file that ships with Loki. It runs Loki in the [simple scalable deployment](https://grafana.com/docs/loki//get-started/deployment-modes/#simple-scalable) mode and includes a sample application to generate logs. Simple Scalable Deployment (SSD) mode is deprecated and scheduled for removal in Loki 4.0; for production deployments, see [deployment modes](https://grafana.com/docs/loki//get-started/deployment-modes/) for currently recommended options. The Docker Compose configuration runs the following components, each in its own container: - **flog**: which generates log lines. [flog](https://github.com/mingrammer/flog) is a log generator for common log formats. - **Grafana Alloy**: which scrapes the log lines from flog, and pushes them to Loki through the gateway. - **Gateway** (nginx) which receives requests and redirects them to the appropriate container based on the request's URL. - **Loki read component**: which runs a Query Frontend and a Querier. - **Loki write component**: which runs a Distributor and an Ingester. - **Loki backend component**: which runs an Index Gateway, Compactor, Ruler, Bloom Planner (deprecated), Bloom Builder (deprecated), and Bloom Gateway (deprecated). - **Minio**: which Loki uses to store its index and chunks. - **Grafana**: which provides visualization of the log lines captured within Loki. {{< figure max-width="75%" src="/media/docs/loki/get-started-flog-v3.png" caption="Getting started sample application" alt="Getting started sample application" >}} ## Before you begin Before you start, you need to have the following installed on your local system: - Install [Docker](https://docs.docker.com/install) - Install [Docker Compose](https://docs.docker.com/compose/install) {{< admonition type="tip" >}} Alternatively, you can try out this example in our interactive learning environment: [Loki Quickstart Sandbox](https://killercoda.com/grafana-labs/course/loki/loki-quickstart). It's a fully configured environment with all the dependencies already installed. ![Interactive](/media/docs/loki/loki-ile.svg) Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). {{< /admonition >}} ## Install Loki and collecting sample logs {{< admonition type="note" >}} This quickstart assumes you are running Linux. {{< /admonition >}} **To install Loki locally, follow these steps:** 1. Create a directory called `evaluate-loki` for the demo environment. Make `evaluate-loki` your current working directory: ```bash mkdir evaluate-loki cd evaluate-loki ``` 2. Download `loki-config.yaml`, `alloy-local-config.yaml`, and `docker-compose.yaml`: {{< tabs >}} {{< tab-content name="wget" >}} ```bash wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/loki-config.yaml -O loki-config.yaml wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/alloy-local-config.yaml -O alloy-local-config.yaml wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/docker-compose.yaml -O docker-compose.yaml ``` {{< /tab-content >}} {{< tab-content name="curl" >}} ```bash curl https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/loki-config.yaml --output loki-config.yaml curl https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/alloy-local-config.yaml --output alloy-local-config.yaml curl https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/docker-compose.yaml --output docker-compose.yaml ``` {{< /tab-content >}} {{< /tabs >}} {{< docs/ignore >}} ```bash wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/loki-config.yaml -O loki-config.yaml wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/alloy-local-config.yaml -O alloy-local-config.yaml wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/docker-compose.yaml -O docker-compose.yaml ``` {{< /docs/ignore >}} 3. Deploy the sample Docker image. With `evaluate-loki` as the current working directory, start the demo environment using `docker compose`: ```bash docker compose up -d ``` At the end of the command, you should see something similar to the following: ```console ✔ Network evaluate-loki_loki Created 0.1s ✔ Container evaluate-loki-minio-1 Started 0.6s ✔ Container evaluate-loki-flog-1 Started 0.6s ✔ Container evaluate-loki-backend-1 Started 0.8s ✔ Container evaluate-loki-write-1 Started 0.8s ✔ Container evaluate-loki-read-1 Started 0.8s ✔ Container evaluate-loki-gateway-1 Started 1.1s ✔ Container evaluate-loki-grafana-1 Started 1.4s ✔ Container evaluate-loki-alloy-1 Started 1.4s ``` 4. (Optional) Verify that the Loki cluster is up and running. - The read component returns `ready` when you browse to [http://localhost:3101/ready](http://localhost:3101/ready). The message `Query Frontend not ready: not ready: number of schedulers this worker is connected to is 0` shows until the read component is ready. - The write component returns `ready` when you browse to [http://localhost:3102/ready](http://localhost:3102/ready). The message `Ingester not ready: waiting for 15s after being ready` shows until the write component is ready. 5. (Optional) Verify that Grafana Alloy is running. - You can access the Grafana Alloy UI at [http://localhost:12345](http://localhost:12345). 6. (Optional) You can check all the containers are running by running the following command: ```bash docker ps -a ``` ## View your logs in Grafana After you have collected logs, you will want to view them. You can view your logs using the command line interface, [LogCLI](/docs/loki//query/logcli/), but the easiest way to view your logs is with Grafana. 1. Use Grafana to query the Loki data source. The test environment includes [Grafana](https://grafana.com/docs/grafana/latest/), which you can use to query and observe the sample logs generated by the flog application. You can access the Grafana cluster by browsing to [http://localhost:3000](http://localhost:3000). The Grafana instance in this demonstration has a Loki [data source](https://grafana.com/docs/grafana/latest/datasources/loki/) already configured. {{< figure src="/media/docs/loki/grafana-query-builder-v2.png" caption="Grafana Explore" alt="Grafana Explore" >}} 1. From the Grafana main menu, click the **Explore** icon (1) to open the Explore tab. To learn more about Explore, refer to the [Explore](https://grafana.com/docs/grafana/latest/explore/) documentation. 1. From the menu in the dashboard header, select the Loki data source (2). This displays the Loki query editor. In the query editor you use the Loki query language, [LogQL](https://grafana.com/docs/loki//query/), to query your logs. To learn more about the query editor, refer to the [query editor documentation](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/). 1. The Loki query editor has two modes (3): - [Builder mode](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#builder-mode), which provides a visual query designer. - [Code mode](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#code-mode), which provides a feature-rich editor for writing LogQL queries. Next we’ll walk through a few simple queries using both the builder and code views. 1. Click **Code** (3) to work in Code mode in the query editor. Here are some sample queries to get you started using LogQL. These queries assume that you followed the instructions to create a directory called `evaluate-loki`. If you installed in a different directory, you’ll need to modify these queries to match your installation directory. After copying any of these queries into the query editor, click **Run Query** (4) to execute the query. 1. View all the log lines which have the container label `evaluate-loki-flog-1`: ```bash {container="evaluate-loki-flog-1"} ``` In Loki, this is a log stream. Loki uses [labels](https://grafana.com/docs/loki//get-started/labels/) as metadata to describe log streams. Loki queries always start with a label selector. In the previous query, the label selector is `{container="evaluate-loki-flog-1"}`. 1. To view all the log lines which have the container label `evaluate-loki-grafana-1`: ```bash {container="evaluate-loki-grafana-1"} ``` 1. Find all the log lines in the `{container="evaluate-loki-flog-1"}` stream that contain the string `status`: ```bash {container="evaluate-loki-flog-1"} |= `status` ``` 1. Find all the log lines in the `{container="evaluate-loki-flog-1"}` stream where the JSON field `status` has the value `404`: ```bash {container="evaluate-loki-flog-1"} | json | status=`404` ``` 1. Calculate the number of logs per second where the JSON field `status` has the value `404`: ```bash sum by(container) (rate({container="evaluate-loki-flog-1"} | json | status=`404` [$__auto])) ``` The final query is a metric query which returns a time series. This makes Grafana draw a graph of the results. You can change the type of graph for a different view of the data. Click **Bars** to view a bar graph of the data. 1. Click the **Builder** tab (3) to return to builder mode in the query editor. 1. In builder mode, click **Kick start your query** (5). 1. Expand the **Log query starters** section. 1. Select the first choice, **Parse log lines with logfmt parser**, by clicking **Use this query**. 1. On the Explore tab, click **Label browser**, in the dialog select a container and click **Show logs**. For a thorough introduction to LogQL, refer to the [LogQL reference](https://grafana.com/docs/loki//query/). ## Sample queries (code view) Here are some more sample queries that you can run using the Flog sample data. To see all the log lines that flog has generated, enter the LogQL query: ```bash {container="evaluate-loki-flog-1"} ``` The flog app generates log lines for simulated HTTP requests. To see all `GET` log lines, enter the LogQL query: ```bash {container="evaluate-loki-flog-1"} |= "GET" ``` To see all `POST` methods, enter the LogQL query: ```bash {container="evaluate-loki-flog-1"} |= "POST" ``` To see every log line with a 401 status (unauthorized error), enter the LogQL query: ```bash {container="evaluate-loki-flog-1"} | json | status="401" ``` To see every log line that doesn't contain the text `401`: ```bash {container="evaluate-loki-flog-1"} != "401" ``` For more examples, refer to the [query documentation](https://grafana.com/docs/loki//query/query_examples/). ## Loki data source in Grafana In this example, the Loki data source is already configured in Grafana. This can be seen within the `docker-compose.yaml` file: ```yaml grafana: image: grafana/grafana:latest environment: - GF_PATHS_PROVISIONING=/etc/grafana/provisioning - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin depends_on: - gateway entrypoint: - sh - -euc - | mkdir -p /etc/grafana/provisioning/datasources cat < /etc/grafana/provisioning/datasources/ds.yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy url: http://gateway:3100 jsonData: httpHeaderName1: "X-Scope-OrgID" secureJsonData: httpHeaderValue1: "tenant1" EOF /run.sh ``` Within the entrypoint section, the Loki data source is configured with the following details: - `Name: Loki` (name of the data source) - `Type: loki` (type of data source) - `Access: proxy` (access type) - `URL: http://gateway:3100` (URL of the Loki data source. Loki uses an nginx gateway to direct traffic to the appropriate component) - `jsonData.httpHeaderName1: "X-Scope-OrgID"` (header name for the organization ID) - `secureJsonData.httpHeaderValue1: "tenant1"` (header value for the organization ID) It is important to note when Loki is configured in any other mode other than monolithic deployment, you are required to pass a tenant ID in the header. Without this, queries will return an authorization error. ## Complete metrics, logs, traces, and profiling example You have completed the Loki Quickstart demo. So where to go next? {{< docs/ignore >}} ## Back to docs Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/get-started/quick-start/). {{< /docs/ignore >}} If you would like to run a demonstration environment that includes Mimir, Loki, Tempo, and Grafana, you can use [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mlt). It's a self-contained environment for learning about Mimir, Loki, Tempo, and Grafana. The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. You can also push the data from the environment to [Grafana Cloud](https://grafana.com/cloud/). # Loki Tutorial # Loki Tutorial This quickstart guide will walk you through deploying Loki in single binary mode (also known as [monolithic mode](https://grafana.com/docs/loki//get-started/deployment-modes/#monolithic-mode)) using Docker Compose. Grafana Loki is only one component of the Grafana observability stack for logs. In this tutorial we will refer to this stack as the **Loki stack**. {{< figure max-width="100%" src="/media/docs/loki/getting-started-loki-stack-3.png" caption="Loki stack" alt="Loki stack" >}} The Loki stack consists of the following components: * **Alloy**: [Grafana Alloy](https://grafana.com/docs/alloy/latest/) is an open source telemetry collector for metrics, logs, traces, and continuous profiles. In this quickstart guide Grafana Alloy has been configured to tail logs from all Docker containers and forward them to Loki. * **Loki**: A log aggregation system to store the collected logs. For more information on what Loki is, see the [Loki overview](https://grafana.com/docs/loki//get-started/overview/). * **Grafana**: [Grafana](https://grafana.com/docs/grafana/latest/) is an open-source platform for monitoring and observability. Grafana will be used to query and visualize the logs stored in Loki. ## Before you begin Before you start, you need to have the following installed on your local system: - Install [Docker](https://docs.docker.com/install) - Install [Docker Compose](https://docs.docker.com/compose/install) {{< admonition type="tip" >}} Alternatively, you can try out this example in our interactive learning environment: [Loki Quickstart Sandbox](https://killercoda.com/grafana-labs/course/loki/loki-getting-started-tutorial). It's a fully configured environment with all the dependencies already installed. ![Interactive](/media/docs/loki/loki-ile.svg) Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). {{< /admonition >}} ## Deploy the Loki stack {{< admonition type="note" >}} This quickstart assumes you are running Linux or MacOS. Windows users can follow the same steps using [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install). {{< /admonition >}} **To deploy the Loki stack locally, follow these steps:** 1. Clone the Loki fundamentals repository and check out the getting-started branch: ```bash git clone https://github.com/grafana/loki-fundamentals.git -b getting-started ``` 1. Change to the `loki-fundamentals` directory: ```bash cd loki-fundamentals ``` 1. With `loki-fundamentals` as the current working directory deploy Loki, Alloy, and Grafana using Docker Compose: ```bash docker compose up -d ``` After running the command, you should see a similar output: ```console ✔ Container loki-fundamentals-grafana-1 Started 0.3s ✔ Container loki-fundamentals-loki-1 Started 0.3s ✔ Container loki-fundamentals-alloy-1 Started 0.4s ``` 1. With the Loki stack running, you can now verify each component is up and running: * **Alloy**: Open a browser and navigate to [http://localhost:12345/graph](http://localhost:12345/graph). You should see the Alloy UI. * **Grafana**: Open a browser and navigate to [http://localhost:3000](http://localhost:3000). You should see the Grafana home page. * **Loki**: Open a browser and navigate to [http://localhost:3100/metrics](http://localhost:3100/metrics). You should see the Loki metrics page. Since Grafana Alloy is configured to tail logs from all Docker containers, Loki should already be receiving logs. The best place to verify log collection is using the Grafana Logs Drilldown feature. To do this, navigate to [http://localhost:3000/drilldown](http://localhost:3000/drilldown). Select **Logs**. You should see the Grafana Logs Drilldown page. {{< figure max-width="100%" src="/media/docs/loki/get-started-drill-down.png" caption="Grafana Logs Drilldown" alt="Grafana Logs Drilldown" >}} If you have only the getting started demo deployed in your Docker environment, you should see three containers and their logs; `loki-fundamentals-alloy-1`, `loki-fundamentals-grafana-1` and `loki-fundamentals-loki-1`. In the `loki-fundamentals-loki-1` container, click **Show Logs** to drill down into the logs for that container. {{< figure max-width="100%" src="/media/docs/loki/get-started-drill-down-container.png" caption="Grafana Drilldown Service View" alt="Grafana Drilldown Service View" >}} We will not cover the rest of the Grafana Logs Drilldown features in this quickstart guide. For more information on how to use the Grafana Logs Drilldown feature, refer to [Get started with Grafana Logs Drilldown](https://grafana.com/docs/grafana/latest/explore/simplified-exploration/logs/get-started/). ## Collect logs from a sample application Currently, the Loki stack is collecting logs about itself. To provide a more realistic example, you can deploy a sample application that generates logs. The sample application is called **The Carnivorous Greenhouse**, a microservices application that allows users to login and simulate a greenhouse with carnivorous plants to monitor. The application consists of seven services: - **User Service:** Manages user data and authentication for the application. Such as creating users and logging in. - **Plant Service:** Manages the creation of new plants and updates other services when a new plant is created. - **Simulation Service:** Generates sensor data for each plant. - **WebSocket Service:** Manages the websocket connections for the application. - **Bug Service:** A service that when enabled, randomly causes services to fail and generate additional logs. - **Main App:** The main application that ties all the services together. - **Database:** A PostgreSQL database that stores user and plant data. The architecture of the application is shown below: {{< figure max-width="100%" src="/media/docs/loki/get-started-architecture.png" caption="Sample Microservice Architecture" alt="Sample Microservice Architecture" >}} To deploy the sample application, follow these steps: 1. With `loki-fundamentals` as the current working directory, deploy the sample application using Docker Compose: ```bash docker compose -f greenhouse/docker-compose-micro.yml up -d --build ``` {{< admonition type="note" >}} This may take a few minutes to complete since the images for the sample application need to be built. Go grab a coffee and come back. {{< /admonition >}} Once the command completes, you should see a similar output: ```console ✔ Container greenhouse-websocket_service-1 Started 0.7s ✔ Container greenhouse-db-1 Started 0.7s ✔ Container greenhouse-user_service-1 Started 0.8s ✔ Container greenhouse-bug_service-1 Started 0.8s ✔ Container greenhouse-plant_service-1 Started 0.8s ✔ Container greenhouse-simulation_service-1 Started 0.7s ✔ Container greenhouse-main_app-1 Started 0.7s ``` 2. To verify the sample application is running, open a browser and navigate to [http://localhost:5005](http://localhost:5005). You should see the login page for the Carnivorous Greenhouse application. Now that the sample application is running, run some actions in the application to generate logs. Here is a list of actions: 1. **Create a user:** Click **Sign Up** and create a new user. Add a username and password and click **Sign Up**. 2. **Login:** Use the username and password you created to login. Add the username and password and click **Login**. 3. **Create a plant:** Once logged in, give your plant a name, select a plant type and click **Add Plant**. Do this a few times if you like. Your greenhouse should look something like this: {{< figure max-width="100%" src="/media/docs/loki/get-started-greenhouse.png" caption="Greenhouse Dashboard" alt="Greenhouse Dashboard" >}} Now that you have generated some logs, you can return to the Grafana Logs Drilldown page [http://localhost:3000/drilldown](http://localhost:3000/drilldown). You should see seven new services such as `greenhouse-main_app-1`, `greenhouse-plant_service-1`, `greenhouse-user_service-1`, etc. ## Querying logs At this point, you have viewed logs using the Grafana Logs Drilldown feature. In many cases this will provide you with all the information you need. However, we can also manually query Loki to ask more advanced questions about the logs. This can be done via **Grafana Explore**. 1. Open a browser and navigate to [http://localhost:3000](http://localhost:3000) to open Grafana. 1. From the Grafana main menu, click the **Explore** icon (1) to open the Explore tab. To learn more about Explore, refer to the [Explore](https://grafana.com/docs/grafana/latest/explore/) documentation. {{< figure src="/media/docs/loki/grafana-query-builder-v2.png" caption="Grafana Explore" alt="Grafana Explore" >}} 1. From the menu in the dashboard header, select the Loki data source (2). This displays the Loki query editor. In the query editor you use the Loki query language, [LogQL](https://grafana.com/docs/loki//query/), to query your logs. To learn more about the query editor, refer to the [query editor documentation](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/). 1. The Loki query editor has two modes (3): - [Builder mode](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#builder-mode), which provides a visual query designer. - [Code mode](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/#code-mode), which provides a feature-rich editor for writing LogQL queries. Next we’ll walk through a few queries using the code view. 1. Click **Code** (3) to work in Code mode in the query editor. Here are some sample queries to get you started using LogQL. After copying any of these queries into the query editor, click **Run Query** (4) to execute the query. 1. View all the log lines which have the `container` label value `greenhouse-main_app-1`: ```logql {container="greenhouse-main_app-1"} ``` In Loki, this is a log stream. Loki uses [labels](https://grafana.com/docs/loki//get-started/labels/) as metadata to describe log streams. Loki queries always start with a label selector. In the previous query, the label selector is `{container="greenhouse-main_app-1"}`. 2. Find all the log lines in the `{container="greenhouse-main_app-1"}` stream that contain the string `POST`: ```logql {container="greenhouse-main_app-1"} |= "POST" ``` ### Extracting attributes from logs Loki by design does not force log lines into a specific schema format. Whether you are using JSON, key-value pairs, plain text, Logfmt, or any other format, Loki ingests these logs lines as a stream of characters. The sample application we are using stores logs in [Logfmt](https://brandur.org/logfmt) format: ```bash ts=2025-02-21 16:09:42,176 level=INFO line=97 msg="192.168.65.1 - - [21/Feb/2025 16:09:42] "GET /static/style.css HTTP/1.1" 304 -" ``` To break this down: - `ts=2025-02-21 16:09:42,176` is the timestamp of the log line. - `level=INFO` is the log level. - `line=97` is the line number in the code. - `msg="192.168.65.1 - - [21/Feb/2025 16:09:42] "GET /static/style.css HTTP/1.1" 304 -"` is the log message. When querying Loki, you can pipe the result of the label selector through a parser. This extracts attributes from the log line for further processing. For example, lets pipe `{container="greenhouse-main_app-1"}` through the `logfmt` parser to extract the `level` and `line` attributes: ```logql {container="greenhouse-main_app-1"} | logfmt ``` When you now expand a log line in the query result, you will see the extracted attributes. {{< admonition type="tip" >}} Before we move on to the next section, let's generate some error logs. To do this, enable the bug service in the sample application. This is done by setting the `Toggle Error Mode` to `On` in the Carnivorous Greenhouse application. This will cause the bug service to randomly cause services to fail. {{< /admonition >}} ### Advanced and Metrics Queries With Error Mode enabled the bug service will start causing services to fail, in these next few LogQL examples we will track down some of these errors. Lets start by parsing the logs to extract the `level` attribute and then filter for logs with a `level` of `ERROR`: ```logql {container="greenhouse-plant_service-1"} | logfmt | level="ERROR" ``` This query will return all the logs from the `greenhouse-plant_service-1` container that have a `level` attribute of `ERROR`. You can further refine this query by filtering for a specific code line: ```logql {container="greenhouse-plant_service-1"} | logfmt | level="ERROR", line="58" ``` This query will return all the logs from the `greenhouse-plant_service-1` container that have a `level` attribute of `ERROR` and a `line` attribute of `58`. LogQL also supports metrics queries. Metrics are useful for abstracting the raw log data aggregating attributes into numeric values. This allows you to utilize more visualization options in Grafana as well as generate alerts on your logs. For example, you can use a metric query to count the number of logs per second that have a specific attribute: ```logql sum(rate({container="greenhouse-plant_service-1"} | logfmt | level="ERROR" [$__auto])) ``` It worth changing the visualization from `lines` to `bars` to visualize the error rate over time since the error count is quite low. Another example is to get the top 10 services producing the highest rate of errors: ```logql topk(10, sum(rate({env="production"} | logfmt | level="ERROR" [5m])) by (service_name)) ``` {{< admonition type="note" >}} `service_name` is a label created by Loki when no service name is provided in the log line. It will use the container name as the service name. A list of all automatically generated labels can be found in [Labels](https://grafana.com/docs/loki/latest/get-started/labels/#default-labels-for-all-users). {{< /admonition >}} Finally, lets take a look at the total log throughput of each container in our production environment: ```logql sum by (service_name) (rate({env="production"} | logfmt [$__auto])) ``` This is made possible by the `service_name` label and the `env` label that we have added to our log lines. Note that `env` is a static label that we added to all log lines as they are processed by Alloy. ## A look under the hood At this point you will have a running Loki stack and a sample application generating logs. You have also queried Loki using Grafana Logs Drilldown and Grafana Explore. In this next section we will take a look under the hood to understand how the Loki stack has been configured to collect logs, the Loki configuration file, and how the Loki data source has been configured in Grafana. ### Grafana Alloy configuration Grafana Alloy is collecting logs from all the Docker containers and forwarding them to Loki. It needs a configuration file to know which logs to collect and where to forward them to. Within the `loki-fundamentals` directory, you will find a file called `config.alloy`: ```alloy // This component is responsible for discovering new containers within the Docker environment discovery.docker "getting_started" { host = "unix:///var/run/docker.sock" refresh_interval = "5s" } // This component is responsible for relabeling the discovered containers discovery.relabel "getting_started" { targets = [] rule { source_labels = ["__meta_docker_container_name"] regex = "/(.*)" target_label = "container" } } // This component is responsible for collecting logs from the discovered containers loki.source.docker "getting_started" { host = "unix:///var/run/docker.sock" targets = discovery.docker.getting_started.targets forward_to = [loki.process.getting_started.receiver] relabel_rules = discovery.relabel.getting_started.rules refresh_interval = "5s" } // This component is responsible for processing the logs (In this case adding static labels) loki.process "getting_started" { stage.static_labels { values = { env = "production", } } forward_to = [loki.write.getting_started.receiver] } // This component is responsible for writing the logs to Loki loki.write "getting_started" { endpoint { url = "http://loki:3100/loki/api/v1/push" } } // Enables the ability to view logs in the Alloy UI in realtime livedebugging { enabled = true } ``` This configuration file can be viewed visually via the Alloy UI at [http://localhost:12345/graph](http://localhost:12345/graph). {{< figure max-width="100%" src="/media/docs/loki/getting-started-alloy-ui.png" caption="Alloy UI" alt="Alloy UI" >}} In this view you can see the components of the Alloy configuration file and how they are connected: * **discovery.docker**: This component queries the metadata of the Docker environment via the Docker socket and discovers new containers, as well as providing metadata about the containers. * **discovery.relabel**: This component converts a metadata (`__meta_docker_container_name`) label into a Loki label (`container`). * **loki.source.docker**: This component collects logs from the discovered containers and forwards them to the next component. It requests the metadata from the `discovery.docker` component and applies the relabeling rules from the `discovery.relabel` component. * **loki.process**: This component provides stages for log transformation and extraction. In this case it adds a static label `env=production` to all logs. * **loki.write**: This component writes the logs to Loki. It forwards the logs to the Loki endpoint `http://loki:3100/loki/api/v1/push`. ### View Logs in realtime Grafana Alloy provides a built-in real time log viewer. This allows you to view current log entries and how they are being transformed via specific components of the pipeline. To view live debugging mode open a browser tab and navigate to: [http://localhost:12345/debug/loki.process.getting_started](http://localhost:12345/debug/loki.process.getting_started). ## Loki Configuration Grafana Loki requires a configuration file to define how it should run. Within the `loki-fundamentals` directory, you will find a file called `loki-config.yaml`: ```yaml auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 log_level: info grpc_server_max_concurrent_streams: 1000 common: instance_addr: 127.0.0.1 path_prefix: /tmp/loki storage: filesystem: chunks_directory: /tmp/loki/chunks rules_directory: /tmp/loki/rules replication_factor: 1 ring: kvstore: store: inmemory query_range: results_cache: cache: embedded_cache: enabled: true max_size_mb: 100 limits_config: metric_aggregation_enabled: true allow_structured_metadata: true volume_enabled: true retention_period: 24h # 24h schema_config: configs: - from: 2020-10-24 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h pattern_ingester: enabled: true metric_aggregation: loki_address: localhost:3100 ruler: enable_alertmanager_discovery: true enable_api: true frontend: encoding: protobuf compactor: working_directory: /tmp/loki/retention delete_request_store: filesystem retention_enabled: true ``` To summarize the configuration file: * **auth_enabled**: This is set to `false`, meaning Loki does not need a [tenant ID](https://grafana.com/docs/loki//operations/multi-tenancy/) for ingest or query. Note that this is not recommended for production environments. When deploying the Loki Helm chart, this is set to `true` by default. * **server**: Defines the ports Loki listens on, the log level, and the maximum number of concurrent gRPC streams. * **common**: Defines the common configuration for Loki. This includes the instance address, storage configuration, replication factor, and ring configuration. * **query_range**: This is configured to tell Loki to use inbuilt caching for query results. In production environments of Loki this is handled by a separate cache service such as memcached. * **limits_config**: Defines the global limits for all Loki tenants. This includes enabling specific features such as metric aggregation and structured metadata. Limits can be defined on a per tenant basis, however this is considered an advanced configuration and for most use cases the global limits are sufficient. * **schema_config**: Defines the schema configuration for Loki. This includes the schema version, the object store, and the index configuration. * **pattern_ingester**: Enables pattern ingesters which are used to discover log patterns. Mostly used by Grafana Logs Drilldown. * **ruler**: Enables the ruler component of Loki. This is used to create alerts based on log queries. * **frontend**: Defines the encoding format for the frontend. In this case it is set to `protobuf`. * **compactor**: Defines the compactor configuration. Used to compact the index and manage chunk retention. The above configuration file is a basic configuration file for Loki. For more advanced configuration options, refer to the [Loki Configuration](https://grafana.com/docs/loki//configuration/) documentation. ### Grafana Loki Data source The final piece of the puzzle is the Grafana Loki data source. This is used by Grafana to connect to Loki and query the logs. Grafana has multiple ways to define a data source: * **Direct**: This is where you define the data source in the Grafana UI. * **Provisioning**: This is where you define the data source in a configuration file and have Grafana automatically create the data source. * **API**: This is where you use the Grafana API to create the data source. In this case we are using the provisioning method. Instead of mounting the Grafana configuration directory, we have defined the data source in this portion of the `docker-compose.yml` file: ```yaml grafana: image: grafana/grafana:latest environment: - GF_FEATURE_TOGGLES_ENABLE=grafanaManagedRecordingRules - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_BASIC_ENABLED=false ports: - 3000:3000/tcp entrypoint: - sh - -euc - | mkdir -p /etc/grafana/provisioning/datasources cat < /etc/grafana/provisioning/datasources/ds.yaml apiVersion: 1 datasources: - name: Loki type: loki access: proxy orgId: 1 url: 'http://loki:3100' basicAuth: false isDefault: true version: 1 editable: true EOF /run.sh networks: - loki ``` The entrypoint overrides the default startup command to first create the datasource provisioning file `ds.yaml`, which configures Loki as the default datasource. It then calls `/run.sh`, the default Grafana startup script included in the `grafana/grafana` Docker image, which starts the Grafana server. Since Loki is running in the same Docker network as Grafana, the datasource URL uses the service name `loki`. ## What next? {{< docs/ignore >}} ### Back to docs Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/get-started/quick-start/). {{< /docs/ignore >}} You have completed the Loki Quickstart demo. So where to go next? Here are a few suggestions: * **Deploy:** Loki can be deployed in multiple ways. For production use cases we recommend deploying Loki via the [Helm chart](https://grafana.com/docs/loki//setup/install/helm/). * **Send Logs:** In this example we used Grafana Alloy to collect and send logs to Loki. However there are many other methods you can use depending upon your needs. For more information see [send data](https://grafana.com/docs/loki//send-data/). * **Query Logs:** LogQL is an extensive query language for logs and contains many tools to improve log retrieval and generate insights. For more information see the [Query section](https://grafana.com/docs/loki//query/). * **Alert:** Lastly you can use the ruler component of Loki to create alerts based on log queries. For more information see [Alerting](https://grafana.com/docs/loki//alert/). ### Complete metrics, logs, traces, and profiling example If you would like to run a demonstration environment that includes Mimir, Loki, Tempo, and Grafana, you can use [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mlt). It's a self-contained environment for learning about Mimir, Loki, Tempo, and Grafana. The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. You can also push the data from the environment to [Grafana Cloud](https://grafana.com/cloud/). # Operations # Manage authentication Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services. The simple scalable and microservices [deployment modes](https://grafana.com/docs/loki//get-started/deployment-modes/) require a reverse proxy to be deployed in front of Loki, to direct client API requests to the various components. By default the Loki Helm chart includes a default reverse proxy configuration, using an nginx container to handle routing traffic and authorization. A list of open-source reverse proxies you can use: - [HAProxy](https://docs.haproxy.org/ ) - [nginx](https://docs.nginx.com/nginx/) using their [guide on restricting access with HTTP basic authentication](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/) - [OAuth2 proxy](https://oauth2-proxy.github.io/oauth2-proxy/) - [Pomerium](https://www.pomerium.com/docs), which has a [guide for securing Grafana](https://www.pomerium.com/docs/guides/grafana) {{< admonition type="note" >}} Loki runs in multi-tenant mode by default (`auth_enabled: true`). In this mode, Loki requires the HTTP header `X-Scope-OrgID` to be set to a string identifying the tenant on every request; the responsibility of populating this value should be handled by the authenticating reverse proxy. A request that does not include this header fails with an HTTP `401 Unauthorized` error and the message `no org id`. To learn how to troubleshoot this error, refer to [No org ID](https://grafana.com/docs/loki//operations/troubleshooting/troubleshoot-operations/#error-no-org-id). For more information, read the [multi-tenancy](https://grafana.com/docs/loki//operations/multi-tenancy/) documentation.{{< /admonition >}} For information on configuring authentication for your log shipping agent, see the [Grafana Alloy documentation](https://grafana.com/docs/alloy/latest/). ## Mutual TLS Loki's HTTP and gRPC servers can require clients to present a valid TLS certificate before a connection is accepted. This is a form of client authentication that Loki supports natively, without a reverse proxy. Mutual TLS (mTLS) authenticates the client at the transport level. It does not populate `X-Scope-OrgID`, so you still need a reverse proxy, your log shipping agent, or your client to set that header when `auth_enabled: true`. To require client certificates, you must first enable TLS on the server by setting a certificate and a key, then set a client authentication policy and the certificate authority (CA) that signs your client certificates. For example: ```yaml server: http_tls_config: cert_file: /etc/loki/certs/server.crt key_file: /etc/loki/certs/server.key # Reject any client that does not present a certificate signed by this CA. client_auth_type: RequireAndVerifyClientCert client_ca_file: /etc/loki/certs/ca.crt ``` Use `client_ca_file` for the path to a CA certificate file, or `client_ca` to include the certificate contents directly in the configuration file. The `grpc_tls_config` block accepts the same options for the gRPC server. The equivalent command line flags are `-server.http-tls-client-auth` and `-server.http-tls-ca-path`, plus `-server.grpc-tls-client-auth` and `-server.grpc-tls-ca-path`. The supported values for `client_auth_type` are: * `NoClientCert` * `RequestClientCert` * `RequireAnyClientCert` * `VerifyClientCertIfGiven` * `RequireAndVerifyClientCert` Use `RequireAndVerifyClientCert` for the strictest setting. It rejects a client that sends no certificate, and it also rejects a certificate that the CA did not sign. The other values check less. For example, `VerifyClientCertIfGiven` allows a client that sends no certificate at all. For the full list of TLS options, refer to the [server block](https://grafana.com/docs/loki//configure/#server) of the configuration reference. ## Enable basic authentication for Loki using nginx This section describes the process of enabling basic authentication for Loki using [nginx](https://docs.nginx.com/nginx/). ### Prerequisites * A running Loki instance * A running nginx instance ### Configure nginx You must create a new nginx configuration file for the Loki instance. This example assumes the following: * nginx is running in `/opt/homebrew` * Loki is running on port 3100 on the local machine * Your Loki tenant id is `fake` * The configuration file is named `/opt/homebrew/etc/nginx/loki.conf` If you used different configuration parameters for Loki, adjust the examples to match your configuration. `loki.conf` configuration: ```conf upstream loki { server 127.0.0.1:3100; keepalive 15; } server { listen 80; server_name loki.localhost; auth_basic "loki auth"; auth_basic_user_file /opt/homebrew/etc/nginx/passwords; location / { proxy_read_timeout 1800s; proxy_connect_timeout 1600s; proxy_pass http://loki; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Keep-Alive"; proxy_set_header Proxy-Connection "Keep-Alive"; proxy_redirect off; } location /ready { proxy_pass http://loki; proxy_http_version 1.1; proxy_set_header Connection "Keep-Alive"; proxy_set_header Proxy-Connection "Keep-Alive"; proxy_redirect off; auth_basic "off"; } } ``` {{< admonition type="tip" >}} This example expects the client to send the `X-Scope-OrgID` header itself, in addition to the basic authentication credentials. If you run Loki with `auth_enabled: true`, you can instead have nginx set the tenant ID from the authenticated username. Add this line to the `location / {}` block: ```conf proxy_set_header X-Scope-OrgID $remote_user; ``` With this line, nginx overwrites any tenant header sent by the client, so a client cannot read or write the data of another tenant. This requires each username in your password file to match a Loki tenant ID. The Loki Helm chart's gateway uses this same pattern.{{< /admonition >}} This configuration must be included in your main nginx configuration, for example, by including it in `nginx.conf` like: ```yaml include /opt/homebrew/etc/nginx/loki.conf; ``` Restart the nginx server to ensure all configuration changes are updated. ### Validate your nginx configuration To validate the nginx configuration for Loki, you can send a `curl` request to two endpoints: * The `/ready` endpoint, which is not protected by a basic authentication mechanism. ```curl % curl -i http://loki.localhost/ready HTTP/1.1 200 OK Server: nginx/1.29.2 Date: Thu, 16 Oct 2025 14:28:31 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 6 Connection: keep-alive X-Content-Type-Options: nosniff ready ``` * The `/` endpoint, which is protected by a basic authentication mechanism. ```curl curl -i http://loki.localhost/ HTTP/1.1 401 Unauthorized Server: nginx/1.29.2 Date: Thu, 16 Oct 2025 14:32:43 GMT Content-Type: text/html Content-Length: 179 Connection: keep-alive WWW-Authenticate: Basic realm="loki auth" 401 Authorization Required

401 Authorization Required

nginx/1.29.2
``` ### Update passwords The password file can be seeded using whatever mechanism you may use for other web services. In this example, `htpasswd` is utilized: ``` % htpasswd -c /opt/homebrew/etc/nginx/passwords loki123 New password: Re-type new password: Adding password for user loki123 ``` Restart the nginx server to ensure all configuration changes are updated. ### Validate passwords Enter your password into a temporary file, such as: ``` % vi lokipw ``` Then, store it as an environment variable:: ``` % pass=$(cat lokipw) ``` You can validate basic authentication is then working by issuing a curl command to the protected resource: ```curl curl -i -u loki123:$pass -H "X-Scope-OrgID:fake" "http://loki.localhost/loki/api/v1/labels" HTTP/1.1 200 OK Server: nginx/1.29.2 Date: Thu, 16 Oct 2025 14:46:09 GMT Content-Type: application/json; charset=UTF-8 Content-Length: 21 Connection: keep-alive {"status":"success"} ``` # Manage large volume log streams with automatic stream sharding Automatic stream sharding can keep streams under a `desired_rate` by adding new labels and values to existing streams. When properly tuned, this can eliminate issues where log producers are rate limited due to the per-stream rate limit. Automatic stream sharding is enabled by default (`shard_streams.enabled` defaults to `true`). To tune it or confirm it's on: 1. Check or set the global [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config) in the Loki configuration file: ```yaml limits_config: shard_streams: enabled: true ``` 1. Optionally lower the `desired_rate` in bytes if you find that the system is still hitting the `per_stream_rate_limit`: ```yaml limits_config: shard_streams: enabled: true desired_rate: 2097152 #2MiB ``` The `desired_rate` defaults to `1536KB` (1.5MB), which is below the default `per_stream_rate_limit` of `3MB`. This gives Loki some headroom to shard a stream before it hits the per-stream rate limit. 1. Optionally enable `time_sharding_enabled` if you need to ingest old, out-of-order logs, for example during a backfill: ```yaml limits_config: shard_streams: enabled: true time_sharding_enabled: true time_sharding_ignore_recent: 40m ``` Time-based sharding adds a `__time_shard__` label to streams, splitting log entries into buckets based on their timestamp. Log entries with a timestamp newer than `time_sharding_ignore_recent` (40 minutes by default) are still ingested, but Loki does not add the `__time_shard__` label to them. This lets very old logs be ingested without triggering out-of-order errors. Refer to [How automatic stream sharding works](#how-automatic-stream-sharding-works) for more detail. 1. Optionally enable `logging_enabled` for debugging stream sharding. {{< admonition type="note" >}} This may affect the ingestion performance of Loki. {{< /admonition >}} ```yaml limits_config: shard_streams: enabled: true logging_enabled: true ``` ## When to use automatic stream sharding Large log streams present several problems for Loki, namely increased and uneven resource usage on Ingesters and Distributors. The general recommendation is to explore existing log streams for additional label values that are both useful for querying and sufficiently low cardinality. There are many cases, however, where no more labels can be extracted, or cardinality for a label is dangerously large. To protect itself from such volume leading to operational failure, Loki implements per-stream rate limits; but the result is that some data is lost. The per-stream limit also needs human intervention to change, which is not ideal when log volumes increase and decrease. Loki uses automatic stream sharding to avoid rate limiting and large streams for any log stream by ensuring it is close to a configured `desired_rate`. ## How automatic stream sharding works Loki supports two independent sharding mechanisms, rate-based sharding and time-based sharding. You can enable either one on its own, or enable both together. ### Rate-based sharding Rate-based sharding works by adding a new label, `__stream_shard__`, to streams and incrementing its value to try and keep all streams below a configured `desired_rate`. The feature adds a new API to Ingesters that reports the size of all existing log streams. Once per second, Distributors query the API to get a picture of all stream rates in the system. Distributors use the existing stream-rate data and a configured `desired_rate` to determine how many shards a given stream should have. The desired number of new log streams are created with the label `__stream_shard__` and logs are divided evenly among the streams. The once-per-second query interval is a Distributor setting, not a `shard_streams` limit. Use the `-distributor.rate-store.stream-rate-update-interval` flag, or the equivalent `rate_store` block in the Distributor configuration, to change how often Distributors refresh their view of stream rates. Because rate-based sharding is reactive and relies on successive calls to Ingesters, the view of current rates is always somewhat behind. As a result, the actual size of sharded streams will always be higher than the `desired_rate`. In practice, this is still sufficient to keep log producers from being rate limited by per-stream rate limits. ### Time-based sharding Time-based sharding works by adding a different label, `__time_shard__`, to streams. Loki calculates the value of this label from each log entry's timestamp, divided into buckets of `max_chunk_age`/2. This lets Loki accept out-of-order log entries that are older than would normally be allowed, because each time bucket becomes its own stream instead of extending an existing one. Loki does not apply time-based sharding to log entries with a timestamp newer than the configured `time_sharding_ignore_recent` value. Those entries are still ingested, but they keep their original labels, so that recent logs are not needlessly split into extra streams. If you enable both rate-based sharding and time-based sharding, Loki applies time-based sharding first, and then applies rate-based sharding to each of the resulting time-sharded streams. ## Automatic stream sharding metrics Use these metrics to help tune Loki so that it is sharding streams aggressively enough to avoid the per-stream rate limit: - `loki_rate_store_refresh_failures_total`: The total number of failed attempts to refresh the distributor's view of stream rates. - `loki_rate_store_streams`: The number of unique streams reported by all Ingesters. Sharded streams are reported as if they were unsharded. - `loki_rate_store_max_stream_shards`: The maximum number of shards for any tenant of the system. - `loki_rate_store_stream_shards`: A histogram of the distribution of shard counts across all streams. - `loki_rate_store_max_stream_rate_bytes`: The maximum stream size in bytes/second for any tenant of the system. Sharded streams are reported as if they are unsharded. - `loki_rate_store_max_unique_stream_rate_bytes`: The maximum size of any stream across all tenants. Stream shards are individually reported. - `loki_rate_store_stream_rate_bytes`: A histogram of the distribution of stream sizes across all tenants in bytes/second. - `loki_stream_sharding_count`: The total number of times that streams have been sharded. Useful for calculating the sharding rate. # Manage varying workloads at scale with autoscaling queriers A microservices deployment of a Loki cluster that runs on Kubernetes typically handles a workload that varies throughout the day. To make Loki easier to operate and optimize the cost of running Loki at scale, we have designed a set of resources to help you autoscale your Loki queriers. ## Prerequisites You need to run Loki in Kubernetes as a set of microservices. You need to use the query-scheduler. We recommend using [Kubernetes Event-Driven Autoscaling (KEDA)](https://keda.sh/) to configure autoscaling based on Prometheus metrics. Refer to [Deploying KEDA](https://keda.sh/docs/latest/deploy) to learn more about setting up KEDA in your Kubernetes cluster. ## Scaling metric Because queriers pull queries from the query-scheduler queue and process them on the querier workers, you should scale metrics based on: - The scheduler queue size. - The queries running in the queriers. The query-scheduler exposes the `loki_query_scheduler_inflight_requests` metric. It tracks the sum of queued queries plus the number of queries currently running in the querier workers. The following query is useful to scale queriers based on the inflight requests. ```promql sum( max_over_time( loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile=""}[] ) ) ``` Use the quantile (Q) and the range (R) parameters to fine-tune the metric. The higher Q is, the more sensitive the metric is to short-lasting spikes. As R increases, you can reduce the variation over time in the metric. A higher R-value helps avoid the autoscaler from modifying the number of replicas too frequently. In our experience, we have found that a Q of 0.75 and an R of 2 minutes work well. You can adjust these values according to your workload. `loki_query_scheduler_inflight_requests` only tracks a fixed set of quantiles: `0.5`, `0.75`, `0.8`, `0.9`, `0.95`, and `0.99`. If you query the metric with any other quantile value, Prometheus returns no data. Choose Q from this list. The metric already smooths its value over a 60 second sliding window before Prometheus scrapes it. Because of this, an R value below about one minute adds little extra smoothing. We recommend keeping R at one minute or higher. ## Cluster capacity planning To scale the Loki queries, you configure the following settings: - The threshold for scaling up and down - The scale down stabilization period - The minimum and the maximum number of queriers Querier workers process queries from the queue. You can configure each Loki querier to run several workers with the `-querier.max-concurrent` flag (`max_concurrent` in YAML configuration), which defaults to `4`. To reserve workforce headroom to address workload spikes, our recommendation is not to use more than 75% of the workers. For example, if you configure the Loki queriers to run 6 workers, set a threshold of `floor(0.75 * 6) = 4`. `-querier.max-concurrent` sets the total number of workers for a querier, but the querier splits that total across its connections to query-schedulers (or query-frontends). If the configured value is lower than the number of query-schedulers a querier connects to, each connection still gets at least one worker, so the real worker count can be higher than the configured value. Check the actual number with the `loki_querier_worker_concurrency` metric before you calculate the threshold. To determine the minimum number of queries that you should run, run at least one querier and determine the average number of inflight requests the system processes 75% of the time over seven days. The target utilization of the queries is 75%. So if we use 6 workers per querier, we will use the following query: ```promql clamp_min(ceil( avg( avg_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.75"}[7d]) ) / scalar(floor(vector(6 * 0.75))) ), 1) ``` The maximum number of queriers to run is equal to the number of queriers required to process all inflight requests 50% of the time during a seven-day timespan. As for the previous example, if each querier runs 6 workers, divide the inflight requests by 6. The resulting query becomes: ```promql ceil( max( max_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.5"}[7d]) ) / 6 ) ``` To minimize the scenario where Loki scales up shortly after scaling down, set a stabilization window for scaling down. ### KEDA configuration If you deploy Loki with the Helm chart, you don't need to write your own `KEDA ScaledObject`. The chart can generate one for you through the `querier.kedaAutoscaling` values, which already default to a Prometheus trigger on `loki_query_scheduler_inflight_requests` with a Q of 0.75 and an R of 2 minutes. Set `querier.kedaAutoscaling.enabled` to `true`, then set `minReplicas` and `maxReplicas` for your workload. You also need to set `defaults.kedaAutoscaling.prometheusAddress` to the address of your Prometheus server, because it is empty by default and the trigger cannot query the metric without it. To replace the default trigger, set `querier.kedaAutoscaling.triggers`. Refer to [Helm chart values](https://grafana.com/docs/loki//setup/install/helm/reference/#querier) for the full list of values. If you don't use the Helm chart, or need a trigger the chart doesn't support, follow the example below to write your own `KEDA ScaledObject`. This [KEDA ScaledObject](https://keda.sh/docs/latest/concepts/scaling-deployments/) example configures autoscaling for the querier deployment in the `loki-cluster` namespace. The example shows the minimum number of replicas set to 10 and the maximum number of replicas set to 50. Because each querier runs 6 workers, aiming to use 75% of those workers, the threshold is set to 4. The metric is served at `http://prometheus.default:9090/prometheus`. We configure a stabilization window of 30 minutes. ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: querier namespace: loki-cluster spec: maxReplicaCount: 50 minReplicaCount: 10 scaleTargetRef: kind: Deployment name: querier triggers: - metadata: metricName: querier_autoscaling_metric query: sum(max_over_time(loki_query_scheduler_inflight_requests{namespace="loki-cluster", quantile="0.75"}[2m])) serverAddress: http://prometheus.default:9090/prometheus threshold: "4" type: prometheus advanced: horizontalPodAutoscalerConfig: behavior: scaleDown: stabilizationWindowSeconds: 1800 ``` ## Prometheus alerting when at capacity Because the configured maximum might not be sufficient, a Prometheus alert can identify when the quantity of queriers has been at its configured maximum for an extended time. The following example specifies three hours (`3h`) as the extended time: ```yaml name: LokiAutoscalerMaxedOut expr: kube_horizontalpodautoscaler_status_current_replicas{namespace=~"loki-cluster"} == kube_horizontalpodautoscaler_spec_max_replicas{namespace=~"loki-cluster"} for: 3h labels: severity: warning annotations: description: HPA {{ $labels.namespace }}/{{ $labels.horizontalpodautoscaler }} has been running at max replicas for longer than 3h; this can indicate underprovisioning. summary: HPA has been running at max replicas for an extended time ``` # Block unwanted queries In certain situations, you may not be able to control the queries being sent to your Loki installation. These queries may be intentionally or unintentionally expensive to run, and they may affect the overall stability or cost of running your service. You can block queries using [per-tenant overrides](https://grafana.com/docs/loki//configure/#runtime-configuration-file), like so: ```yaml overrides: "tenant-id": blocked_queries: # block this query exactly - pattern: 'sum(rate({env="prod"}[1m]))' # block any query matching this regex pattern - pattern: '.*prod.*' regex: true # block all metric queries - types: metric # block any filter or limited queries matching this regex pattern - pattern: '.*prod.*' regex: true types: filter,limited # block any query that matches this query hash - hash: 2943214005 # hash of {stream="stdout",pod="loki-canary-9w49x"} types: filter,limited # block queries originating from specific sources via X-Query-Tags # Keys and values are matched case-insensitively. - pattern: '.*' # optional; if pattern and regex are omitted they will default to '.*' and true regex: true query_tags: source: grafana feature: beta ``` {{< admonition type="note" >}} Changes to these configurations **do not require a restart**; they are defined in the [runtime configuration file](https://grafana.com/docs/loki//configure/#runtime-configuration-file). {{< /admonition >}} The available query types are: - `metric`: a query with an aggregation, e.g. `sum(rate({env="prod"}[1m]))` - `filter`: a query with a log filter, e.g. `{env="prod"} |= "error"` - `limited`: a query without a filter or a metric aggregation The `hash` option uses a [32-bit FNV-1](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) hash of the query string, represented as a 32-bit unsigned integer. This can often be easier to use than query strings that are long or require lots of string escaping. A `query_hash` field is logged with every query request in the `query-frontend` and `querier` logs, for easy reference. Here's an example log line: ```logfmt level=info ts=2023-03-30T09:08:15.2614555Z caller=metrics.go:152 component=frontend org_id=29 latency=fast query="{stream=\"stdout\",pod=\"loki-canary-9w49x\"}" query_hash=2943214005 query_type=limited range_type=range ... ``` {{< admonition type="note" >}} The order of patterns is preserved, and Loki stops at the first rule whose `pattern` or `hash` matches the query text. If that rule's `types` or `query_tags` constraints then don't match, the query is **not** blocked, and Loki does not continue checking any later rules, even if a later rule would have matched and blocked the query. For example, with this configuration: ```yaml overrides: "tenant-id": blocked_queries: - pattern: '{env="prod"}' types: metric - pattern: '{env="prod"}' ``` A query for `{env="prod"}` (a `limited` query) matches the first rule's pattern, but not its `types: metric` constraint, so it is not blocked. Loki does not fall through to the second rule, which has no `types` restriction and would otherwise have blocked it. To block a query with more than one rule that share a pattern, combine the constraints into a single rule instead of relying on a later rule as a fallback. {{< /admonition >}} ## Observing blocked queries Blocked queries are logged, as well as counted in the `loki_blocked_queries` metric on a per-tenant basis. When a policy matches by pattern/hash/regex, Loki logs whether the query type and request tags matched that policy: ```logfmt level=warn msg="query blocker matched with regex policy" user=29 type=metric pattern=".*rate\\(.*\\).*" query="sum(rate({app=\"foo\"}[5m]))" typesMatched=true tagsMatched=false blocked=false ``` If tag constraints fail to match, Loki emits a debug log showing the missing key and the raw header value that was received: ```logfmt level=debug msg="query blocker tags mismatch: missing or mismatched key" key=feature tagsRaw="Source=grafana,Feature=alpha" ``` ## Scope Query blocking is enforced by the LogQL query engine when it evaluates a query. This covers: - Range and instant queries sent to the `/loki/api/v1/query` and `/loki/api/v1/query_range` API endpoints. - [Alerting and recording rules](https://grafana.com/docs/loki//alert/), in both local and remote rule evaluation modes. Remote evaluation sends the rule's query back through the query frontend and querier, so it is blocked the same way as an API query. Query blocking does **not** apply to: - Log tailing (the `/loki/api/v1/tail` endpoint). Tailing reads from the ingesters and the store without using the query engine, so no block policy is applied. - Metadata endpoints, such as `labels`, `series`, `index/stats`, `index/volume`, and `detected_fields`. These endpoints don't run a LogQL expression through the query engine, so there is nothing for the query blocker to match against. ## Tag-based blocking You can scope a blocked query rule to requests that include specific key=value pairs in the `X-Query-Tags` header. - Header format: `key=value` pairs separated by commas, for example: `Source=grafana,Feature=beta`. - Allowed characters are alphanumeric plus space, comma, equals, '@', '.', and '-'. Any other characters are replaced with `_`. - Parsing keeps only canonical `key=value` tokens; malformed tokens are ignored. - Matching rules: - Keys are matched case-insensitively (the server lowercases keys). - Values are matched case-insensitively. - All specified `tags:` pairs in the rule must be present in the request to apply the block. Examples: ```yaml overrides: tenant-a: blocked_queries: # Block only metric queries from a beta feature flag - types: metric query_tags: feature: beta # Combine with regex to narrow scope further - pattern: '.*rate\\(.*\\).*' regex: true query_tags: source: grafana ``` ### Ruler query tags When the ruler evaluates alerting and recording rules in remote evaluation mode (`-ruler.evaluation.mode=remote`), it automatically sets `X-Query-Tags` on the query it sends to `source=ruler,rule_name=,rule_type=`. You can match on these tags to block or scope a policy to rule evaluations specifically, for example: ```yaml overrides: "tenant-id": blocked_queries: # block all queries that come from rule evaluation - query_tags: source: ruler ``` {{< admonition type="note" >}} The ruler doesn't set `X-Query-Tags` in local evaluation mode, so you can't match on these tags there. Be careful when you match on `rule_name`, because a rule name that contains a comma or an equals sign doesn't survive tag parsing. A comma ends the value early, so `rule_name=my,rule` is read as `rule_name=my`. An equals sign makes the pair invalid, so `rule_name=a=b` is dropped completely. Refer to [Tag-based blocking](#tag-based-blocking) for the full parsing rules. {{< /admonition >}} # Manage bloom filter building and querying (Deprecated) {{< admonition type="warning" >}} In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is deprecated and will be removed in a future release. Engineering and on-call support is not available. No SLA is provided. Note that this feature is intended for users who are ingesting more than 75TB of logs a month, as it is designed to accelerate queries against large volumes of logs. {{< /admonition >}} Loki leverages [bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) to speed up queries by reducing the amount of data Loki needs to load from the store and iterate through. Loki is often used to run "needle in a haystack" queries; these are queries where a large number of log lines are searched, but only a few log lines match the query. Some common use cases are searching all logs tied to a specific trace ID or customer ID. An example of such queries would be looking for a trace ID on a whole cluster for the past 24 hours: ```logql {cluster="prod"} | traceID="3c0e3dcd33e7" ``` Without accelerated filtering, Loki downloads all the chunks for all the streams matching `{cluster="prod"}` for the last 24 hours and iterates through each log line in the chunks, checking if the [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata) key `traceID` with value `3c0e3dcd33e7` is present. With accelerated filtering, Loki is able to skip most of the chunks and only process the ones where we have a statistical confidence that the structured metadata pair might be present. To learn how to write queries to use bloom filters, refer to [Query acceleration](https://grafana.com/docs/loki//query/query_acceleration). ## Enable bloom filters {{< admonition type="warning" >}} Building and querying bloom filters are by design not supported in single binary deployment, it can only be used with distributed microservice mode. The reason is that bloom filters also come with a relatively high cost for both building and querying the bloom filters that only pays off at large scale deployments. {{< /admonition >}} To start building and using blooms you need to: - Deploy the [Bloom Planner and Builder](#bloom-planner-and-builder) components (as [microservices](https://grafana.com/docs/loki//get-started/deployment-modes/#microservices-mode) or via the [SSD](https://grafana.com/docs/loki//get-started/deployment-modes/#simple-scalable) `backend` target) and enable the components in the [Bloom Build config](https://grafana.com/docs/loki//configure/#bloom_build). - Deploy the [Bloom Gateway](#bloom-gateway) component (as a [microservice](https://grafana.com/docs/loki//get-started/deployment-modes/#microservices-mode) or via the [SSD](https://grafana.com/docs/loki//get-started/deployment-modes/#simple-scalable) `backend` target) and enable the component in the [Bloom Gateway config](https://grafana.com/docs/loki//configure/#bloom_gateway). - Enable blooms building and filtering for each tenant individually, or for all of them by default. ```yaml # Configuration block for the bloom creation. bloom_build: enabled: true planner: planning_interval: 8h builder: planner_address: bloom-planner..svc.cluster.local:9095 # Configuration block for bloom filtering. bloom_gateway: enabled: true client: addresses: dnssrvnoa+_bloom-gateway-grpc._tcp.bloom-gateway-headless..svc.cluster.local # Enable blooms creation and filtering for all tenants by default # or do it on a per-tenant basis. limits_config: bloom_creation_enabled: true bloom_split_series_keyspace_by: 1024 bloom_gateway_enable_filtering: true ``` For more configuration options refer to the [Bloom Gateway](https://grafana.com/docs/loki//configure/#bloom_gateway), [Bloom Build](https://grafana.com/docs/loki//configure/#bloom_build) and [per tenant-limits](https://grafana.com/docs/loki//configure/#limits_config) configuration docs. We strongly recommend reading the whole documentation for this deprecated feature before using it. ## Bloom Planner and Builder Building bloom filters from the chunks in the object storage is done by two components: the Bloom Planner and the Bloom Builder, where the planner creates tasks for bloom building, and sends the tasks to the builders to process and upload the resulting blocks. Bloom filters are grouped in bloom blocks spanning multiple streams (also known as series) and chunks from a given day. To learn more about how blocks and metadata files are organized, refer to the [Building blooms](#building-blooms) section below. The Bloom Planner runs as a single instance and calculates the gaps in fingerprint ranges for a certain time period for a tenant for which bloom filters need to be built. It dispatches these tasks to the available builders. The planner also applies the [blooms retention](#retention). {{< admonition type="warning" >}} Do not run more than one instance of the Bloom Planner. {{< /admonition >}} The Bloom Builder is a stateless horizontally scalable component and can be scaled independently of the planner to fulfill the processing demand of the created tasks. You can find all the configuration options for these components in the [Configure section for the Bloom Builder](https://grafana.com/docs/loki//configure/#bloom_build). Refer to the [Enable bloom filters](#enable-bloom-filters) section above for a configuration snippet enabling this feature. ### Retention The Bloom Planner applies bloom block retention on object storage. Retention is disabled by default. When enabled, retention is applied to all tenants. The retention for each tenant is the longest of its [configured](https://grafana.com/docs/loki//configure/#limits_config) general retention (`retention_period`) and the streams retention (`retention_stream`). For example, in the following example, tenant A has a bloom retention of 30 days, and tenant B a bloom retention of 40 days for the `{namespace="prod"}` stream. ```yaml overrides: "A": retention_period: 30d "B": retention_period: 30d retention_stream: - selector: '{namespace="prod"}' priority: 1 period: 40d ``` ### Planner and Builder sizing and configuration The single planner instance runs the planning phase for bloom blocks for each tenant in the given interval and puts the created tasks to an internal task queue. Builders process tasks sequentially by pulling them from the queue. The amount of builder replicas required to complete all pending tasks before the next planning iteration depends on the value of `-bloom-build.split-keyspace-by`, the number of tenants, and the log volume of the streams. The maximum block size is configured per tenant via `-bloom-build.max-block-size`. The actual block size might exceed this limit given that we append streams blooms to the block until the block is larger than the configured maximum size. The maximum size of a single stream's bloom is configured per tenant via `-bloom-build.max-bloom-size`. A stream whose bloom exceeds this size is discarded rather than added to a block. Blocks are created in memory and as soon as they are written to the object store they are freed. Chunks and TSDB files are downloaded from the object store to the file system. We estimate that builders are able to process 4MB worth of data per second per core. ### Planning strategies The Bloom Planner supports two strategies for splitting the bloom-building work into tasks, configured per tenant via `bloom_planning_strategy`: - `split_keyspace_by_factor` (default): splits the series keyspace into a fixed number of parts, controlled by `bloom_split_series_keyspace_by`. This is the strategy described in the sizing guidance above. - `split_by_series_chunks_size`: sizes each task by a target amount of chunk data instead of a fixed split count, controlled by `bloom_task_target_series_chunk_size`. A few additional per-tenant limits let you tune task scheduling, block encoding, and gateway prefetching: - `bloom_build_max_builders` limits how many builders can work on a tenant's tasks concurrently (`0` allows unlimited builders). - `bloom_build_builder_response_timeout` sets how long the planner waits for a builder to finish a task before requeuing it (`0` disables the timeout). - `bloom_build_task_max_retries` limits how many times a failed task is retried before it is abandoned. - `bloom_block_encoding` sets the compression algorithm used for bloom block pages. - `bloom_prefetch_blocks` prefetches blocks on Bloom Gateways as soon as they are built. Refer to the [per-tenant limits](https://grafana.com/docs/loki//configure/#limits_config) configuration docs for the full list of flags, defaults, and descriptions. ## Bloom Gateway Bloom Gateways handle chunks filtering requests from the [index gateway](https://grafana.com/docs/loki//get-started/components/#index-gateway). The service takes a list of chunks and a filtering expression and matches them against the blooms, filtering out those chunks not matching the given filter expression. This component is horizontally scalable and every instance only owns a subset of the stream fingerprint range for which it performs the filtering. The sharding of the data is performed on the client side using DNS discovery of the server instances and the [jumphash](https://arxiv.org/abs/1406.2294) algorithm for consistent hashing and even distribution of the stream fingerprints across Bloom Gateway instances. You can find all the configuration options for this component in the Configure section for the [Bloom Gateways](https://grafana.com/docs/loki//configure/#bloom_gateway). Refer to the [Enable bloom filters](#enable-bloom-filters) section above for a configuration snippet enabling this feature. ### Gateway sizing and configuration Bloom Gateways use their local file system as a Least Recently Used (LRU) cache for blooms that are downloaded from object storage. The size of the blooms depend on the ingest volume and number of unique structured metadata key-value pairs, as well as on build settings of the blooms, namely false-positive-rate. With default settings, bloom filters make up <1% of the raw structured metadata size. Since reading blooms depends heavily on disk IOPS, Bloom Gateways should make use of multiple, locally attached SSD disks (NVMe) to increase I/O throughput. Multiple directories on different disk mounts can be specified using the `-bloom.shipper.working-directory` [setting](https://grafana.com/docs/loki//configure/#storage_config) when using a comma separated list of mount points, for example: ```yaml -bloom.shipper.working-directory="/mnt/data0,/mnt/data1,/mnt/data2,/mnt/data3" ``` Bloom Gateways need to deal with relatively large files: the bloom filter blocks. Even though the binary format of the bloom blocks allows for reading them into memory in smaller pages, the memory consumption depends on the number of pages that are concurrently loaded into memory for processing. The product of three settings control the maximum amount of bloom data in memory at any given time: `-bloom-gateway.worker-concurrency`, `-bloom-gateway.block-query-concurrency`, and `-bloom.max-query-page-size`. Example, assuming 4 CPU cores: ```yaml -bloom-gateway.worker-concurrency=4 // 1x NUM_CORES -bloom-gateway.block-query-concurrency=8 // 2x NUM_CORES -bloom.max-query-page-size=64MiB 4 x 8 x 64MiB = 2048MiB ``` Here, the memory requirement for block processing is 2GiB. To get the minimum requirements for the Bloom Gateways, you need to double the value. ## Building blooms Bloom filters are built per stream and aggregated together into block files. Streams are assigned to blocks by their fingerprint, following the same ordering scheme as Loki’s TSDB and sharding calculation. This gives a data locality benefit when querying as streams in the same shard are likely to be in the same block. In addition to blocks, builders maintain a list of metadata files containing references to bloom blocks and the TSDB index files they were built from. Gateways and the planner use these metadata files to discover existing blocks. Every `-bloom-build.planner.interval`, the planner will load the latest TSDB files for all tenants for which bloom building is enabled, and compares the TSDB files with the latest bloom metadata files. If there are new TSDB files or any of them have changed, the planner will create a task for the streams and chunks referenced by the TSDB file. The builder pulls a task from the planner's queue and processes the containing streams and chunks. For a given stream, the builder will iterate through all the log lines inside its new chunks and build a bloom for the stream. In case of changes for a previously processed TSDB file, builders will try to reuse blooms from existing blocks instead of building new ones from scratch. The builder converts structured metadata from each log line of each chunk of a stream and appends the hash of each key, and key-value pair to the bloom, followed by the hashes combined with the chunk identifier. The first set of hashes allows gateways to skip whole streams, while the latter is for skipping individual chunks. For example, given structured metadata `foo=bar` in the chunk `c6dj8g`, we append to the stream bloom the following hashes: `hash("foo")`, `hash("foo=bar")`, `hash("c6dj8g" + "foo")` and `hash("c6dj8g" + "foo=bar")`. ## Query sharding Query acceleration does not just happen while processing chunks, but also happens from the query planning phase where the query frontend applies [query sharding](https://lokidex.com/posts/tsdb/#sharding). Loki 3.0 introduces a new [per-tenant configuration](https://grafana.com/docs/loki//configure/#limits_config) flag `tsdb_sharding_strategy` which defaults to computing shards as in previous versions of Loki by using the index stats to come up with the closest power of two that would optimistically divide the data to process in shards of roughly the same size. Unfortunately, the amount of data each stream has is often unbalanced with the rest, therefore, some shards end up processing more data than others. Query acceleration introduces a new sharding strategy: `bounded`, which uses blooms to reduce the chunks to be processed right away during the planning phase in the query frontend, as well as evenly distributes the amount of chunks each sharded query will need to process. # Configure caches to speed up queries Loki supports caching for query results, chunks, and index lookups to speed up query performance and reduce calls to the storage layer. Memcached is included in the Loki Helm chart and enabled by default for the `chunksCache` and `resultsCache`. This section describes the recommended Memcached configuration to enable caching for chunks and query results. ## Cache backends Every cache in Loki (`chunk_cache_config`, `results_cache`, and so on) can use one of three interchangeable backends: Memcached, Redis, or an embedded (in-process) cache. Memcached is the backend used throughout this page and is the recommended choice for production, multi-replica deployments, because it is shared across all instances of a component. Redis is supported as an alternative external cache. The embedded cache stores data in the memory of the Loki process itself, so it isn't shared between replicas; it is best suited to single-binary or small deployments. Configure only one backend per cache. If you configure both Memcached and Redis for the same cache, Loki fails to start. If you don't configure Memcached or Redis for a cache, Loki automatically enables the embedded cache for it, so that caching still works out of the box. This applies to the chunks cache and to the results caches. It does not apply to the index queries cache or the write dedupe cache, which stay disabled unless you configure them. Because the embedded cache is per-process, this automatic fallback is not a substitute for Memcached or Redis in a deployment with more than one replica. For more information about which caching backend to use for each cache, refer to [Configure caches](https://grafana.com/docs/loki//configure/#cache_config) and [Components](https://grafana.com/docs/loki//get-started/components). For help diagnosing cache configuration problems, refer to [Troubleshoot operations](https://grafana.com/docs/loki//operations/troubleshooting/troubleshoot-operations/). ### Results cache The results cache stores results so that Loki can reuse them on subsequent queries, and it supports negative caching for log queries. It is sometimes called the frontend cache in some configurations. The results cache is consulted by query-frontends. If the cached results are incomplete, the query frontend calculates the required sub-queries and sends them further along to be executed in queriers, then also caches those results. To orchestrate all of the above, the results cache uses a query hash as the key that is computed and stored in the headers. The results cache is really a set of six independently configurable caches under `query_range`, one per query type: | Query type | Enable with | Cache config | | --- | --- | --- | | Metric and log queries | `cache_results` | `results_cache` | | Index-stats queries | `cache_index_stats_results` | `index_stats_results_cache` | | Volume queries | `cache_volume_results` | `volume_results_cache` | | Instant metric queries | `cache_instant_metric_results` | `instant_metric_results_cache` | | Series queries | `cache_series_results` | `series_results_cache` | | Label queries | `cache_label_results` | `label_results_cache` | If a per-query-type cache is enabled but its cache config is left empty, Loki falls back to the `results_cache` configuration for that query type. This means you only need to configure `results_cache` to get caching for all query types, and you only need to give a query type its own cache config if you want it to use a different Memcached, Redis, or embedded cache instance. For details of each supported request type, refer to the [Components section](https://grafana.com/docs/loki//get-started/components). The index lookup cache only supports the legacy BoltDB index storage and is configured to be in-memory by default. Since moving to the TSDB indexes the attached disks/persistent volumes are utilized as cache and in-memory index lookup cache is obsolete. ### Chunks cache The chunks are cached using the `chunkRef` as the cache key, which is the unique reference to a chunk when it's cut in the Loki ingesters. The chunk cache is consulted by queriers each time a set of `chunkRef`s are calculated to serve the query, before going to the storage layer. Query results are significantly smaller compared to chunks. As the Loki cluster gets bigger in ingested volume, the results cache can continue to perform, whereas the chunks cache will need to grow in proportion to demand more memory. To be able to support the growing needs of a cluster, in 2023 we introduced support for memcached-extstore. Extstore is an additional feature on Memcached which supports attaching SSD disks to memcached pods to maximize their capacity. Please see this [blog post](https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/) on Loki's experience with memcached-extstore for our SaaS offering, Grafana Cloud. For more information on how to tune memcached-extstore please consult the open source [memcached documentation](https://docs.memcached.org/advisories/grafanaloki/). ### L2 chunks cache As an alternative, or a complement, to memcached-extstore, Loki supports a second, "L2" chunk cache tier configured with `chunk_cache_config_l2` and `l2_chunk_cache_handoff`. The `l2_chunk_cache_handoff` setting is an age threshold. Chunks younger than this age are written to and read from the primary (L1) cache that you configure in `chunk_cache_config`. Chunks older than this age are written to and read from the L2 cache instead. A value of `0` disables the L2 cache. Loki does not copy or move entries between the two tiers. Each chunk is routed to one tier or the other based on its age at the time of the request. On the read path, Loki widens the threshold by 10% so that chunks near the boundary are still looked up in the L1 cache. This lets you keep a small, fast cache of recent chunks while routing older, less frequently accessed chunks to a larger and cheaper second tier, such as a disk-backed Memcached instance. ```yaml chunk_store_config: l2_chunk_cache_handoff: 24h chunk_cache_config_l2: memcached: batch_size: 256 parallelism: 10 memcached_client: host: service: ``` If the Helm chart is used, the L2 cache is configured under `chunksCache.l2`, which is disabled by default (`chunksCache.l2.enabled: false`) and defaults to a four day handoff (`chunksCache.l2.l2ChunkCacheHandoff: 345600s`). The Helm chart's `chunksCache.l2.persistence` settings can attach a persistent volume to the L2 cache pods, which is a chart-native alternative to memcached-extstore for adding disk-backed capacity. ## Before you begin - It is recommended to deploy separate Memcached type as separate components (`memcached_frontend` and `memcached_chunks`). - Use the Memcached image version shipped with the current release of the [Loki Helm chart](https://github.com/grafana-community/helm-charts/blob/main/charts/loki/values.yaml) (`memcached.image.tag`) rather than pinning to a specific version here, since the recommended version changes over time. - Consult the Loki ksonnet [memcached](https://github.com/grafana/loki/blob/main/production/ksonnet/loki/memcached.libsonnet) deployment and the ksonnet [memcached library](https://github.com/grafana/jsonnet-libs/tree/master/memcached). - Index caching is not required for the [TSDB](https://grafana.com/docs/loki//operations/storage/tsdb/#index-caching-not-required) index format. - For recommendations on scaling the cache, refer to the [Size the cluster](https://grafana.com/docs/loki//setup/size/) page. ## Steps To enable and configure Memcached: 1. Deploy each Memcached service with at least three replicas and configure each as follows: 1. Chunk cache ```yaml --memory-limit=4096 --max-item-size=2m --conn-limit=1024 ``` 1. Query result cache ```yaml --memory-limit=1024 --max-item-size=5m --conn-limit=1024 ``` 1. Configure Loki to use the cache. 1. If the Helm chart is used Set `chunksCache.addresses` to the Memcached address for the chunk cache and `resultsCache.addresses` to the Memcached address for the query result cache, then set `chunksCache.enabled=true` and `resultsCache.enabled=true`. Ensure that the connection limit of Memcached is at least `number_of_clients * max_idle_conns`. By default, the chart also deploys and manages the Memcached servers themselves. To use your own Memcached instances instead, set `memcached.enabled` to `false` to stop the chart from deploying its built-in Memcached server, keep `chunksCache.enabled` and `resultsCache.enabled` set to `true` so that Loki still uses a Memcached-based cache, and point the `addresses` values at your own service. Use the same comma-separated DNS Service Discovery format that the chart uses by default: ```yaml memcached: enabled: false chunksCache: enabled: true addresses: "dnssrvnoa+_memcached-client._tcp.chunk-cache-memcached.loki.svc" resultsCache: enabled: true addresses: "dnssrvnoa+_memcached-client._tcp.results-cache-memcached.loki.svc" ``` The Helm chart's `chunksCache` and `resultsCache` values only manage the chunk cache and the main query-range results cache. The other five results caches (`index_stats_results_cache`, `volume_results_cache`, `instant_metric_results_cache`, `series_results_cache`, and `label_results_cache`) don't have dedicated Helm values. You can still enable and configure them by adding the settings to the chart's `loki.query_range` value, which the chart copies into the Loki configuration file as it is written. For example: ```yaml loki: query_range: cache_index_stats_results: true cache_volume_results: true ``` 1. If the Loki configuration is used, modify the following two sections in the Loki configuration file. 1. Configure the chunk cache ```yaml chunk_store_config: chunk_cache_config: memcached: batch_size: 256 parallelism: 10 memcached_client: host: service: ``` 1. Configure the query result cache ```yaml query_range: cache_results: true results_cache: cache: memcached_client: consistent_hash: true host: service: max_idle_conns: 16 timeout: 200ms update_interval: 1m ``` # Kafka-based ingestion (Experimental) {{< admonition type="warning" >}} Kafka-based ingestion is an [experimental feature](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided. Configuration options and metric names may change in a future release. {{< /admonition >}} Grafana Loki can use Apache Kafka, or a Kafka-protocol-compatible system such as [WarpStream](https://www.warpstream.com/), as an optional durable buffer on the write path. This topic explains what Kafka-based ingestion is, when to use it, how it works, and how to configure and monitor it. ## What it is By default, the [distributor](https://grafana.com/docs/loki//get-started/components/#distributor) sends log streams directly to [ingesters](https://grafana.com/docs/loki//get-started/components/#ingester) over gRPC. When you enable Kafka-based ingestion, the distributor instead (or also) writes log streams as records to a Kafka topic. Each ingester then reads, or *consumes*, records from one partition of that topic and processes them the same way it processes directly pushed data. Kafka sits between the distributor and the ingester as a durable, replicated buffer. This decouples the two components so that a slow or restarting ingester doesn't block or fail writes. Loki uses the [franz-go](https://github.com/twmb/franz-go) Kafka client library. If you need to debug Kafka client behavior in your logs, look for the component name `kafka_client`. Kafka client metrics use the `loki_kafka_client_` prefix. ## When and why to use it Consider enabling Kafka-based ingestion if you want to: - Decouple ingestion durability from ingester availability. - Absorb ingester restarts and rolling updates without blocking or failing writes. - Automatically replay buffered data from the last committed offset when an ingester restarts, instead of relying only on the ingester's write-ahead log (WAL). - Scale write throughput independently of ingester replica count, by adding Kafka partitions. Kafka-based ingestion also underpins the ingest-limits service and the experimental data object builders used for columnar storage. These features require Kafka even if you don't otherwise need the durability benefits above. If you don't have an existing Kafka deployment and don't need these benefits, you can continue to use the default gRPC write path. Enabling Kafka adds an operational dependency: you must run, or pay for, a Kafka cluster (or a compatible managed service) and monitor it. ## Architecture summary {{< mermaid >}} flowchart LR A[Distributor] -->|produces records| B[(Kafka topic, partitioned)] B -->|partition 0| C1[Ingester A] B -->|partition 1| C2[Ingester B] B -->|partition N| C3[Ingester N] D[Partition ring] -.assigns partition ownership.-> C1 D -.-> C2 D -.-> C3 {{< /mermaid >}} The distributor writes each stream as one or more Kafka records to a topic. The topic has multiple partitions. Each ingester owns exactly one partition and consumes records only from that partition. Ingesters use the [partition ring](https://grafana.com/docs/loki//get-started/components/#kafka-based-ingestion-experimental) to coordinate which ingester owns which partition. Every ingester process always registers in the classic hash ring it uses for gRPC writes, whether or not Kafka-based ingestion is enabled. When you also enable `ingester.kafka_ingestion.enabled`, that same process additionally joins the partition ring. Loki supports two ways to deploy this: - **A single, shared ingester pool**: the same ingester replicas serve both the classic hash ring and the partition ring at once. This is the simplest setup, and works well if you're experimenting with Kafka-based ingestion or plan to cut over quickly. - **Two separate ingester pools**: a new, dedicated pool of Kafka-consuming ingesters runs alongside your existing ingesters, isolated from the classic hash ring by giving it a distinct `ingester.ring.kvstore.prefix`. This is the only path Grafana Labs has tested in production, and is the approach documented in [Migrate to Kafka-based ingestion](https://grafana.com/docs/loki//setup/migrate/migrate-to-kafka/). Because the two rings assign ownership of a tenant's streams differently (token-hash with replication, compared to shuffle-sharded partitions with a single owner), the querier has to be told explicitly which ring to use when it looks up ingesters. See [Read path](#read-path) and `querier.query_partition_ingesters` under [Configuration overview](#configuration-overview). {{< admonition type="note" >}} You can use any system that implements the Kafka wire protocol, such as WarpStream, instead of running Apache Kafka yourself. Loki only needs the broker address and, optionally, SASL credentials. No code changes are required. {{< /admonition >}} ## Read path Enabling Kafka-based ingestion primarily changes the **write path**. [LogQL](https://grafana.com/docs/loki//query/) queries still read recent data from ingester memory and older data from long-term object storage, the same way they do when Kafka-based ingestion is disabled. However, the querier must be told which ring to use to find the ingesters that hold recent data: the classic hash ring (the default), or the partition ring that Kafka-based ingestion uses. Set `querier.query_partition_ingesters: true` to switch the querier, and any locally evaluating rulers, over to the partition ring. Until you set this, the querier keeps using the classic ring even if the distributor and ingesters are fully configured for Kafka-based ingestion. For the full migration sequence and when to flip this setting, refer to [Migrate to Kafka-based ingestion](https://grafana.com/docs/loki//setup/migrate/migrate-to-kafka/). ## Configuration overview This section groups the relevant configuration blocks at a high level. For the full list of fields and their defaults, refer to the [Configuration reference](https://grafana.com/docs/loki//reference/loki-config-ref/). - **`kafka_config`**: the Kafka client configuration shared by every Loki component that reads or writes Kafka records. It includes the broker addresses (`reader_config.address` and `writer_config.address`), the topic name, SASL credentials, producer tuning (`producer_linger`, `producer_max_inflight_requests_per_broker`, `producer_max_record_size_bytes`), and consumer tuning (`max_consumer_lag_at_startup`, `consumer_group_offset_commit_interval`, `max_consumer_workers`). - **`distributor.kafka_writes_enabled`** and **`distributor.ingester_writes_enabled`**: control whether the distributor writes to Kafka, to ingesters over gRPC, or both. At least one must be `true`. - **`ingester.kafka_ingestion`**: controls whether ingesters consume from Kafka, and configures the partition ring key-value store that ingesters, distributors, queriers, and rulers use to coordinate partition ownership. - **`querier.query_partition_ingesters`**: controls whether the querier, and any ruler evaluating rules locally, looks up ingesters using the partition ring instead of the classic hash ring. Set this to `true` only after the Kafka-consuming ingesters have caught up and held data for at least `querier.query_ingesters_within` (default: 3 hours); otherwise recent queries can return incomplete results. Defaults to `false`. - **`limits_config.ingestion_partitions_tenant_shard_size`**: the number of Kafka partitions a single tenant's data is shuffle-sharded across. The default, `0`, means a tenant's data uses all partitions. - **`ingest_limits`**: configuration for the optional ingest-limits service, which uses its own Kafka topic, partition count, and consumer group to track stream metadata. ## Authentication limitations The only Kafka authentication mechanism Loki supports today is **SASL PLAIN**, configured with `kafka_config.sasl_username` and `kafka_config.sasl_password`. Both fields must be set together, or both left empty. Loki has no built-in Transport Layer Security (TLS) or mutual TLS (mTLS) configuration for the Kafka connection. If your organization requires an encrypted connection to Kafka, you must terminate TLS at the network layer, for example with a sidecar proxy or a load balancer in front of your Kafka brokers. ## Partition sizing guidance The number of Kafka partitions in your topic limits how many ingesters can actively consume from it, because each ingester owns exactly one partition. Keep the following in mind when you choose a partition count: - `kafka_config.auto_create_topic_default_partitions` defaults to `1000`. This is a cluster-wide Kafka broker setting (equivalent to Kafka's `num.partitions`), not a per-topic setting, and it only takes effect if you let Loki auto-create the topic. - The partition count sets the maximum number of ingesters that can actively consume from the topic at once. If you have more ingesters than partitions, the extra ingesters have no partition to own and stay idle for Kafka-based ingestion. - `limits_config.ingestion_partitions_tenant_shard_size` controls how many partitions a single tenant's streams are shuffle-sharded across. The default, `0`, disables shuffle sharding, so a tenant's data spreads across all partitions. - Start with more partitions than you expect to need. Partitions are inexpensive to create in Kafka, but reducing the partition count of an existing topic later is difficult and usually requires creating a new topic. ## Offset management and restart behavior When Kafka-based ingestion is enabled, each ingester follows this sequence to safely resume consuming after a restart: 1. On startup, the ingester fetches the last offset it committed for its partition. 1. If no committed offset exists, for example on first startup, the ingester starts from the earliest available offset. 1. The ingester replays Kafka records from that offset until its consumption lag is within `kafka_config.max_consumer_lag_at_startup` (default: 15 seconds). 1. Only after catching up does the ingester mark itself `ACTIVE` in the partition ring and pass its readiness check. 1. During normal operation, the ingester commits its current offset to Kafka every `kafka_config.consumer_group_offset_commit_interval` (default: 1 second). Because of this sequence, ingester restarts are safe: any data produced to Kafka while an ingester was down is still in the topic, and the ingester automatically replays it after it comes back up. ## Monitoring Kafka ingestion This is the full list of Kafka-related metrics, organized by component. For the highest-signal subset of these metrics and example alerting queries, refer to [Key metrics for monitoring Loki](https://grafana.com/docs/loki//operations/meta-monitoring/metrics/#kafka-based-ingestion-experimental). **Distributor (producer side):** - `loki_distributor_kafka_appends_total` (labels: `partition`, `status`): total number of Kafka produce attempts. - `loki_distributor_kafka_latency_seconds`: end-to-end produce latency. - `loki_distributor_kafka_sent_bytes_total`: bytes written to Kafka. - `loki_distributor_kafka_records_per_write_request`: how many Kafka records each incoming push request was split into. **Ingester (consumer side):** - `loki_ingester_partition_records_batch_process_duration_seconds`: time to process a batch of Kafka records. - `loki_ingester_partition_current_offset`: the offset the ingester has most recently consumed. Useful for lag alerting. - `loki_ingester_partition_push_latency_seconds`: latency of the internal push after consuming a record from Kafka. - `loki_ingester_partition_consume_workers_count`: number of consumer worker goroutines. **Kafka client internals (reader and writer):** - `loki_kafka_client_partition_reader_consumption_lag_seconds` (labels: `phase`, `partition_state`): estimated time the reader is behind the latest produced record. Alert if this grows over time. - `loki_kafka_client_partition_reader_fetch_errors_total`: errors fetching records from Kafka brokers. - `loki_kafka_client_partition_reader_fetches_total` and `loki_kafka_client_partition_reader_records_per_fetch`: fetch throughput. - `loki_kafka_client_partition_reader_offset_commit_requests_total` and `loki_kafka_client_partition_reader_offset_commit_failures_total`: health of offset commits to Kafka. - `loki_kafka_client_produce_write_wait_seconds`, `loki_kafka_client_produce_write_time_seconds`, `loki_kafka_client_produce_read_wait_seconds` and `loki_kafka_client_produce_read_time_seconds` (labels: `node_id`, `host`): native histograms of how long each stage of a Produce request to each broker takes: waiting to be written, writing, waiting for the response, and reading the response. ## Configuration examples ### Dual-write for a safe rollout Write to both Kafka and ingesters at the same time so you can validate Kafka-based ingestion before relying on it. Queries still use the classic hash ring at this stage (`querier.query_partition_ingesters` stays `false`). ```yaml distributor: kafka_writes_enabled: true ingester_writes_enabled: true ingester: kafka_ingestion: enabled: true kafka_config: topic: loki-logs reader_config: address: kafka:9092 writer_config: address: kafka:9092 ``` ### Kafka-only write path Once you've validated dual-write and switched queriers to the partition ring, you can disable the direct gRPC writes to ingesters. Refer to [Migrate to Kafka-based ingestion](https://grafana.com/docs/loki//setup/migrate/migrate-to-kafka/) for the full, safely ordered sequence. ```yaml distributor: kafka_writes_enabled: true ingester_writes_enabled: false ingester: kafka_ingestion: enabled: true querier: query_partition_ingesters: true kafka_config: topic: loki-logs reader_config: address: kafka:9092 writer_config: address: kafka:9092 ``` ### Managed Kafka with SASL authentication This example connects to a managed Kafka-protocol-compatible service, such as WarpStream, using SASL PLAIN authentication and separate reader and writer broker addresses. ```yaml distributor: kafka_writes_enabled: true ingester_writes_enabled: true ingester: kafka_ingestion: enabled: true kafka_config: topic: loki-logs reader_config: address: reader.kafka.example.com:9092 writer_config: address: writer.kafka.example.com:9092 sasl_username: ${KAFKA_SASL_USERNAME} sasl_password: ${KAFKA_SASL_PASSWORD} ``` ## Links - [Migrate to Kafka-based ingestion](https://grafana.com/docs/loki//setup/migrate/migrate-to-kafka/) - [Configuration reference](https://grafana.com/docs/loki//reference/loki-config-ref/) - [Troubleshoot Kafka integration errors](https://grafana.com/docs/loki//operations/troubleshooting/troubleshoot-operations/#kafka-integration-errors) # Use label-based access control with Loki {{< admonition type="note" >}} This feature is generally available from v4.0. For the latest releases, refer to the [Release notes](https://grafana.com/docs/loki//release-notes/). {{< /admonition >}} Label-based access control (LBAC) restricts the logs a tenant can query to those that match one or more [Prometheus label selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). You associate a set of selectors with a tenant, and queries from that tenant only return data from log streams that match at least one of the selectors. Because selectors can be combined into "OR" groups, where each group is a list of things joined by "AND" you can express any required policy. For example, `(A and B and C) or (D and E) or (F)`. LBAC builds on Loki's multi-tenancy: policies are scoped per tenant, so you must run Loki in multi-tenant mode. For more information, refer to the [multi-tenancy](https://grafana.com/docs/loki//operations/multi-tenancy/) documentation. ## Why use LBAC LBAC lets you share a single Loki tenant across multiple teams or users while still limiting what each of them can see. Common reasons to use LBAC include: - **Restrict a tenant to a subset of its own data.** For example, one team should only see logs from its own Kubernetes namespace, even though the tenant contains logs from many namespaces. - **Exclude sensitive log streams from some users.** For example, you can hide streams labeled `secret="true"` from most users, while still letting a smaller group of users query them. - **Enforce access boundaries per tenant without creating a separate Loki tenant for every team.** Running one tenant per team adds operational overhead. LBAC lets you keep a single tenant and instead scope access with label policies. ## Security model Loki does not include its own authentication layer. Both the tenant (`X-Scope-OrgID`) and the label policy (`X-Prom-Label-Policy`) are read from trusted HTTP headers on incoming requests. {{< admonition type="warning" >}} Never expose the Loki API directly to clients. A client that can reach Loki can set the `X-Scope-OrgID` and `X-Prom-Label-Policy` headers itself and bypass label-based access control entirely. You must run an authentication gateway in front of Loki that authenticates the request, sets the tenant, and attaches the correct label policy. {{< /admonition >}} The gateway is responsible for stripping any user supplied `X-Scope-OrgID` and `X-Prom-Label-Policy` and replacing them with values it derives from the authenticated identity. One open-source gateway that sets these headers is [db-auth-gateway](https://github.com/grafana/db-auth-gateway/). For general guidance on putting an authenticating reverse proxy in front of Loki, refer to the [authentication](https://grafana.com/docs/loki//operations/authentication/) documentation. ## How LBAC works At a high level, your gateway attaches a label policy to each request, and Loki carries that policy through the read path so that every component filters data to match it. 1. Your authentication gateway authenticates the request and sets the `X-Prom-Label-Policy` HTTP header. Each value in the header has the form `:`, for example `tenant1:%7Benv%3D%22dev%22%7D` for the selector `{env="dev"}`. 2. An HTTP middleware inside Loki parses this header on every incoming request and stores the resulting policies in the request context. 3. As the request moves between Loki's internal components, for example from the query frontend to the queriers, ingesters, and index gateway, the policies travel with it as gRPC metadata. This means the same policy applies no matter which component ultimately reads the data. 4. Loki enforces the policy at several points in the read path: - It filters chunks and streams while running the query, for both the original query engine and the newer query engine. - It filters the results of label-values queries against the store and the ingesters. - In the query frontend, it filters the response of volume queries, and it rewrites aggregated-metrics queries so they only return permitted data. The following diagram shows the request flow for a query from a client through the gateway and into Loki: ```mermaid sequenceDiagram participant Client participant Gateway as Authentication gateway participant QF as Query frontend participant Q as Querier / Ingester Client->>Gateway: Query request Gateway->>Gateway: Authenticate request Gateway->>QF: Query request + X-Scope-OrgID + X-Prom-Label-Policy QF->>Q: Query request (policy propagated over gRPC) Q->>Q: Filter streams and chunks by policy Q->>QF: Filtered results QF->>Gateway: Filtered response Gateway->>Client: Filtered response ``` ### LBAC and ingestion Label-based access control is not enforced on write (push) requests. It only restricts what a tenant can read. A tenant that is allowed to write can push log lines with any labels, regardless of the label policy that applies to its queries. ### LBAC and alertmanager and ruler Label policies are not enforced by the Alertmanager or ruler HTTP endpoints. This means that the requests they serve contain everything for a particular tenant without applying label-based access control. For example, listing all rule groups in the ruler returns all rule groups for the tenant, even if a label selector in the policy would exclude some of the labels on the rules. However it does apply when you query the metrics those components generate, such as `ALERTS`. ### LBAC and querying LogQL queries use the following format: ```logql { log stream selector } | log pipeline ``` LBAC only applies to the log stream selector portion of a query, that is the part of the query that comes before the pipe (`|`). It does not apply to label filter expressions or other parts of the log pipeline. For more information, refer to the [Log stream selector](https://grafana.com/docs/loki//query/log_queries/#log-stream-selector) and [Label filter expression](https://grafana.com/docs/loki//query/log_queries/#label-filter-expression) sections in the documentation about [Log queries](https://grafana.com/docs/loki//query/log_queries/). ## Enable label-based access control LBAC requires multi-tenant mode, which is the default. Ensure authentication is enabled so that requests carry a tenant: ```yaml auth_enabled: true ``` Then enable LBAC: ```yaml lbac: enabled: true ``` This can also be set with the `-lbac.enabled` command-line flag. When enabled, Loki parses the `X-Prom-Label-Policy` header on incoming requests and enforces the policies it contains. ## Configuration reference LBAC currently has a single configuration setting. There are no per-tenant settings in Loki's YAML configuration: policies for each tenant arrive dynamically in the `X-Prom-Label-Policy` header, set by your authentication gateway, not through Loki's configuration file. ```yaml lbac: # Enables label based access control through the X-Prom-Label-Policy header. [enabled: | default = false] ``` | Setting | CLI flag | Default | Description | | --- | --- | --- | --- | | `lbac.enabled` | `-lbac.enabled` | `false` | Enables label-based access control through the `X-Prom-Label-Policy` header. This setting is [experimental](https://grafana.com/docs/release-life-cycle/). | ## Setting up a label policy A label policy is conveyed to Loki in the `X-Prom-Label-Policy` request header. This header is set by your authentication gateway, not by clients. Each header value has the form: ```http : ``` The selector is a standard label matcher set such as `{env="dev"}`, URL-encoded so that it is safe to carry in a header. For example, the policy `{env="dev"}` for tenant `tenant1` is encoded as: ```http X-Prom-Label-Policy: tenant1:%7Benv%3D%22dev%22%7D ``` To associate multiple selectors with a tenant, set multiple values. You can either repeat the header or separate the encoded policies with a comma (`,`). The selectors are combined with `OR` into groups where each group is a list of things joined by `AND`: a stream is returned if any of the streams matches any one of the selector groups. For example, a policy written `{a='true', b='true, c='true'} OR {x='true', y='true'}` reads as `{a='true' AND b='true'AND c='true'} OR {x='true'AND y='true'}`. ## Exclude a label One common use case for an LBAC policy is to exclude logs that have a specific label. For example, to exclude all log lines with the label `secret=true`, use a selector with `secret!="true"`: ```logql {secret!="true"} ``` ## Use multiple selectors To allow access to both the production and development environments while excluding logs with the label `secret=true` in the production environment, use multiple selectors: ```logql {secret!="true", env="prod"} {env="dev"} ``` These selectors enforce the policy as follows: * `{secret!="true", env="prod"}` matches and returns log lines from the production environment that do not have the `secret: true` label. * `{env="dev"}` matches and returns log lines from the development environment, even if they have the `secret: true` label. ## Real-world examples The following examples use labels that Loki assigns by default when logs are collected through OpenTelemetry, such as `service_name`, `k8s_namespace_name`, and `deployment_environment_name`. For the full list of default OpenTelemetry-derived labels, refer to [Default labels](https://grafana.com/docs/loki//get-started/labels/#default-labels-for-opentelemetry). Each example shows the LogQL selector and the resulting `X-Prom-Label-Policy` header value that your gateway would set, following the same format as [Setting up a label policy](#setting-up-a-label-policy). ### Restrict a team to one namespace Scope a tenant so that it can only see logs from the `checkout` Kubernetes namespace. This is useful when a platform team gives a single application team access to a shared tenant, but the application team should only see its own namespace. Selector: ```logql {k8s_namespace_name="checkout"} ``` Header: ```http X-Prom-Label-Policy: team-checkout:%7Bk8s_namespace_name%3D%22checkout%22%7D ``` ### Restrict by environment Give separate policies to a production support team and a staging support team that share the same tenant. Production team selector: ```logql {deployment_environment_name="production"} ``` Header: ```http X-Prom-Label-Policy: team-prod:%7Bdeployment_environment_name%3D%22production%22%7D ``` Staging team selector: ```logql {deployment_environment_name="staging"} ``` Header: ```http X-Prom-Label-Policy: team-staging:%7Bdeployment_environment_name%3D%22staging%22%7D ``` ### Restrict by service, with an exclusion Combine a service restriction with an exclusion, similar to [Use multiple selectors](#use-multiple-selectors), but scoped to a specific service. Allow access to the `payments-api` service, but exclude any of its streams labeled `secret="true"`, and separately allow access to every other service: ```logql {service_name="payments-api", secret!="true"} {service_name!="payments-api"} ``` Header: ```http X-Prom-Label-Policy: team-payments:%7Bservice_name%3D%22payments-api%22%2C%20secret%21%3D%22true%22%7D,team-payments:%7Bservice_name%21%3D%22payments-api%22%7D ``` ### Restrict by region for a distributed team Scope a regional on-call team so that it only sees logs from its own region. Selector: ```logql {cloud_region="us-west-1"} ``` Header: ```http X-Prom-Label-Policy: team-uswest:%7Bcloud_region%3D%22us-west-1%22%7D ``` ### Combine selectors across multiple namespaces Grant a tenant access to logs from two Kubernetes namespaces by giving it two separate selectors for the same tenant: ```logql {k8s_namespace_name="frontend"} {k8s_namespace_name="checkout"} ``` Because a tenant's selectors are combined with OR, a stream is returned if it matches either one: the `frontend` namespace or the `checkout` namespace. Header: ```http X-Prom-Label-Policy: team-multi:%7Bk8s_namespace_name%3D%22frontend%22%7D,team-multi:%7Bk8s_namespace_name%3D%22checkout%22%7D ``` # Manage tenant isolation Grafana Loki is a multi-tenant system; requests and data for tenant A are isolated from tenant B. Requests to the Loki API should include an HTTP header (`X-Scope-OrgID`) that identifies the tenant for the request. Tenant IDs must not be longer than 150 bytes and can only include a specific set of characters; see [Restrictions](#restrictions) below for the full rules. Operators are recommended to use a reasonable limit for uniquely identifying tenants; 20 bytes is usually enough. Loki defaults to running in multi-tenant mode. Multi-tenant mode is set in the configuration with `auth_enabled: true`, or with the equivalent command line flag `-auth.enabled` (default `true`). When configured with `auth_enabled: false`, Loki uses a single tenant. The `X-Scope-OrgID` header is not required in Loki API requests. The single tenant ID defaults to `fake` and can be changed via the `no_auth_tenant` configuration option. On a fresh cluster with an empty bucket this is safe to change at any time; on a cluster that has already written data, changing the tenant ID requires migrating existing data to the new tenant path (see [Loki Migrate Tool](https://github.com/grafana/loki/tree/main/cmd/migrate)). When `auth_enabled: true`, a request that does not include the `X-Scope-OrgID` header fails with an HTTP `401 Unauthorized` error and the message `no org id`. For more information about setting this header, including examples of configuring a reverse proxy to add it, refer to [Manage authentication](https://grafana.com/docs/loki//operations/authentication/). To troubleshoot this error, refer to [No org ID](https://grafana.com/docs/loki//operations/troubleshooting/troubleshoot-operations/#error-no-org-id). ## Multi-tenant Queries In multi-tenant mode, queries may gather results from multiple tenants. Set the querier configuration option `multi_tenant_queries_enabled: true`, or the equivalent command line flag `-querier.multi-tenant-queries-enabled` (default `false`), to enable queries across tenants. The query API request defines the tenants. Specify multiple tenants in the query request HTTP header `X-Scope-OrgID` by separating the tenant IDs with the pipe character (`|`). For example, a query for tenants `A` and `B` requires the header `X-Scope-OrgID: A|B`. Only query endpoints support multi-tenant calls. Calls to `GET /loki/api/v1/tail`, `POST /loki/api/v1/push`, and `POST /otlp/v1/logs` will return an HTTP 400 error if more than one tenant is defined in the HTTP header. Instant and range queries support label filtering using tenant IDs. For example, the query ```logql {app="foo", __tenant_id__=~"a.+"} | logfmt ``` will return results for all tenants that have a tenant ID that begins with the character `a`. The `__tenant_id__` label is not stored with the log data. Loki adds it to the results only when a query spans more than one tenant. If the request names a single tenant, the results do not include this label. If the label `__tenant_id__` is already present in a log stream, it is prepended with the string `original_`. Tenant ID filtering in stages is not supported. An example of a query that will _not_ work: ```logql {app="foo"} | __tenant_id__="1" | logfmt ``` ## Restrictions Tenant IDs must not be longer than 150 bytes and can only include the following characters: - Alphanumeric characters - `0-9` - `a-z` - `A-Z` - Special characters - Exclamation point (`!`) - Hyphen (`-`) - Underscore (`_`) - Single period (`.`) - Asterisk (`*`) - Single quote (`'`) - Open parenthesis (`(`) - Close parenthesis (`)`) {{< admonition type="note" >}} For security reasons, `.` and `..` aren't valid tenant IDs. {{< /admonition >}} {{< admonition type="warning" >}} Do not include a colon (`:`) in the `X-Scope-OrgID` header. A colon is not a valid tenant ID character. Most Loki components stop reading the tenant ID at the first colon and ignore the rest of the value, and the ignored part is not checked against the rules on this page. Other components read the whole header value instead, so a header that contains a colon can produce inconsistent results. {{< /admonition >}} # Monitor tenant limits using the Overrides Exporter Loki is a multi-tenant system that supports applying limits to each tenant as a mechanism for resource management. The `overrides-exporter` module exposes these limits as Prometheus metrics in order to help operators better understand tenant behavior. ## Context Configuration updates to tenant limits can be applied to Loki without restart via the [`runtime_config`](https://grafana.com/docs/loki//configure/#runtime_config) feature. ## Example The `overrides-exporter` module is disabled by default. We recommend running a single instance per cluster to avoid issues with metric cardinality. The `overrides-exporter` creates one metric for every scalar field in the limits configuration under the metric `loki_overrides_defaults` with the default value for that field after loading the Loki configuration. It also exposes another metric for _every_ differing field for _every_ tenant. Using an example `runtime.yaml`: ```yaml overrides: "tenant_1": ingestion_rate_mb: 10 max_streams_per_user: 100000 max_chunks_per_query: 100000 ``` Launch an instance of the `overrides-exporter`: ```shell loki -target=overrides-exporter -runtime-config.file=runtime.yaml -config.file=basic_schema_config.yaml -server.http-listen-port=8080 ``` To inspect the tenant limit overrides: ```shell $ curl -sq localhost:8080/metrics | grep override # HELP loki_overrides Resource limit overrides applied to tenants # TYPE loki_overrides gauge loki_overrides{limit_name="ingestion_rate_mb",user="tenant_1"} 10 loki_overrides{limit_name="max_chunks_per_query",user="tenant_1"} 100000 loki_overrides{limit_name="max_streams_per_user",user="tenant_1"} 100000 # HELP loki_overrides_defaults Default values for resource limit overrides applied to tenants # TYPE loki_overrides_defaults gauge loki_overrides_defaults{limit_name="block_ingestion_status_code"} 260 loki_overrides_defaults{limit_name="cardinality_limit"} 100000 loki_overrides_defaults{limit_name="creation_grace_period"} 6e+11 loki_overrides_defaults{limit_name="ingestion_burst_size_mb"} 6 loki_overrides_defaults{limit_name="ingestion_rate_mb"} 4 loki_overrides_defaults{limit_name="max_cache_freshness_per_query"} 6e+11 loki_overrides_defaults{limit_name="max_chunks_per_query"} 2e+06 loki_overrides_defaults{limit_name="max_concurrent_tail_requests"} 10 loki_overrides_defaults{limit_name="max_entries_limit_per_query"} 5000 loki_overrides_defaults{limit_name="max_global_streams_per_user"} 5000 loki_overrides_defaults{limit_name="max_label_name_length"} 1024 loki_overrides_defaults{limit_name="max_label_names_per_series"} 15 loki_overrides_defaults{limit_name="max_label_value_length"} 2048 loki_overrides_defaults{limit_name="max_line_size"} 262144 loki_overrides_defaults{limit_name="max_metadata_cache_freshness"} 8.64e+13 loki_overrides_defaults{limit_name="max_queriers_per_tenant"} 0 loki_overrides_defaults{limit_name="max_query_capacity"} 0 loki_overrides_defaults{limit_name="max_query_length"} 2.5956e+15 loki_overrides_defaults{limit_name="max_query_lookback"} 0 loki_overrides_defaults{limit_name="max_query_parallelism"} 32 loki_overrides_defaults{limit_name="max_query_series"} 500 loki_overrides_defaults{limit_name="max_streams_matchers_per_query"} 1000 loki_overrides_defaults{limit_name="max_streams_per_user"} 0 loki_overrides_defaults{limit_name="max_structured_metadata_size"} 65536 loki_overrides_defaults{limit_name="min_sharding_lookback"} 0 loki_overrides_defaults{limit_name="per_stream_rate_limit"} 3.145728e+06 loki_overrides_defaults{limit_name="per_stream_rate_limit_burst"} 1.572864e+07 loki_overrides_defaults{limit_name="per_tenant_override_period"} 1e+10 loki_overrides_defaults{limit_name="query_timeout"} 6e+10 loki_overrides_defaults{limit_name="reject_old_samples_max_age"} 6.048e+14 loki_overrides_defaults{limit_name="retention_period"} 0 loki_overrides_defaults{limit_name="ruler_max_rule_groups_per_tenant"} 0 loki_overrides_defaults{limit_name="ruler_max_rules_per_rule_group"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_batch_send_deadline"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_capacity"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_backoff"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_samples_per_send"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_max_shards"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_backoff"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_queue_min_shards"} 0 loki_overrides_defaults{limit_name="ruler_remote_write_timeout"} 0 loki_overrides_defaults{limit_name="split_queries_by_interval"} 3.6e+12 loki_overrides_defaults{limit_name="tsdb_max_bytes_per_shard"} 6.291456e+08 loki_overrides_defaults{limit_name="tsdb_max_query_parallelism"} 128 loki_overrides_defaults{limit_name="volume_enabled"} 1 loki_overrides_defaults{limit_name="volume_max_series"} 1000 ``` {{< admonition type="note" >}} This output is a partial example. The exact set of metrics depends on your Loki version, since `loki_overrides_defaults` includes one series for every scalar field in the [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config) block, and that block grows with each release. Query your own instance's `/metrics` endpoint for the authoritative, current list rather than relying on a fixed example. {{< /admonition >}} Alerts can be created based on these metrics to inform operators when tenants are close to hitting their limits allowing for increases to be applied before the tenant limits are exceeded. # Manage recording rules Recording rules are queries that run in an interval and produce metrics from logs that can be pushed to a Prometheus compatible backend. Recording rules are evaluated by the `ruler` component. Each `ruler` acts as its own `querier`, in the sense that it executes queries against the store without using the `query-frontend` or `querier` components. It will respect all query [limits](https://grafana.com/docs/loki//configure/#limits_config) put in place for the `querier`. The Loki implementation of recording rules largely reuses Prometheus' code. Samples generated by recording rules are sent to Prometheus using Prometheus' **remote-write** feature. ## Write-Ahead Log (WAL) All samples generated by recording rules are written to a WAL. The WALs main benefit is that it persists the samples generated by recording rules to disk, which means that if your `ruler` crashes, you won't lose any data. We are trading off extra memory usage and slower start-up times for this functionality. A WAL is created per tenant; this is done to prevent cross-tenant interactions. If all samples were to be written to a single WAL, this would increase the chances that one tenant could cause data-loss for others. A typical scenario here is that Prometheus will, for example, reject a remote-write request with 100 samples if just 1 of those samples is invalid in some way. ### Start-up When the `ruler` starts up, it deletes (wipes) its remote-write WAL directory and starts each tenant with a fresh, empty WAL. WAL data left over from a previous run is not replayed, so any samples that had not yet been flushed to remote storage before the restart are discarded. ### Truncation WAL files are regularly truncated to reduce their size on disk. [This guide](https://ganeshvernekar.com/blog/prometheus-tsdb-wal-and-checkpoint/#wal-truncation-and-checkpointing) from one of the Prometheus maintainers (Ganesh Vernekar) gives an excellent overview of the truncation, checkpointing, and replaying of the WAL. ### Cleaner The WAL Cleaner watches for abandoned WALs (tenants who no longer have recording rules associated) and deletes them. Enable this feature only if you are running into storage concerns with WALs that are too large. WALs should not grow excessively large due to truncation. The WAL Cleaner is disabled by default. Enable it by setting `-ruler.wal-cleaner.period` to a non-zero value. ## Scaling See Mimir's guide for [configuring Grafana Mimir hash rings](https://grafana.com/docs/mimir/latest/configure/configure-hash-rings/) for scaling the ruler using a ring. {{< admonition type="note" >}} By default, the `ruler` shards by rule _group_, not by individual rules. This is an artifact of the fact that Prometheus recording rules need to run in order since one recording rule can reuse another, but this is not possible in Loki. To shard by individual rule instead, set `-ruler.sharding-algo="by-rule"`. For details, refer to [Tuning](#tuning). {{< /admonition >}} ## Deployment The `ruler` needs to persist its WAL files to disk, and it incurs a bit of a start-up cost by reading these WALs into memory. As such, it is recommended that you try to minimize churn of individual `ruler` instances, since recording rule samples produced while a tenant's WAL is being read from disk are dropped rather than persisted (see [Start-up](#start-up)). ### Kubernetes It is recommended that you run the `rulers` using `StatefulSets`. The `ruler` will write its WAL files to persistent storage, so a `Persistent Volume` should be utilized. ## Remote-Write ### Client configuration Remote-write client configuration is fully compatible with [prometheus configuration format](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). ```yaml remote_write: clients: mimir: url: http://mimir/api/v1/push write_relabel_configs: - action: replace target_label: job replacement: loki-recording-rules ``` ### Per-Tenant Limits Remote-write can be configured at a global level in the base configuration, and certain parameters tuned specifically on a per-tenant basis. Most of the configuration options [defined here](https://grafana.com/docs/loki//configure/#ruler) have [override options](https://grafana.com/docs/loki//configure/#limits_config) (which can be also applied at runtime!). ### Tuning Remote-write can be tuned if the default configuration is insufficient (see [Failure Modes](#failure-modes) below). There is a [guide](https://prometheus.io/docs/practices/remote_write/) on the Prometheus website, all of which applies to Loki, too. Rules can be evenly distributed across available rulers by using `-ruler.enable-sharding=true` and `-ruler.sharding-algo="by-rule"`. `-ruler.enable-sharding` and `-ruler.sharding-strategy` control which `ruler` instances own which tenants (ring-based sharding, with optional shuffle sharding across rulers). For details, refer to [Shuffle sharding](https://grafana.com/docs/loki//operations/shuffle-sharding/#shuffle-sharding-in-the-ruler). `-ruler.sharding-algo` controls how a single tenant's rule groups are split up for evaluation once sharding is enabled. Rule groups execute in order; this is a feature inherited from Prometheus' rule engine (which Loki uses), but Loki has no need for this constraint because rules cannot depend on each other. The default algorithm, `by-group`, shards by rule group, but this may be undesirable, since some rule groups could contain more expensive rules, which can lead to subsequent rules missing evaluations. The `by-rule` algorithm creates one rule group for each rule the ruler instance "owns" (based on its hash ring), and these groups are all executed concurrently. ## Observability Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus' observability. Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with `loki_ruler_wal_`. For example: `prometheus_remote_storage_bytes_total` → `loki_ruler_wal_prometheus_remote_storage_bytes_total` Additional metrics are exposed, also with the prefix `loki_ruler_wal_`. All per-tenant metrics contain a `tenant` label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large. Some key metrics to note are: - `loki_ruler_wal_appender_ready`: whether a WAL appender is ready to accept samples (1) or not (0) - `loki_ruler_wal_prometheus_remote_storage_samples_total`: number of samples sent per tenant to remote storage - `loki_ruler_wal_prometheus_remote_storage_samples_pending`: samples currently buffered in memory, waiting to be sent to remote storage. This is a gauge, not a counter, so don't use `rate()` or `increase()` with it. - `loki_ruler_wal_prometheus_remote_storage_samples_failed_total`: samples that failed when sent to remote storage - `loki_ruler_wal_prometheus_remote_storage_samples_dropped_total`: samples dropped by relabel configurations - `loki_ruler_wal_prometheus_remote_storage_samples_retried_total`: samples re-resent to remote storage - `loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds`: highest timestamp of sample appended to WAL - `loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds`: highest timestamp of sample sent to remote storage. We've created a basic [dashboard in our loki-mixin](https://github.com/grafana/loki/tree/main/production/loki-mixin/dashboards/recording-rules.libsonnet) which you can use to administer recording rules. ## Failure Modes ### Remote-Write Lagging Remote-write can lag behind for many reasons: 1. Remote-write storage (Prometheus) is temporarily unavailable 2. A tenant is producing samples too quickly from a recording rule 3. Remote-write is tuned too low, creating backpressure It can be determined by subtracting `loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds` from `loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds`. In case 1, the `ruler` will continue to retry sending these samples until the remote storage becomes available again. Be aware that if the remote storage is down for longer than `ruler.wal.max-age`, data loss may occur after truncation occurs. In cases 2 and 3, you should consider [tuning](#tuning) remote-write appropriately. Further reading: see [this blog post](https://grafana.com/blog/how-to-troubleshoot-remote-write-issues-in-prometheus/) by Prometheus maintainer Callum Styan. ### Appender Not Ready Each tenant's WAL has an "appender" internally; this appender is used to _append_ samples to the WAL. The appender is marked as _not ready_ until the WAL storage has been initialized upon startup. You can alert on `loki_ruler_wal_appender_ready < 1` to detect tenants whose WAL is not yet ready to accept samples. ### Corrupt WAL If a disk fails or the `ruler` does not terminate correctly, there's a chance one or more tenant WALs can become corrupted. The `ruler` wipes its WAL directory on startup, so any WAL left over from a previous run — corrupt or not — is discarded and recreated fresh rather than replayed. ### Found another failure mode? Open an [issue](https://github.com/grafana/loki/issues) and tell us about it! # Enforce rate limits and push request validation Loki will reject requests if they exceed a usage threshold (rate limit error) or if they are invalid (validation error). All occurrences of these errors can be observed using the `loki_discarded_samples_total` and `loki_discarded_bytes_total` metrics. The sections below describe the various possible reasons specified in the `reason` label of these metrics. It is recommended that Loki operators set up alerts or dashboards with these metrics to detect when rate limits or validation errors occur. ## Terminology - **sample**: a log line with [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) - **stream**: samples with a unique combination of labels - **active stream**: streams that are present in the ingesters - these have recently received log lines within the `chunk_idle_period` period (default: 30m) ## Rate-Limit Errors Rate-limits are enforced when Loki cannot handle more requests from a tenant. Rate-limit errors reject the whole request before Loki stores any of it. The tables below list `Sample discarded: No` for these errors because Loki never accepts the samples in the first place, but the request still increments `loki_discarded_samples_total` and `loki_discarded_bytes_total`. Because these errors are retryable, a client can resend the same data after the rate limit condition clears. ### `rate_limited` This rate limit is enforced when a tenant has exceeded their configured log ingestion rate limit. One solution if you're seeing samples dropped due to `rate_limited` is simply to increase the rate limits on your Loki cluster. These limits can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. The config options to use are `ingestion_rate_mb` and `ingestion_burst_size_mb`. Note that you'll want to make sure your Loki cluster has sufficient resources provisioned to be able to accommodate these higher limits. Otherwise your cluster may experience performance degradation as it tries to handle this higher volume of log lines to ingest. Another option to address samples being dropped due to `rate_limits` is simply to decrease the rate of log lines being sent to your Loki cluster. Consider collecting logs from fewer targets or setting up [drop stages](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagedrop-block) in Alloy to filter out certain log lines. You can also use Alloy's [rate limiting](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagelimit-block) to control the volume of logs sent to your Loki cluster. | Property | Value | |-------------------------|-------------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | Yes | | Sample discarded | No | | Configurable per tenant | Yes | | HTTP status code | `429 Too Many Requests` | ### `per_stream_rate_limit` This limit is enforced when a single stream reaches its rate limit. Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to (the size of that set is equal to the `replication_factor` value). This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. The config options to adjust are `per_stream_rate_limit` and `per_stream_rate_limit_burst`. The default values are `3MB` for `per_stream_rate_limit` and `15MB` for `per_stream_rate_limit_burst`. Another option you could consider to decrease the rate of samples dropped due to `per_stream_rate_limit` is to split the stream that is getting rate limited into several smaller streams. A third option is to use the Alloy [`stage.limit` block](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagelimit-block) to limit the rate of samples sent to the stream hitting the `per_stream_rate_limit`. We typically recommend setting `per_stream_rate_limit` no higher than 5MB, and `per_stream_rate_limit_burst` no higher than 20MB. | Property | Value | |-------------------------|-------------------------| | Enforced by | `ingester` | | Outcome | Request rejected | | Retryable | Yes | | Sample discarded | No | | Configurable per tenant | Yes | | HTTP status code | `429 Too Many Requests` | ### `stream_limit` This limit is enforced when a tenant reaches their maximum number of active streams. Active streams are held in memory buffers in the ingesters, and if this value becomes sufficiently large then it will cause the ingesters to run out of memory. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. To increase the allowable active streams, adjust `max_global_streams_per_user`. Alternatively, the number of active streams can be reduced by removing extraneous labels or removing excessive unique label values. {{< admonition type="note" >}} Under normal conditions, this limit only blocks the creation of genuinely new streams. Log lines for streams that are already active continue to be accepted. However, during ingester scaling (autoscaling, rollouts, or ring changes such as a partition-ingester migration), streams can be rebalanced to ingesters that were not previously handling them. The new ingester has no record of that stream in memory, so it treats the next write for that stream as a stream creation and checks it against its local share of the limit. If that share is already used up, for example because the same tenant is also adding new high-cardinality streams during the rebalance, writes to the already-active stream can be rejected with `stream_limit`, even though the tenant has not added new label cardinality for it. This effect is temporary. It is bounded by the number of ingesters added or restarted, and it typically resolves once the ring is stable and stream ownership is recalculated. {{< /admonition >}} | Property | Value | |-------------------------|-------------------------| | Enforced by | `ingester` | | Outcome | Request rejected | | Retryable | Yes | | Sample discarded | No | | Configurable per tenant | Yes | | HTTP status code | `429 Too Many Requests` | ## Validation Errors Validation errors occur when a request violates a validation rule defined by Loki. ### `line_too_long` This error occurs when a log line exceeds the maximum allowable length in bytes. The HTTP response will include the stream to which the offending log line belongs as well as its size in bytes. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. To increase the maximum line size, adjust `max_line_size`. We recommend that you do not increase this value above 256kb for performance reasons. Alternatively, Loki can be configured to ingest truncated versions of log lines over the length limit by using the `max_line_size_truncate` option. When truncation is enabled, you can use `max_line_size_truncate_identifier` to append a marker string to each truncated log line, so that truncated lines are easy to identify later. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `invalid_labels` This error occurs when one or more labels in the submitted streams fail validation. Loki uses the [same validation rules as Prometheus](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels) for validating labels. > Label names may contain ASCII letters, numbers, as well as underscores. They must match the regex `[a-zA-Z_][a-zA-Z0-9_]*`. Label names beginning with __ are reserved for internal use. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | No | | HTTP status code | `400 Bad Request` | ### `missing_labels` This validation error is returned when a stream is submitted without any labels. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | No | | HTTP status code | `400 Bad Request` | ### `too_far_behind` and `out_of_order` The `too_far_behind` validation error is returned when lines in question are older than half of `-ingester.max-chunk-age` compared to the newest line in the stream. {{< admonition type="note" >}} `unordered_writes` and its related flag `-ingester.unordered-writes` are deprecated and will be removed in a future major release. Once removed, out-of-order writes will be accepted unconditionally, and this setting will no longer be configurable. {{< /admonition >}} More details can be found [here](https://grafana.com/docs/loki//configuration/#accept-out-of-order-writes) about the Loki ordering constraints. The `unordered_writes` config value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. `max_chunk_age`, on the other hand, is a global-only configuration and cannot be set per tenant. This problem can be solved by ensuring that log delivery is configured correctly, or by increasing the `max_chunk_age` value. It is recommended to resist modifying the default value of `max_chunk_age` as this has other implications, and to instead try track down the cause for delayed logged delivery. It should also be noted that this a per-stream error, so by simply splitting streams (adding more labels) this problem can be circumvented, especially if multiple hosts are sending samples for a single stream. | Property | Value | |-------------------------|---------------------| | Enforced by | `ingester` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes, for `unordered_writes`. No, for `max_chunk_age`, which is global only. | | HTTP status code | `400 Bad Request` | ### `greater_than_max_sample_age` If the `reject_old_samples` config option is set to `true` (it is by default), then samples will be rejected with `reason=greater_than_max_sample_age` if they are older than the `reject_old_samples_max_age` value. You should not see samples rejected for `reason=greater_than_max_sample_age` if `reject_old_samples=false`. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. This error can be solved by increasing the `reject_old_samples_max_age` value, or investigating why log delivery is delayed for this particular stream. The stream in question will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `too_far_in_future` If a sample's timestamp is greater than the current timestamp, Loki allows for a certain grace period during which samples will be accepted. If the grace period is exceeded, the error will occur. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. This error can be solved by increasing the `creation_grace_period` value, or investigating why this particular stream has a timestamp too far into the future. The stream in question will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `max_label_names_per_series` If a sample is submitted with more labels than Loki has been configured to allow, it will be rejected with the `max_label_names_per_series` reason. Note that 'series' is the same thing as a 'stream' in Loki - the 'series' term is a legacy name. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. This error can be solved by increasing the `max_label_names_per_series` value. The stream to which the offending sample (i.e. the one with too many label names) belongs will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `label_name_too_long` If a sample is sent with a label name that has a length in bytes greater than Loki has been configured to allow, it will be rejected with the `label_name_too_long` reason. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. This error can be solved by increasing the `max_label_name_length` value, though we do not recommend raising it significantly above the default value of `1024` for performance reasons. The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `label_value_too_long` If a sample has a label value with a length in bytes greater than Loki has been configured to allow, it will be rejected for the `label_value_too_long` reason. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. This error can be solved by increasing the `max_label_value_length` value. The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `duplicate_label_names` If a sample is sent with two or more identical labels, it will be rejected for the `duplicate_label_names` reason. The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | No | | HTTP status code | `400 Bad Request` | ### `request_body_too_large` This error occurs when the raw push request exceeds a size limit before Loki can parse individual streams and log lines out of it. Unlike the other validation errors on this page, `request_body_too_large` can be triggered in two different ways: - The decompressed request body exceeds the `max_recv_msg_size` or `max_decompressed_size` settings in the `distributor` configuration block. These are cluster-wide settings, not `limits_config` values, so they cannot be set per tenant. `max_recv_msg_size` defaults to 100MB, and `max_decompressed_size` defaults to 5000MB (50 times the default `max_recv_msg_size`). - A single stream's label string exceeds 16MB. This limit is hardcoded and cannot be changed through any configuration option. It exists because the Prometheus label parser cannot handle labels above this size. To resolve this error, reduce the size of the push request (for example by sending fewer streams per request, or by reducing the number or size of labels on a stream), or increase `max_recv_msg_size` and `max_decompressed_size` if your cluster has the resources to handle larger requests. {{< admonition type="note" >}} This reason only appears in the `loki_discarded_bytes_total` metric, not in `loki_discarded_samples_total`. Loki rejects the request before it can parse the body, so it cannot count how many log lines the request contained. {{< /admonition >}} | Property | Value | |-------------------------|--------------------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | No | | HTTP status code | `413 Request Entity Too Large` | ### `disallowed_structured_metadata` This error occurs when a stream includes [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) but the tenant is not allowed to send it. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. To allow structured metadata, set `allow_structured_metadata` to `true` (this is the default). The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `structured_metadata_too_large` This error occurs when the [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) attached to a single log line exceeds the maximum allowable size in bytes. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. To increase the limit, adjust `max_structured_metadata_size` (default: `64kb`). The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `structured_metadata_too_many` This error occurs when a single log line includes more [structured metadata](https://grafana.com/docs/loki//get-started/labels/structured-metadata/) entries than Loki has been configured to allow. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. To increase the limit, adjust `max_structured_metadata_entries_count` (default: `128`). The offending stream will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | ### `blocked_ingestion` and `blocked_ingestion_policy` {{< admonition type="note" >}} This is an experimental feature. {{< /admonition >}} These reasons occur when an operator has deliberately blocked ingestion for a tenant (`blocked_ingestion`) or for a specific policy (`blocked_ingestion_policy`), rather than because a client sent invalid or excessive data. Policies are defined using the `policy_stream_mapping` setting in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block. This is configured using `block_ingestion_until` (for a tenant) or `block_ingestion_policy_until` (for a policy), both of which take a date in RFC3339 format. Ingestion is blocked until that date. These values can be set globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, but they are normally set for a single tenant in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file, because a global value blocks ingestion for every tenant. The `block_ingestion_status_code` option controls what HTTP status code Loki returns while ingestion is blocked. The default is a custom status code, `260`, so that blocked requests can be distinguished from ordinary errors. If `block_ingestion_status_code` is set to `200`, Loki silently drops the blocked data without returning an error to the client, although the data is still counted in `loki_discarded_samples_total` and `loki_discarded_bytes_total`. | Property | Value | |-------------------------|------------------------| | Enforced by | `distributor` | | Outcome | Request rejected, or silently dropped if `block_ingestion_status_code` is `200` | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `260` by default, configurable with `block_ingestion_status_code` | ### `missing_enforced_labels` {{< admonition type="note" >}} This is an experimental feature. {{< /admonition >}} This error occurs when a tenant has been configured to require certain labels on every stream, and a stream is submitted that is missing one or more of them. This value can be modified globally in the [`limits_config`](https://grafana.com/docs/loki//configuration/#limits_config) block, or on a per-tenant basis in the [runtime overrides](https://grafana.com/docs/loki//configuration/#runtime-configuration-file) file. Use `enforced_labels` to require labels on all streams, or `policy_enforced_labels` to require them only on streams matching a specific policy defined by `policy_stream_mapping`. The stream and the missing labels will be returned in the body of the HTTP response. | Property | Value | |-------------------------|-------------------| | Enforced by | `distributor` | | Outcome | Request rejected | | Retryable | **No** | | Sample discarded | **Yes** | | Configurable per tenant | Yes | | HTTP status code | `400 Bad Request` | # Manage larger production deployments When needing to scale Loki due to increased log volume, operators should consider running several Loki processes partitioned by role (ingester, distributor, querier, and so on) rather than a single Loki process. Grafana Labs' [production setup](https://github.com/grafana/loki/blob/main/production/ksonnet/loki) contains `.libsonnet` files that demonstrates configuring separate components and scaling for resource usage. ## Separate Query Scheduler The Query frontend has an in-memory queue that can be moved out into a separate process similar to the [Grafana Mimir query-scheduler](https://grafana.com/docs/mimir/latest/operators-guide/architecture/components/query-scheduler/). This allows running multiple query frontends. To run with the Query Scheduler, configure both the frontend and the querier worker with the scheduler's address. Using CLI flags: - Frontend: `-frontend.scheduler-address=` - Querier: `-querier.scheduler-address=` Using the configuration file: ```yaml frontend: scheduler_address: frontend_worker: scheduler_address: ``` {{< admonition type="note" >}} The querier's `scheduler_address` is configured under `frontend_worker`, not under `querier`. The [`frontend_worker`](https://grafana.com/docs/loki//configure/#frontend_worker) block configures the worker running within the querier that pulls and executes queries. {{< /admonition >}} It is not valid to start the querier with both a configured frontend address and a scheduler address. The query scheduler process itself can be started via the `-target=query-scheduler` option of the Loki Docker image. For instance, `docker run grafana/loki:latest -config.file=/etc/loki/config.yaml -target=query-scheduler -server.http-listen-port=8009 -server.grpc-listen-port=9009` starts the query scheduler listening on ports `8009` and `9009`. ### Scheduler discovery using a ring Instead of configuring a static `scheduler_address` on the frontend and the querier worker, you can have query schedulers register themselves in a [hash ring](https://grafana.com/docs/loki//get-started/hash-rings/#about-the-query-scheduler-ring). Queriers and query frontends then discover query schedulers through the ring instead of using a fixed address. To enable this, set `use_scheduler_ring: true`, or use the `-query-scheduler.use-scheduler-ring` CLI flag: ```yaml query_scheduler: use_scheduler_ring: true ``` {{< admonition type="note" >}} Set this option in the configuration used by your query schedulers, your queriers, and your query frontends. Queriers and query frontends only read the ring if `use_scheduler_ring` is true in their own configuration. If all of your components share one configuration file, you only need to set it once. {{< /admonition >}} The ring needs a key-value store. In most deployments you do not need to configure one for the query scheduler specifically, because the scheduler ring inherits the store from the [`common.ring`](https://grafana.com/docs/loki//configure/#common) block, and Loki uses `memberlist` for all rings when you configure a `memberlist` section. To set the store for the scheduler ring alone, use the `scheduler_ring` block: ```yaml query_scheduler: use_scheduler_ring: true scheduler_ring: kvstore: store: memberlist ``` If you do not configure a frontend address, a scheduler address, or a downstream URL anywhere in your configuration, Loki automatically enables the scheduler ring for you. Each component that takes part in the ring exposes the ring state at the `/scheduler/ring` endpoint, which you can use to check that all query schedulers registered as expected. ## Memory ballast In compute-constrained environments, garbage collection can become a significant performance factor. Frequently-run garbage collection interferes with running the application by using CPU resources. The use of memory ballast can mitigate the issue. Memory ballast allocates extra, but unused virtual memory in order to inflate the quantity of live heap space. Garbage collection is triggered by the growth of heap space usage. The inflated quantity of heap space reduces the perceived growth, so garbage collection occurs less frequently. Configure memory ballast using the ballast_bytes configuration option. ## Remote rule evaluation _This feature was first proposed in [`LID-0002`](https://github.com/grafana/loki/pull/8129); it contains the design decisions which informed the implementation._ By default, the `ruler` component embeds a query engine to evaluate rules. This generally works fine, except when rules are complex or have to process a large amount of data regularly. Poor performance of the `ruler` manifests as recording rules metrics with gaps or missed alerts. This situation can be detected by alerting on the `loki_prometheus_rule_group_iterations_missed_total` metric when it has a non-zero value. A solution to this problem is to externalize rule evaluation from the `ruler` process. The `ruler` embedded query engine is single-threaded, meaning that rules are not split, sharded, or otherwise accelerated like regular Loki queries. The `query-frontend` component exists explicitly for this purpose and, when combined with a number of `querier` instances, can massively improve rule evaluation performance and lead to fewer missed iterations. It is generally recommended to create a separate `query-frontend` deployment and `querier` pool from your existing one - which handles adhoc queries via Grafana, `logcli`, or the API. Rules should be given priority over adhoc queries because they are used to produce metrics or alerts which may be crucial to the reliable operation of your service; if you use the same `query-frontend` and `querier` pool for both, your rules will be executed with the same priority as adhoc queries which could lead to unpredictable performance. To enable remote rule evaluation, set the following configuration options: ```yaml ruler: evaluation: mode: remote query_frontend: address: dns:///: ``` If the `query-frontend` connection requires TLS, set `tls_enabled: true` under `query_frontend` and configure the accompanying TLS options. To reduce contention when many rules evaluate at the same time, set `max_jitter` to add a bounded, random delay before each rule evaluation. This option applies to both local and remote evaluation modes: ```yaml ruler: evaluation: max_jitter: 5s ``` Refer to the [`ruler` configuration reference](https://grafana.com/docs/loki//configure/#ruler) for further configuration options. When you enable remote rule evaluation, the `ruler` component becomes a gRPC client to the `query-frontend` service; this will result in far lower `ruler` resource usage because the majority of the work has been externalized. The LogQL queries coming from the `ruler` will be executed against the given `query-frontend` service. Requests will be load-balanced across all `query-frontend` IPs if the `dns:///` prefix is used. {{< admonition type="note" >}} Queries that fail to execute are _not_ retried. {{< /admonition >}} ### Limits and Observability Remote rule evaluation can be tuned with the following options: - `ruler_remote_evaluation_timeout`: maximum allowable execution time for rule evaluations - `ruler_remote_evaluation_max_response_size`: maximum allowable response size over gRPC connection from `query-frontend` to `ruler` Both of these can be specified globally in the [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config) section or on a [per-tenant basis](https://grafana.com/docs/loki//configure/#runtime-configuration-file). `max_jitter`, described earlier in this topic, is a global-only setting under `ruler.evaluation` rather than a per-tenant limit. Remote rule evaluation exposes a number of metrics: - `loki_ruler_remote_eval_request_duration_seconds`: time taken for rule evaluation (histogram) - `loki_ruler_remote_eval_response_bytes`: number of bytes in rule evaluation response (histogram) - `loki_ruler_remote_eval_response_samples`: number of samples in rule evaluation response (histogram) - `loki_ruler_remote_eval_success_total`: successful rule evaluations (counter) - `loki_ruler_remote_eval_failure_total`: unsuccessful rule evaluations with reasons (counter) Each of these metrics are per-tenant, so cardinality must be taken into consideration. # Manage version upgrades - [Upgrade](https://grafana.com/docs/loki//setup/upgrade/) from one Loki version to a newer version. - [Upgrade Helm](https://grafana.com/docs/loki//setup/upgrade/) from Helm v2.x to Helm v3.x. # Speed up ingester rollout using zone awareness The Loki zone aware ingesters are used by Grafana Labs in order to allow for easier rollouts of large Loki deployments. You can think of them as three logical zones, however with some extra Kubernetes configuration you could deploy them in separate zones. By default, an incoming log stream's logs are replicated to 3 random ingesters. Except in the case of some replica scaling up or down, a given stream will always be replicated to the same 3 ingesters. This means that if one of those ingesters is restarted no data is lost. However two or more ingesters restarting can result in data loss and also impacts the systems ability to ingest logs because of an unhealthy ring status. With zone awareness enabled, an incoming log line will be replicated to one ingester in each zone. This means that we're not only concerned about ingesters in multiple zones restarting at the same time, we can now rollout or lose an entire zone at once without impacting writes. This allows deployments with a large number of ingesters to be deployed much more quickly. At Grafana Labs, we also make use of [rollout-operator](https://github.com/grafana/rollout-operator) to manage rollouts to the 3 StatefulSets gracefully. The rollout-operator looks for labels on StatefulSets to know which ones are part of a certain rollout group, and coordinates rollouts of pods only from a single StatefulSet in the group at a time. See the README in the rollout-operator repo for a more in depth explanation. {{< admonition type="note" >}} This guide describes the [jsonnet](https://github.com/grafana/loki/blob/main/production/ksonnet/loki/multi-zone.libsonnet) deployment method used internally at Grafana Labs. New jsonnet deployments run zone aware ingesters by default, so this guide is only needed if you have an existing deployment that still uses a single ingester StatefulSet. If you deploy Loki with the [Helm chart](https://grafana.com/docs/loki//setup/install/helm/) instead, refer to [Zone-aware replication](https://grafana.com/docs/loki//setup/install/helm/concepts/#zone-aware-replication) in the Helm chart components documentation, which uses `ingester.zoneAwareReplication` Helm values rather than the jsonnet configuration described here. The number of zones is fixed at 3 in the jsonnet library described in this guide. There is no configuration option to run a different number of zones. {{< /admonition >}} ## Migration Migrating from a single ingester StatefulSet to 3 zone aware ingester StatefulSets. The migration follows a few general steps, regardless of deployment method. 1. Configure your existing ingesters to be part of a zone, for example `zone-default`, this will allow us to later exclude them from the write path while still allowing for graceful shutdowns. 1. Prep for the increase in active streams (due to the way streams are split between ingesters) by increasing the number of active streams allowed for your tenants. 1. Add and scale up your new zone-aware ingester StatefulSets such that each has 1/3rd of the total number of replicas you want to run. 1. Enable zone awareness on the write path by setting `distributor.zone-awareness-enabled` to `true` for distributors and rulers. 1. Wait some time to ensure that the new zone-aware ingesters have data for the time period they are queried for (`query_ingesters_within`). 1. Enable zone awareness on the read path by setting `distributor.zone-awareness-enabled` to true for queriers. {{< admonition type="note" >}} `distributor.zone-awareness-enabled` and `distributor.excluded-zones` are CLI flag names. Despite the `distributor.` prefix, they configure the ingester ring, so each process that uses that ring (distributor, ruler, and querier) reads the same flag. In YAML, the equivalent keys are `zone_awareness_enabled` and `excluded_zones` under `ingester.lifecycler.ring`, not under a top-level `distributor:` block. {{< /admonition >}} 1. Configure distributors and rulers to exclude ingesters in the `zone-default` so those ingesters no longer receive write traffic using `distributor.excluded-zones`. 1. Use the shutdown endpoint to flush data from the default ingesters, then scale down and remove the associated StatefulSet. 1. Clean up any config remaining from the migration. ### Detailed Migration Steps The following are steps to live migrate (no downtime) an existing Loki deployment from a single ingester StatefulSet to 3 zone aware ingester StatefulSets. These instructions assume you are using the zone aware ingester jsonnet deployment code from this repo, see [here](https://github.com/grafana/loki/blob/main/production/ksonnet/loki/multi-zone.libsonnet). **If you are not using jsonnet see the relevant annotations in some steps that describe how to perform that step manually.** 1. Configure the zone for the existing ingester StatefulSet as `zone-default` by setting `multi_zone_default_ingester_zone: true`, this allows us to later filter out that zone from the write path. If you're not using jsonnet, this is done by setting the `-ingester.availability-zone=zone-default` flag (or the equivalent `availability_zone` field under the ingester's `lifecycler` YAML configuration) on the existing ingesters. 1. Configure ingester-pdb with `maxUnavailable` as 0 and deploy 3x zone-aware StatefulSets with 0 replicas by setting ```jsonnet _config+:: { multi_zone_ingester_enabled: true, multi_zone_ingester_migration_enabled: true, multi_zone_ingester_replicas: 0, // These last two lines are necessary now that we enable zone aware ingester by default // so that newly created cells will not be migrated later on. If you miss them you will // break writes in the cell. multi_zone_ingester_replication_write_path_enabled: false, multi_zone_ingester_replication_read_path_enabled: false, }, ``` Setting `multi_zone_ingester_migration_enabled: true` already configures the existing `ingester` PodDisruptionBudget (PDB) with `maxUnavailable: 0` for you, so no manual PDB change is needed when using jsonnet. If you're not using jsonnet, manually set `maxUnavailable` to `0` on the existing ingester-pdb. The new ingester StatefulSets should have a label with `rollout-group: ingester`, annotation `rollout-max-unavailable: x` (put a placeholder value in, later you should set the value of this to be some portion of the StatefulSets total replicas, for example in jsonnet we template this so that each StatefulSet runs 1/3 of the total replicas and the max unavailable is 1/3 of each StatefulSets replicas), set the update strategy to `OnDelete`, and set `-ingester.availability-zone=zone-` for each of the three new StatefulSets. 1. Diff ingester and ingester-zone-a StatefulSets and make sure all config matches ```bash kubectl get statefulset -n loki-dev-008 ingester -o yaml > ingester.yaml kubectl get statefulset -n loki-dev-008 ingester-zone-a -o yaml > ingester-zone-a.yaml diff ingester.yaml ingester-zone-a.yaml ``` Expected in diffs are values like: creation time and revision number, the zone, fields used by rollout operator, number of replicas, anything related to kustomize/flux, and PVC for the WAL since the containers don't exist yet. 1. Temporarily double max series limits for users that are using more than 50% of their current limit, the queries are as follows (add label selectors as appropriate): ```logql sum by (tenant)(sum (loki_ingester_memory_streams) by (cluster, namespace, tenant) / on (namespace) group_left max by(namespace) (loki_distributor_replication_factor)) > on (tenant) ( max by (tenant) (label_replace(loki_overrides{limit_name="max_global_streams_per_user"} / 2.5, "tenant", "$1", "user", "(.+)")) ) ``` ```logql (sum (loki_ingester_memory_streams) by (cluster, namespace, tenant) / on (namespace) group_left max by(namespace) (loki_distributor_replication_factor) ) / ignoring(tenant) group_left max by (cluster, namespace)(loki_overrides_defaults{limit_name="max_global_streams_per_user"}) > 0.4) unless on (tenant) ( (label_replace(loki_overrides{limit_name="max_global_streams_per_user"},"tenant", "$1", "user", "(.+)"))) ``` 1. Scale up zone-aware StatefulSets until they have 1/3rd of replicas each. In smaller cells you can do this all at once, in larger cells it is safer to do it in chunks. The config value you need to change is `multi_zone_ingester_replicas: 6`, the value will be split across the three StatefulSets. In this case, each StatefulSet would run 2 replicas. If you're not using jsonnet, this is the step where you would also set the annotation `rollout-max-unavailable` to some value that is less than or equal to the number of replicas each StatefulSet is running. By default, each zone's StatefulSet uses pod anti-affinity so that ingesters from two different zones are never scheduled on the same node. Two ingesters from the same zone are still allowed to share a node. This anti-affinity rule is controlled by `ingester_allow_multiple_replicas_on_same_node`, which defaults to `false`. To also spread the ingesters within a single zone evenly across nodes, set `multi_zone_ingester_use_topology_spread: true`. This adds a [topology spread constraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) to each zone's StatefulSet, which you can tune with `multi_zone_ingester_topology_spread_max_skew` and `multi_zone_ingester_topology_spread_when_unsatisfiable`. 1. Enable zone awareness on the write path by setting `multi_zone_ingester_replication_write_path_enabled: true`, this causes distributors and rulers to reshuffle series to distributors in each zone. Be sure to check that all the distributors and rulers have restarted properly. If you're not using jsonnet, enable zone awareness on the write path by setting `distributor.zone-awareness-enabled` to true for distributors and rulers. 1. Wait for `query_ingesters_within` configured hours. The default is `3h`. This ensures that no data will be missing if we query a new ingester. However, because we cut chunks at least every 30m due to `chunk_idle_period` we can likely reduce this amount of time. 1. Check that rule evaluations are still correct on the migration, look for increases in the rate for metrics with names with the following suffixes: ``` rule_evaluations_total rule_evaluation_failures_total rule_group_iterations_missed_total ``` 1. Enable zone-aware replication on the read path `multi_zone_ingester_replication_read_path_enabled: true`. If you're not using jsonnet, set `distributor.zone-awareness-enabled` to true for queriers. 1. Check that queries are still executing correctly, for example look at `loki_logql_querystats_latency_seconds_count` to see that you don't have a big increase in latency or error count for a specific query type. 1. Configure distributor / ruler to exclude ingesters in the `zone-default` so those ingesters no longer receive write traffic by setting `multi_zone_ingester_exclude_default: true`. If you're not using jsonnet set `distributor.excluded-zones` on distributors and rulers. It is a good idea to check rules evaluations again at this point, and also that the zone aware ingester StatefulSet is now receiving all the write traffic, you can compare `sum(loki_ingester_memory_streams{cluster="",job=~"()/ingester"})` to `sum(loki_ingester_memory_streams{cluster="",job=~"()/ingester-zone.*"})` 1. If you're using an automated reconciliation or deployment system like flux, disable it now (for example using flux ignore) if possible for just the default ingester StatefulSet. 1. Shutdown flush the default ingesters, unregistering them from the ring, you can do this by port-forwarding each ingester Pod and using the endpoint: `"http://url:PORT/ingester/shutdown?flush=true&delete_ring_tokens=true&terminate=false"` See [Flush in-memory chunks and shut down](https://grafana.com/docs/loki//reference/loki-http-api/#flush-in-memory-chunks-and-shut-down) for details on this endpoint. For Kubernetes-native scale-down automation, you can instead use [`POST /ingester/prepare_shutdown`](https://grafana.com/docs/loki//reference/loki-http-api/#prepare-ingester-shutdown), which arms the ingester to flush and unregister on its next `SIGTERM` rather than shutting it down immediately. 1. manually scale down the default ingester StatefulSet to 0 replicas, we do this via `tk apply` but you could do it via modifying the yaml. 1. merge a PR to your central config repo to keep the StatefulSet 0'd, and then remove the flux ignore. 1. clean up any remaining temporary config from the migration, for example `multi_zone_ingester_migration_enabled: true` is no longer needed. 1. ensure that all the old default ingester PVC/PV are removed. # Deploy Loki meta-monitoring with chart version 3 The primary method for collecting and monitoring a Loki cluster is to use the [Kubernetes Monitoring Helm](https://github.com/grafana/k8s-monitoring-helm/) chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying version 3.8.x of the Kubernetes Monitoring Helm chart to monitor your Loki cluster. {{< admonition type="note" >}} This page covers chart version 3.8.x, which is a maintained stable release line. If you do not already have a reason to stay on chart 3.8.x, use [Deploy Loki meta-monitoring](https://grafana.com/docs/loki//operations/meta-monitoring/deploy/) instead, which covers the current chart 4.x line. For help moving from chart 3.x to 4.x, refer to the [Kubernetes Monitoring Helm chart migration guide](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/helm-chart/migrate-helm-chart/). {{< /admonition >}} {{< admonition type="note" >}} We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure. {{< /admonition >}} ## Prerequisites - [kubectl](https://kubernetes.io/docs/reference/kubectl/) - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A running Kubernetes cluster with a running Loki deployment. - A Grafana Cloud account or a separate LGTM stack for monitoring. ## Preparing your environment Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment. 1. Add the Grafana Helm repository: ```bash helm repo add grafana https://grafana.github.io/helm-charts ``` 1. Update your Helm repositories: ```bash helm repo update ``` 1. Create a namespace for the monitoring stack: ```bash kubectl create namespace meta ``` ### Authentication The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example. 1. Create a new Cloud Access Policy in Grafana Cloud. 1. Sign into [Grafana Cloud](https://grafana.com/auth/sign-in/). 1. In the main menu, select **Security > Access Policies**. 1. Click **Create access policy**. 1. Give the policy a **Name** and select the following permissions: - Metrics: Write - Logs: Write 1. Click **Create**. 1. Click **Add Token**. Give the token a name and click **Create**. Save the token for later use. 1. Collect `url` and `username` for Prometheus and Loki. 1. Navigate to the Grafana Cloud Portal **Overview** page. 1. Click the **Details** button for your Prometheus instance. 1. From the **Sending metrics using Grafana Alloy** section, collect the instance **username** and **url**. 1. Navigate back to the **Overview** page. 1. Click the **Details** button for your Loki instance. 1. From the **Sending Logs to Grafana Cloud using Grafana Alloy** section, collect the instance **username** and **url**. 2. Navigate back to the **Overview** page. 1. Create the Kubernetes Secrets with the collected credentials from Grafana Cloud. ```bash kubectl create secret generic metrics -n meta \ --from-literal=username= \ --from-literal=password= kubectl create secret generic logs -n meta \ --from-literal=username= \ --from-literal=password= ``` Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including: - Bearer Tokens - OAuth2 - SigV4 - External Secrets For further information on how to configure these methods, see the [Kubernetes Monitoring Helm examples](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/auth). ## Deploy the Kubernetes Monitoring Helm chart Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a `values.yaml` file to our local machine and modify it to include the necessary configuration. 1. Download the `values-chart-v3.yaml` file from the Kubernetes Monitoring Helm chart repository: ```bash curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values-chart-v3.yaml ``` 1. Open the `values-chart-v3.yaml` file in a text editor of your choosing and add the Prometheus and Loki endpoints. ```yaml destinations: - name: prometheus type: prometheus url: https:///api/prom/push auth: type: basic usernameKey: username passwordKey: password secret: create: false name: metrics namespace: meta - name: loki type: loki url: https:///loki/api/v1/push auth: type: basic usernameKey: username passwordKey: password secret: create: false name: logs namespace: meta ``` 1. (Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud. ```yaml # Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster. cluster: name: loki-meta-monitoring-cluster ``` 1. The default values file assumes that you have deployed Loki in the `loki` namespace and will deploy the Kubernetes monitoring stack in the `meta` namespace. If you have deployed Loki in a different namespace, or the monitoring stack in a namespace other than `meta`, you will need to update several keys in `values-chart-v3.yaml`. There is no single `namespaces` key; namespace scoping is set independently for each feature: | Key | Format | Purpose | | --- | --- | --- | | `integrations.loki.instances[0].namespaces` | list | Namespace(s) to discover Loki instances in. | | `integrations.alloy.instances[0].namespaces` | list | Namespace(s) to discover the monitoring stack's own Alloy collectors in. | | `clusterEvents.namespaces` | list | Namespace(s) to capture Kubernetes events from. | | `clusterMetrics.cadvisor.metricsTuning.includeNamespaces` | list | Namespace(s) to keep cadvisor metrics for. This is a metric filter, not a discovery scope. | | `clusterMetrics.kube-state-metrics.namespaces` | list, or comma-separated string | Namespace(s) to collect Kubernetes resource state from. | | `podLogs.namespaces` | list | Namespace(s) to collect pod logs from. | For example, to add a namespace to the Loki discovery scope: ```yaml integrations: loki: instances: - name: loki namespaces: - loki ``` Each of these keys defaults to an empty list, which means "collect from all namespaces". Removing a key restores that default for that feature only; it does not affect the other keys in this table. Also note that `destinations[].secret.namespace` (used for both the `metrics` and `logs` destinations) must match the namespace where you created the `metrics` and `logs` secrets, which is `meta` by default. Update it if you installed the monitoring stack into a different namespace. 1. Deploy the Kubernetes Monitoring Helm chart using the modified `values-chart-v3.yaml` file, pinning the chart to the 3.x line: ```bash helm install meta-loki grafana/k8s-monitoring \ --namespace meta \ --version "^3" \ -f values-chart-v3.yaml ``` 1. Verify that the Kubernetes Monitoring Helm chart has been deployed successfully: ```bash kubectl get pods -n meta ``` You should see a list of pods running in the `meta` namespace. ```console NAME READY STATUS RESTARTS ... meta-loki-alloy-singleton-6d7f8d8b86-sg4wx 2/2 Running 0 ... meta-loki-kube-state-metrics-64bdcfcbd-5snqz 1/1 Running 0 ... meta-loki-node-exporter-855l5 1/1 Running 0 ... meta-loki-node-exporter-b976b 1/1 Running 0 ... meta-loki-node-exporter-vsm4s 1/1 Running 0 ... ``` ## Next Steps You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see [Install the Loki Mixin](https://grafana.com/docs/loki//operations/meta-monitoring/mixins). # Deploy Loki meta-monitoring The primary method for collecting and monitoring a Loki cluster is to use the [Kubernetes Monitoring Helm](https://github.com/grafana/k8s-monitoring-helm/) chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying the Kubernetes Monitoring Helm chart to monitor your Loki cluster. {{< admonition type="note" >}} We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure. If you run Loki in monolithic mode outside of this Helm chart, refer to [Single binary meta-monitoring](https://grafana.com/docs/loki//operations/meta-monitoring/single-binary/) instead. {{< /admonition >}} {{< admonition type="note" >}} This page covers chart version 4.x, the current line. If you need to stay on the maintained 3.8.x stable line instead, use [Deploy Loki meta-monitoring with chart version 3](https://grafana.com/docs/loki//operations/meta-monitoring/deploy-chart-v3/). {{< /admonition >}} ## Prerequisites - [kubectl](https://kubernetes.io/docs/reference/kubectl/) - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A running Kubernetes cluster with a running Loki deployment. - A Grafana Cloud account or a separate LGTM stack for monitoring. ## Preparing your environment Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment. 1. Add the Grafana Helm repository: ```bash helm repo add grafana https://grafana.github.io/helm-charts ``` 1. Update your Helm repositories: ```bash helm repo update ``` 1. Create a namespace for the monitoring stack: ```bash kubectl create namespace meta ``` ### Authentication The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example. 1. Create a new Cloud Access Policy in Grafana Cloud. 1. Sign into [Grafana Cloud](https://grafana.com/auth/sign-in/). 1. In the main menu, select **Security > Access Policies**. 1. Click **Create access policy**. 1. Give the policy a **Name** and select the following permissions: - Metrics: Write - Logs: Write 1. Click **Create**. 1. Click **Add Token**. Give the token a name and click **Create**. Save the token for later use. 1. Collect `url` and `username` for Prometheus and Loki. 1. Navigate to the Grafana Cloud Portal **Overview** page. 1. Click the **Details** button for your Prometheus instance. 1. From the **Sending metrics using Grafana Alloy** section, collect the instance **username** and **url**. 1. Navigate back to the **Overview** page. 1. Click the **Details** button for your Loki instance. 1. From the **Sending Logs to Grafana Cloud using Grafana Alloy** section, collect the instance **username** and **url**. 2. Navigate back to the **Overview** page. 1. Create the Kubernetes Secrets with the collected credentials from Grafana Cloud. ```bash kubectl create secret generic metrics -n meta \ --from-literal=username= \ --from-literal=password= kubectl create secret generic logs -n meta \ --from-literal=username= \ --from-literal=password= ``` Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including: - Bearer Tokens - OAuth2 - SigV4 - External Secrets For further information on how to configure these methods, see the [Kubernetes Monitoring Helm examples](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/auth). ## Deploy the Kubernetes Monitoring Helm chart Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a `values.yaml` file to our local machine and modify it to include the necessary configuration. 1. Download the `values.yaml` file from the Kubernetes Monitoring Helm chart repository: ```bash curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values.yaml ``` 1. Open the `values.yaml` file in a text editor of your choosing and add the Prometheus and Loki endpoints. Chart 4.x configures destinations as a map keyed by name, rather than a list: ```yaml destinations: prometheus: type: prometheus url: https:///api/prom/push auth: type: basic usernameKey: username passwordKey: password secret: create: false name: metrics namespace: meta loki: type: loki url: https:///loki/api/v1/push auth: type: basic usernameKey: username passwordKey: password secret: create: false name: logs namespace: meta ``` 1. (Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud. ```yaml # Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster. cluster: name: loki-meta-monitoring-cluster ``` 1. Confirm that at least one collector is defined under `collectors:`. Chart 4.x requires this; the install fails if `collectors:` is empty or missing. The default `values.yaml` already defines the `alloy-singleton` collector that every feature in this file is assigned to: ```yaml collectors: alloy-singleton: presets: [singleton] ``` If you add or rename a collector, update every `collector: alloy-singleton` reference in the file to match, or the install fails with an error naming the missing collector. 1. The default values file assumes that you have deployed Loki in the `loki` namespace and will deploy the Kubernetes monitoring stack in the `meta` namespace. If you have deployed Loki in a different namespace, or the monitoring stack in a namespace other than `meta`, you will need to update several keys in `values.yaml`. There is no single `namespaces` key; namespace scoping is set independently for each feature: | Key | Format | Purpose | | --- | --- | --- | | `integrations.loki.instances[0].namespaces` | list | Namespace(s) to discover Loki instances in. | | `integrations.alloy.instances[0].namespaces` | list | Namespace(s) to discover the monitoring stack's own Alloy collectors in. | | `clusterEvents.namespaces` | list | Namespace(s) to capture Kubernetes events from. | | `clusterMetrics.cadvisor.metricsTuning.includeNamespaces` | list | Namespace(s) to keep cadvisor metrics for. This is a metric filter, not a discovery scope. | | `telemetryServices.kube-state-metrics.namespaces` | list, or comma-separated string | Namespace(s) to collect Kubernetes resource state from. In chart 4.x this key must be set under `telemetryServices`, not under `clusterMetrics`. | | `podLogsViaKubernetesApi.namespaces` | list | Namespace(s) to collect pod logs from. | For example, to add a namespace to the Loki discovery scope: ```yaml integrations: loki: instances: - name: loki namespaces: - loki ``` Each of these keys defaults to an empty list, which means "collect from all namespaces". Removing a key restores that default for that feature only; it does not affect the other keys in this table. Also note that `destinations.prometheus.secret.namespace` and `destinations.loki.secret.namespace` must match the namespace where you created the `metrics` and `logs` secrets, which is `meta` by default. Update them if you installed the monitoring stack into a different namespace. 1. Deploy the Kubernetes Monitoring Helm chart using the modified `values.yaml` file, pinning the chart to the 4.x line: ```bash helm install meta-loki grafana/k8s-monitoring \ --namespace meta \ --version "^4" \ -f values.yaml ``` 1. Verify that the Kubernetes Monitoring Helm chart has been deployed successfully: ```bash kubectl get pods -n meta ``` You should see a list of pods running in the `meta` namespace, similar to the following. Exact pod names and counts depend on your cluster size and which features you have enabled: ```console NAME READY STATUS RESTARTS ... meta-loki-alloy-operator-7d8f9c6b5d-x2k9p 1/1 Running 0 ... meta-loki-alloy-singleton-6d7f8d8b86-sg4wx 2/2 Running 0 ... meta-loki-kube-state-metrics-64bdcfcbd-5snqz 1/1 Running 0 ... meta-loki-node-exporter-855l5 1/1 Running 0 ... meta-loki-node-exporter-b976b 1/1 Running 0 ... meta-loki-node-exporter-vsm4s 1/1 Running 0 ... ``` ## Next Steps You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see [Install the Loki Mixin](https://grafana.com/docs/loki//operations/meta-monitoring/mixins). # Key metrics for monitoring Loki Loki exposes many metrics, and each component behaves differently under load. This page focuses on the highest-signal metrics for detecting negative trends early. {{< admonition type="note" >}} The example queries on this page are PromQL. Run them against the Prometheus-compatible data source where your Loki metrics are stored (for example, Prometheus, Mimir, or Grafana Cloud Metrics). {{< /admonition >}} For setup and prebuilt dashboards and alerts, refer to: - [Deploy Loki meta-monitoring](https://grafana.com/docs/loki//operations/meta-monitoring/deploy/) - [Install dashboards, alerts, and recording rules](https://grafana.com/docs/loki//operations/meta-monitoring/mixins/) ## Request error rate Watch request failures first. A sustained increase in 5xx responses is usually the earliest sign of user-visible impact. Key metric: - `loki_request_duration_seconds`, a histogram with the labels `method`, `route`, `status_code`, and `ws`. Use the `_count` and `_bucket` suffixes to derive request rate and latency percentiles. {{< admonition type="note" >}} `cluster`, `namespace`, and `job` in the example queries on this page come from your Prometheus scrape configuration, not from Loki itself. They are not labels on the metric. {{< /admonition >}} Example query: ```promql 100 * sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[2m])) by (cluster, namespace, job, route) / sum(rate(loki_request_duration_seconds_count[2m])) by (cluster, namespace, job, route) ``` Abnormal behavior: - Any sustained increase in 5xx ratio. - The Loki mixin alert `LokiRequestErrors` fires when this ratio is greater than 10% for 15 minutes. ## Request latency (p99) Latency degradation can appear before hard failures. Track p99 for read and write routes. Key metric: - `loki_request_duration_seconds_bucket` (histogram buckets of the same metric described in [Request error rate](#request-error-rate)) Example query: ```promql histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, cluster, namespace, job, route)) ``` Abnormal behavior: - Rising p99 over time, especially in query-frontend and distributor paths. - The Loki mixin alert `LokiRequestLatency` fires when p99 exceeds 1 second for 15 minutes. This alert excludes tail and scheduler routes, so the example query above can show a higher p99 than the alert reacts to. ## Panics Panics are high-severity faults and should stay at zero. Key metric: - `loki_panic_total` Example query: ```promql sum(increase(loki_panic_total[10m])) by (cluster, namespace, job) ``` Abnormal behavior: - Any value above zero. The Loki mixin alert `LokiRequestPanics` treats this as critical. ## Discarded samples Discarded samples indicate data that Loki rejected or dropped. This is one of the most important ingestion-quality signals. Key metric: - `loki_discarded_samples_total` (counter with labels including `tenant` and `reason`, plus `retention_hours`, `policy`, and `format`) Example query: ```promql topk(10, sum by (tenant, reason) (rate(loki_discarded_samples_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))) ``` Abnormal behavior: - Increasing discard rate. - New or growing `reason` values (for example, tenant limits or stream limits). ## Compaction health Compaction issues can silently degrade read performance and retention behavior over time. {{< admonition type="note" >}} Prior to Loki 4.0 the compaction and retention metrics used a `loki_boltdb_shipper_` prefix for historical reasons. The compactor emitted these metrics regardless of which index type you use, including TSDB. After removing the legacy BoltDB Shipper index, the prefix has been renamed to `loki_tsdb_shipper_`. {{< /admonition >}} Key metrics: - `loki_tsdb_shipper_compactor_running` - `loki_tsdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds` - `loki_tsdb_shipper_compact_tables_operation_total` - `loki_tsdb_shipper_compact_tables_operation_duration_seconds` Example queries: ```promql sum(loki_tsdb_shipper_compactor_running) by (cluster, namespace) ``` ```promql time() - (loki_tsdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds > 0) ``` Abnormal behavior: - More than one compactor running. Only one compactor instance should run at a time. Running more than one can lead to data loss. The Loki mixin alert `LokiTooManyCompactorsRunning` fires when more than one compactor runs for 5 minutes. - No successful compaction for multiple hours. The mixin alert `LokiCompactorHasNotSuccessfullyRunCompaction` triggers when the last successful run is more than 3 hours old, and fires after that condition holds for 1 additional hour, so it takes about 4 hours in total to alert. ## Ingester health and flush behavior Ingester pressure often appears as memory growth, poor chunk utilization, or flush backlog. Key metrics: - `loki_ingester_memory_streams` - `loki_ingester_memory_stream_shards` - `loki_ingester_memory_chunks` - `loki_ingester_flush_queue_length` - `loki_ingester_chunk_utilization` - `loki_ingester_chunks_flushed_total` Example queries: ```promql sum(loki_ingester_memory_streams{cluster="$cluster", namespace="$namespace"}) ``` The share of in-memory streams that were created by stream sharding in the distributors: ```promql sum(loki_ingester_memory_stream_shards{cluster="$cluster", namespace="$namespace"}) / sum(loki_ingester_memory_streams{cluster="$cluster", namespace="$namespace"}) ``` ```promql sum(loki_ingester_flush_queue_length{cluster="$cluster", namespace="$namespace"}) ``` Abnormal behavior: - Persistent growth in in-memory streams or chunks. - Increasing flush queue length. - Low chunk utilization for long periods. ## Distributor throughput Throughput changes help identify upstream sender issues, sudden traffic shifts, or ingestion bottlenecks. Key metrics: - `loki_distributor_bytes_received_total` - `loki_distributor_lines_received_total` Example queries: ```promql sum(rate(loki_distributor_bytes_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) ``` ```promql sum(rate(loki_distributor_lines_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) ``` Abnormal behavior: - Sharp drops (possible data path interruption). - Unexpected spikes (possible overload or noisy tenants). ## Kafka-based ingestion (experimental) {{< admonition type="note" >}} These metrics only apply if you've enabled [Kafka-based ingestion](https://grafana.com/docs/loki//operations/kafka/), an experimental write-path feature. {{< /admonition >}} **Key metrics:** - `loki_kafka_client_partition_reader_consumption_lag_seconds`: the single most important Kafka health signal. Growing lag means ingesters can't keep up with the rate at which the distributor produces records. - `loki_distributor_kafka_appends_total` (label `status`): produce success and failure rate. A rising failure rate means writes aren't reaching Kafka. - `loki_distributor_kafka_latency_seconds`: produce latency. A sustained increase points to Kafka broker pressure or network issues. - `loki_kafka_client_partition_reader_fetch_errors_total`: errors fetching records from Kafka brokers. This should stay near zero. **Example queries:** ```promql # Consumption lag p99 by partition state histogram_quantile(0.99, sum(rate(loki_kafka_client_partition_reader_consumption_lag_seconds_bucket{phase="running", cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (le, cluster, namespace)) ``` ```promql # Kafka produce error rate sum(rate(loki_distributor_kafka_appends_total{status="fail", cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (cluster, namespace) ``` **Abnormal behavior:** - Consumption lag growing over time (ingesters can't keep up with the produce rate). - Any sustained produce failures. - Rising produce latency (broker pressure, network issues, or too few partitions). - Fetch errors (broker unavailability or a protocol version mismatch). For a complete list of Kafka metrics, refer to [Kafka-based ingestion](https://grafana.com/docs/loki//operations/kafka/#monitoring-kafka-ingestion). ## Object store operations Object store latency and failures directly impact query and retention workflows. Key metrics: - `loki_objstore_bucket_operations_total` - `loki_objstore_bucket_operation_failures_total` - `loki_objstore_bucket_operation_duration_seconds` Example queries: ```promql sum by (operation) (rate(loki_objstore_bucket_operation_failures_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) ``` ```promql histogram_quantile(0.99, sum(rate(loki_objstore_bucket_operation_duration_seconds_bucket{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (le, operation)) ``` Abnormal behavior: - Growing failure rates by operation. - Increasing p99 latency for `get`, `get_range`, or `upload`. Connection metrics just for S3: - `net_conntrack_dialer_conn_attempted_total` - `net_conntrack_dialer_conn_established_total` - `net_conntrack_dialer_conn_failed_total` - `net_conntrack_dialer_conn_closed_total` Abnormal behavior: - Non-zero rate of failed connections. - High rate of closed connections under ordinary load. ## Resource and runtime health Resource pressure can explain or predict service degradation before alert thresholds are crossed. Common signals to track: - Container CPU usage - Container memory working set - Go heap in use - Disk read and write rates - Container restarts Abnormal behavior: - Repeated restart spikes. - Sustained CPU saturation. - Memory growth without recovery. ## Loki Canary (end-to-end data verification) If you run Loki Canary, use it as an end-to-end correctness signal, not only a performance signal. Key metrics: - `loki_canary_missing_entries_total` - `loki_canary_spot_check_missing_entries_total` - `loki_canary_response_latency_seconds_bucket` Example query: ```promql sum(increase(loki_canary_missing_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range])) / sum(increase(loki_canary_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range])) * 100 ``` Abnormal behavior: - Any non-zero missing rate sustained over time. ## Internal error log rate Internal logs provide fast context when metrics indicate degradation. Key metric: - `loki_internal_log_messages_total` Use this metric with component logs to correlate where failures begin. ## Retention and sweeper progress Retention and sweeper lag can cause storage growth and delayed data lifecycle actions. Key metrics: - `loki_compactor_apply_retention_last_successful_run_timestamp_seconds` - `loki_tsdb_shipper_retention_sweeper_marker_file_processing_current_time` - `loki_tsdb_shipper_retention_sweeper_chunk_deleted_duration_seconds_count` Example query: ```promql time() - (loki_tsdb_shipper_retention_sweeper_marker_file_processing_current_time{cluster="$cluster", namespace="$namespace"} > 0) ``` Abnormal behavior: - Increasing sweeper lag. - Falling delete throughput or sustained delete failures. ## Next steps - Install and keep the latest Loki mixin dashboards and alerts. - Build component-specific alerts on top of these baseline signals. - Pair metrics with Loki component logs, including `metrics.go`, during incident response. # Install dashboards, alerts, and recording rules Loki is instrumented to expose metrics about itself via the `/metrics` endpoint, designed to be scraped by Prometheus. Each Loki release includes a mixin. The Loki mixin provides a set of Grafana dashboards, Prometheus recording rules and alerts for monitoring Loki. To set up monitoring using the mixin, you need to: 1. Deploy the Kubernetes Monitoring Helm chart. Follow the instructions in the [Deploy Loki Meta-monitoring](https://grafana.com/docs/loki//operations/meta-monitoring/deploy) documentation. 1. Be actively storing metrics from your Loki cluster in Grafana Cloud or a separate LGTM stack. This procedure assumes that you have set up Loki using the Helm chart. {{< admonition type="note" >}} Be sure to update the commands and configuration to match your own deployment. {{< /admonition >}} ## Install Loki dashboards in Grafana After Loki metrics are scraped by the Kubernetes Monitoring Helm chart and stored in a Prometheus compatible time-series database, you can monitor Loki’s operation using the Loki mixin. Each Loki release includes a mixin that includes: - Relevant dashboards for overseeing the health of Loki as a whole, as well as its individual Loki components - [Recording rules](https://grafana.com/docs/loki//alert/#recording-rules) that compute metrics that are used in the dashboards - Alerts that trigger when Loki generates metrics that are outside of normal parameters To install the mixins in Grafana and a Prometheus-compatible rules store, the general steps are as follows: 1. Download the mixin dashboards from the Loki repository. 1. Import the dashboards in your Grafana instance. 1. Upload `alerts.yaml` and `rules.yaml` files to Prometheus or Mimir with `mimirtool`. ### Download the `loki-mixin` dashboards 1. First, clone the Loki repository from Github: ```bash git clone https://github.com/grafana/loki cd loki ``` 1. Check out the tag that matches the version of Loki you are running, so that the dashboards, alerts, and recording rules match the metrics your Loki cluster exposes. For example, if you are running Loki 3.7.6: ```bash git checkout v3.7.6 ``` 1. Once you have a local copy of the repository at the matching version, navigate to the `production/loki-mixin-compiled` directory. ```bash cd production/loki-mixin-compiled ``` This directory contains a compiled version of the alert and recording rules, as well as the dashboards. {{< admonition type="note" >}} If you want to change any of the mixins, make your updates in the `production/loki-mixin` directory. Use the instructions in the [README](https://github.com/grafana/loki/tree/main/production/loki-mixin) in that directory to regenerate the files. {{< /admonition >}} ### Import the dashboards to Grafana The `dashboards` directory includes the monitoring dashboards that can be installed into your Grafana instance. Refer to [Import a dashboard](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) in the Grafana documentation. {{< admonition type="note" >}} The dashboards and alerts expect your metrics to carry these labels: `cluster`, `namespace`, `job`, `container`, `pod`, and `instance`. The Kubernetes Monitoring Helm chart provides them, because it adds the cluster name as a global label and collects container and node metrics using cAdvisor, kube-state-metrics, and Node Exporter. If your setup uses different label names, or does not collect container and node metrics, some panels and alerts show no data until you relabel your metrics to match. {{< /admonition >}} {{< admonition type="tip" >}} Install all dashboards. You can only import one dashboard at a time. Create a new folder in the Dashboards area, for example “Loki Monitoring”, as an easy location to save the imported dashboards. {{< /admonition >}} To create a folder: 1. Open your Grafana instance and select **Dashboards**. 1. Click the **New** button. 1. Select **New folder** from the **New** menu. 1. Name your folder, for example, “Loki Monitoring”. 1. Click **Create**. To import a dashboard: 1. Open your Grafana instance and select **Dashboards**. 1. Click the **New** button. 1. Select **Import** from the **New** menu. 1. On the **Import dashboard** screen, select **Upload dashboard JSON file.** 1. Browse to `production/loki-mixin-compiled/dashboards` and select the dashboard to import. Or, drag the dashboard file, for example, `loki-operational.json`, onto the **Upload** area of the **Import dashboard** screen. 1. Select a folder in the **Folder** menu where you want to save the imported dashboard. For example, select "Loki Monitoring" created in the earlier steps. 1. Click **Import**. The imported files are listed in the Loki Monitoring dashboard folder. To view the dashboards in Grafana: 1. Select **Dashboards** in your Grafana instance. 1. Select **Loki Monitoring**, or the folder where you uploaded the imported dashboards. 1. Select any file in the folder to view the dashboard. ### Add alerts and recording rules to Prometheus or Mimir The rules and alerts need to be installed into a Prometheus instance, Mimir or a Grafana Enterprise Metrics cluster. You can find the YAML files for alerts and rules in the following directories in the Loki repo: * `production/loki-mixin-compiled/alerts.yaml` (Optional) * `production/loki-mixin-compiled/rules.yaml` (Required for some dashboard panels to show data) You use `mimirtool` to load the mixin alerts and rules definitions into a Prometheus instance, Mimir, or a Grafana Enterprise Metrics cluster. The following examples show how to load the mixin alerts and rules into a Grafana Cloud instance. 1. Download [mimirtool](https://github.com/grafana/mimir/releases). 1. Export the authentication credentials for connecting to your Grafana Cloud Mimir instance. ```bash export MIMIR_ADDRESS= export MIMIR_API_USER= export MIMIR_API_KEY= export MIMIR_TENANT_ID= # Same as MIMIR_API_USER when using Grafana Cloud ``` The best place to locate these credentials is to: 1. Sign into [Grafana Cloud](https://grafana.com/auth/sign-in/) and create a new access policy. 1. In the main menu, select **Security > Access Policies**. 1. Click **Create access policy**. 1. Give the policy a **Name** and select the following permissions: - Alerts: Write & Read - Rules: Write & Read 1. Click **Create**. 1. Click **Add Token**. Give the token a name and click **Create**. 1. Collect `URL` and `user` for Prometheus 1. Navigate to the Grafana Cloud Portal **Overview** page. 1. Click the **Details** button for your Alerts instance. 1. From the **Configuring your Alerting Stacks** section, collect the instance **User** and **URL** for **Metrics Authentication Settings**. 1. Using the same terminal we exported the Mimir environment variables into earlier, run the following command to load the recording rules: ```bash mimirtool rules load rules.yaml ``` 1. (Optional) Load the alert rules: ```bash mimirtool rules load alerts.yaml ``` Refer to the [mimirtool](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/) documentation for more information. ## Next steps After you have installed the Loki mixin dashboards, alerts, and recording rules, you can now monitor your production Loki cluster using Grafana. Make sure you review the mixins when you upgrade Loki to make sure you are using the latest version of the mixin. You can now move onto: * **Send Logs:** Ready to start sending your own logs to Loki, there a several methods you can use. For more information, refer to [send data](https://grafana.com/docs/loki//send-data/). * **Query Logs:** LogQL is an extensive query language for logs and contains many tools to improve log retrieval and generate insights. For more information see the [Query section](https://grafana.com/docs/loki//query/). * **Alert:** Lastly you can use the ruler component of Loki to create alerts based on log queries. For more information refer to [Alerting](https://grafana.com/docs/loki//alert/). # Single binary meta-monitoring Meta monitoring for [monolithic mode](https://grafana.com/docs/loki//get-started/deployment-modes/#monolithic-mode) deployments involves some additional configuration. This approach does not use the Kubernetes Monitoring Helm chart. ## Metrics Configure Alloy or Prometheus to scrape the Loki metrics endpoint, adding the additional labels that are expected by the [mixin](https://grafana.com/docs/loki//operations/meta-monitoring/mixins) dashboards, alerts, and recording rules: ### Alloy ```alloy prometheus.scrape "loki" { targets = [{ __address__ = "localhost:3100", cluster = "prod", namespace = "default", job = "default/loki-single-binary", pod = "loki-single-binary", container = "loki", }] forward_to = [prometheus.remote_write.default.receiver] } prometheus.remote_write "default" { endpoint { url = "" } } ``` ### Prometheus ```yaml scrape_configs: - job_name: loki static_configs: - targets: ['localhost:3100'] labels: cluster: prod namespace: default job: default/loki-single-binary pod: loki-single-binary container: loki ``` ## Mixins To generate the [mixins](https://grafana.com/docs/loki//operations/meta-monitoring/mixins): 1. Install [jb](https://github.com/jsonnet-bundler/jsonnet-bundler). 1. Install [mixtool](https://github.com/monitoring-mixins/mixtool). 1. Clone the Loki repository from Github. ```bash git clone https://github.com/grafana/loki cd loki ``` 1. Check out the tag that matches the version of Loki you are running, so that the dashboards, alerts, and recording rules match the metrics your Loki cluster exposes. For example, if you are running Loki 3.7.6: ```bash git checkout v3.7.6 ``` 1. Create the output directory. ```bash mkdir production/loki-mixin-compiled-single-binary ``` 1. Navigate to the mixin directory. ```bash cd production/loki-mixin ``` 1. Use `jb` to install mixin dependencies and generate the `vendor` directory. ```bash jb install ``` 1. Create the mixin configuration file `single-binary.libsonnet`. ```bash cat < single-binary.libsonnet local loki = import 'mixin.libsonnet'; loki + { _config+:: { meta_monitoring+: { enabled: true, }, canary+: { // Whether or not to include the loki-canary dashboard. // Set to false if you don't run loki-canary. enabled: true, }, promtail+: { // Whether or not to include promtail specific dashboards enabled: false, }, // Tunes histogram recording rules to aggregate over this interval. // Set to at least twice the scrape interval; otherwise, recording rules will output no data. // Set to four times the scrape interval to account for edge cases: https://www.robustperception.io/what-range-should-i-use-with-rate/ recording_rules_range_interval: '5m', }, } EOF ``` 1. Generate dashboards, alerts, and recording rules. ```bash mixtool generate all \ --output-alerts ../loki-mixin-compiled-single-binary/alerts.yaml \ --output-rules ../loki-mixin-compiled-single-binary/rules.yaml \ --directory ../loki-mixin-compiled-single-binary/dashboards \ single-binary.libsonnet ``` 1. See the generated dashboards, alerts, and recording rules in the `production/loki-mixin-compiled-single-binary` directory. 1. Follow the instructions in the [Install Mixins](https://grafana.com/docs/loki//operations/meta-monitoring/mixins) documentation to set up the dashboards, alerts, and recordings. # Horizontal scaling of Compactor {{< admonition type="caution" >}} Compactor horizontal scaling is an experimental feature. Use it with caution in your production environments. {{< /admonition >}} Grafana Loki saw a major improvement in its operational complexity and cost-effectiveness with the introduction of object-storage-based indexes. This change also led to the addition of a singleton Compactor service, initially responsible only for index compaction. However, as new features like [Custom Retention](https://grafana.com/docs/loki//operations/storage/retention/) and [Deletion of Logs with line filters](https://grafana.com/docs/loki//operations/storage/logs-deletion/) were introduced, the Compactor's responsibilities grew. With increasing scale and more demanding features, especially log deletion with line filters, the singleton Compactor began to show its scaling limits. Now, you can run the Loki Compactor in a horizontally scalable mode. Since log deletion with line filters is the compactor's most operationally intensive work, initially, this horizontally scalable architecture will be leveraged to speed up and distribute the workload specifically for its log deletion with line filters functionality. ## How compactor horizontal scaling works We have introduced two new modes to the Compactor for operating in horizontally scalable mode: 1. Main: * Runs all Compactor functions and distributes chunk processing for log line deletion with filters to the workers. * Should be deployed as a singleton with access to a disk, like the current singleton deployment. 2. Worker: * Connects to the Main Compactor over gRPC to get and execute the jobs. * Multiple replicas can be deployed to achieve higher job processing throughput. * Only needs access to Object Storage for reading/writing chunks. ### Compactor horizontal scaling implementation details Although, Horizontally Scalable Compactor currently only supports distributing the work of log line deletion with filters, in the future, we might add support for distributing more kind of work to the Workers. We are going to use the current functionality to see in detail some of the implementation details. #### Main mode For Distribution of Chunk processing work from Main compactor, there are following core components involved: 1. **Deletion Manifest Builder**: The manifest builder works on a batch of delete request to discover all the chunks covered by them based on their labels filters and time range. Using the discovered chunks, it creates structured manifests and stores it to the Object Storage. Here is what comprises Deletion Manifest: - *Segments*: Groups of up to 100K chunks per segment. Also acts as partition for chunks by Loki Tenant/Table. - *Manifest*: Complete metadata about all segments and requests to process. 2. **Job Builder**: The job builder converts manifests into discrete jobs and manages their lifecycle: - *Job Creation*: Breaks segments into jobs of up to 1K chunks each. Also includes line filters to apply on the chunks to remove the log lines requested for deletion. - *Progress Tracking*: Monitors job completion and stops processing of manifest when any of the jobs fail. - *Job Queueing*: Sends jobs to the Job Queue for processing. - *Storage Updates*: Collects storage updates suggested by Workers and stores them to Object Storage for each segment. - *Post-processing Cleanup*: Marks requests as processed and removes all the files from the storage. 3. **Job Queue**: The job queue manages job distribution and Worker-Job Builder communication. - *Job Distribution*: Sends jobs to available workers via gRPC streaming. - *Retry Logic*: Automatically retries failed or timed-out jobs until allowed number of attempts. - *Job Response*: Sends the job processing response to Job Builder. Also notifies about failed jobs after running out of retries. #### Worker mode Workers connect to the Main Compactor via gRPC to fetch and execute jobs, returning results on the same gRPC stream. It uses `compactor_grpc_address` under the [common config](https://grafana.com/docs/loki//configure/#common) to connect to the Main Compactor. When handling Deletion jobs, a worker downloads the listed chunks, applies the specified filters to rebuild them without the filtered lines, and then provides a comprehensive storage update as job execution response. The storage update details which chunks to delete from Object Storage and which newly created chunks to index. ## Configure compactor mode The `horizontal_scaling_mode` configuration option in the compactor controls the enablement of horizontally scalable compactor deployment. It supports setting the following modes: - **`disabled`** (default): Keeps the horizontal scaling mode disabled. Locally runs all the functions of the compactor. - **`main`**: Runs all functions of the compactor. Distributes work to workers where possible. - **`worker`**: Runs the compactor in worker mode, only working on jobs built by the main compactor. ### Configure Main mode To run Compactor in Main mode, the Horizontal Scaling Mode needs to be set to "main": ```yaml compactor: # CLI flag: -compactor.horizontal-scaling-mode horizontal_scaling_mode: "main" ``` Additionally, there are the below config options available for the Main compactor to configure some aspects of Job Building: ```yaml compactor: jobs_config: deletion: # Object storage path prefix for storing deletion manifests. # CLI flag: -compactor.jobs.deletion.deletion-manifest-store-prefix [deletion_manifest_store_prefix: | default = "__deletion_manifest__/"] # Maximum time to wait for a job before considering it failed and retrying. # CLI flag: -compactor.jobs.deletion.timeout [timeout: | default = 15m] # Maximum number of times to retry a failed or timed out job. # CLI flag: -compactor.jobs.deletion.max-retries [max_retries: | default = 3] ``` ### Configure Worker mode To run Compactor in Worker mode, the Horizontal Scaling Mode needs to be set to "worker" and Main compactor's GRPC address needs to be set: ```yaml common: compactor_grpc_address: : compactor: # CLI flag: -compactor.horizontal-scaling-mode horizontal_scaling_mode: "worker" ``` Additionally, there are the below config options available for the Worker to configure some aspects of Job execution: ```yaml compactor: jobs_config: deletion: # Maximum number of chunks to process concurrently in each worker. # CLI flag: -compactor.jobs.deletion.chunk-processing-concurrency [chunk_processing_concurrency: | default = 3] worker_config: # Number of sub-workers to run for concurrent processing of jobs. This # value must be greater than 0, or the compactor fails to start. # CLI flag: -compactor.worker.num-sub-workers [num_sub_workers: | default = 4] ``` # Filesystem object store The filesystem object store is the easiest way to get started with Grafana Loki. This topic describes its pros and cons. The filesystem object store keeps every object, such as chunks, in a directory that you specify. The recommended way to configure this directory is in the `common.storage.filesystem` block: ```yaml common: storage: filesystem: chunks_directory: /tmp/loki/chunks rules_directory: /tmp/loki/rules ``` You can also configure the chunks directory directly under `storage_config`: ```yaml storage_config: filesystem: directory: /tmp/loki/ ``` Loki creates a folder for every tenant. All the chunks for one tenant are stored in that tenant's folder. If you run Loki in single-tenant mode, Loki puts all the chunks in a folder named `fake`. This is the synthesized tenant name that Loki uses in single-tenant mode. See [multi-tenancy](https://grafana.com/docs/loki//operations/multi-tenancy/) for more information. {{< admonition type="note" >}} Loki also supports an alternative, opt-in filesystem client based on `thanos-io/objstore`. Enable it by setting `use_thanos_objstore: true` and configuring `storage_config.object_store.filesystem.dir`. This client will become the default way to configure object store clients in a future release. For configuration examples, see [Thanos storage examples](https://grafana.com/docs/loki//configure/examples/thanos-storage-configs/) and the [storage client migration guide](https://grafana.com/docs/loki//setup/migrate/migrate-storage-clients/). {{< /admonition >}} ## Pros The filesystem object store is very simple. It requires no additional software to run Loki, and it works with TSDB, which is the recommended index store. It's great for low volume applications, proof of concepts, and just playing around with Loki. ## Cons Grafana Labs does not support the filesystem object store for production environments, including for customers who have purchased a support contract. ### Scaling At some point there is a limit to how many chunks you can store in a single directory. For example, see [issue #1502](https://github.com/grafana/loki/issues/1502), which explains how a Loki user ran into a strange error with about **5.5 million chunk files** in their file store. That issue also describes a workaround for the problem. Loki writes one chunk per stream, so keeping the number of active streams low reduces the number of chunk files. You can also tune the following ingester settings to reduce how many chunks get flushed, although lower flush rates trade off for higher memory consumption: - `chunk_target_size` (default 1.5 MB): a target compressed size for each chunk. - `max_chunk_age` (default 2h): the maximum time a chunk stays in memory before Loki flushes it. Consider increasing this beyond the default. - `chunk_idle_period` (default 30m): how long an idle chunk stays in memory before Loki flushes it. Consider increasing this to match `max_chunk_age`. It's still possible to store terabytes of log data with the filesystem store, but keep in mind the limitations on how many files a filesystem can efficiently store in a single directory. ### Durability The durability of your objects depends entirely on the underlying filesystem. Other object stores, such as S3 and GCS, do a lot of work behind the scenes to offer much higher durability for your data. ### High availability Running Loki as a cluster is not possible with the filesystem store, unless you share the filesystem in some way, for example over NFS. Using a shared filesystem is likely to give you a poor experience with Loki, just as it does with almost every other application. ### Retention and deletion When you use the filesystem chunk store, Loki does not delete chunks based on disk usage or free space. Loki only deletes data according to your retention configuration, so you must handle disk-full scenarios yourself, outside of Loki. Retention-based deletion runs through the compactor. When you enable retention with `compactor.retention_enabled`, you must also set `compactor.delete_request_store`. Set it to `filesystem` to store delete requests in this object store. For more information, see [Retention](https://grafana.com/docs/loki//operations/storage/retention/) and the [storage retention section](https://grafana.com/docs/loki//configure/storage/#retention). # Log entry deletion Grafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are those that will be deleted. Log entry deletion is supported when the TSDB index is configured as the index store. It is also supported on the deprecated BoltDB Shipper index, but BoltDB Shipper is being removed in Loki 4.0, so new deployments should use TSDB. The compactor component exposes REST [endpoints](https://grafana.com/docs/loki//reference/loki-http-api#compactor) that process delete requests. Hitting the endpoint specifies the streams and the time window. The deletion of the log entries takes place after a configurable cancellation time period expires. Log entry deletion relies on configuration of the custom logs retention workflow as defined for the [compactor](https://grafana.com/docs/loki//operations/storage/retention/#compactor). The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not. ## Configuration Enable log entry deletion by setting `retention_enabled` to `true` in the compactor's configuration (`-compactor.retention-enabled` on the command line). The tenant's `deletion_mode` must also be `filter-only` or `filter-and-delete`, which is the case by default. `delete_request_store` also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests. {{< admonition type="warning" >}} Be very careful when enabling retention. It is strongly recommended that you also enable versioning on your objects in object storage to allow you to recover from accidental misconfiguration of a retention setting. If you want to enable deletion but do not want to enforce retention, configure the `retention_period` setting with a value of `0s`. {{< /admonition >}} `deletion_mode` is a global and per-tenant setting in [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config). Its default value is `filter-and-delete`, so log entry deletion is active for every tenant as soon as `retention_enabled` is `true` and `delete_request_store` is configured, unless you explicitly change the mode. Set `deletion_mode` in the main configuration file for a global default, or override it per tenant in the [runtime configuration file](https://grafana.com/docs/loki//configure/#runtime-configuration-file). `deletion_mode` supports three values: - `disabled`: Deletion is not allowed. Requests to the deletion API endpoints are rejected with a `403 Forbidden` response. - `filter-only`: Log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage. - `filter-and-delete`: Log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage. This is the default. A delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the compactor's YAML configuration or on the command line when invoking Loki. Its default value is 24h. To cancel a delete request that has already started processing, pass the `force=true` query parameter to the cancellation endpoint. Delete requests that include a line filter are split into smaller requests that each cover no more than `delete_max_interval` (24h by default). You can request a smaller split for a single request with the `max_interval` query parameter, but it cannot be larger than `delete_max_interval`. Delete requests without a line filter are not split. Refer to the [HTTP API reference](https://grafana.com/docs/loki//reference/loki-http-api/#request-log-deletion) for the full set of request parameters. Delete requests themselves are stored using the database type set by `delete_request_store_db_type`, which defaults to `boltdb`. You can instead use `sqlite`. When you migrate from one database type to another, you can set `backup_delete_request_store_db_type` to `boltdb` so that delete requests are also written to a backup database. Refer to the [compactor configuration reference](https://grafana.com/docs/loki//configure/#compactor) for details. The deletion API endpoints are registered only when `compactor.retention_enabled` is `true`. When retention is not enabled, the endpoints are unavailable for every tenant, whatever the `deletion_mode` value is. When retention is enabled, use the `deletion_mode` tenant override to control which tenants can use the deletion API. {{< admonition type="note" >}} Deletion of log lines with line filters is one of the compactor's most resource-intensive operations. If you delete large volumes of data with line filters, refer to [Horizontal scaling of Compactor](https://grafana.com/docs/loki//operations/storage/compactor-horizontal-scaling/) to distribute that work across multiple compactor instances. {{< /admonition >}} # Log retention Retention in Grafana Loki is achieved through the [Compactor](#compactor). By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever. {{< admonition type="note" >}} If you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period. {{< /admonition >}} Granular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor. ## Compactor The Compactor is responsible for compaction of index files and applying log retention. {{< admonition type="note" >}} Run the Compactor as a singleton (a single instance). {{< /admonition >}} The Compactor runs compaction and retention on two independent schedules: - Compaction runs every `compactor.compaction-interval`. - Retention runs every `compactor.apply-retention-interval`. If you leave this at its default of `0s`, Loki sets it to the same value as `compactor.compaction-interval`. Whenever the two intervals are equal, Loki extends the retention interval by the smaller of `10m` or half the interval, so that compaction and retention do not run at the same time. For example, with the default `compaction_interval` of `10m`, retention runs every `15m`. If the Compactor falls behind on either schedule, it runs the corresponding operation as soon as possible. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off. {{< admonition type="note" >}} Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested. {{< /admonition >}} The Compactor's algorithm to apply retention is as follows: - For each day or table (one table per day with 24h index period): - Compact multiple index files in the table into per-tenant index files. Result of compaction is a single index file per tenant per day. - Traverse the per-tenant index. Use the tenant configuration to identify the chunks that need to be removed. - Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk. - Upload the new modified index files. Chunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting `-compactor.retention-delete-delay`. Marker files are used to keep track of the chunks pending for deletion. Chunks cannot be deleted immediately for the following reasons: - Index Gateway downloads a copy of the index files to serve queries and refreshes them at a regular interval. Having a delay allows the index gateways to pull the modified index file which would not contain any reference to the chunks marked for deletion. Without the delay, index files (that are stale) on the gateways could refer to already deleted chunks leading to query failures. - It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake. By default, marker files are written to local disk, under `/retention/_/markers/`. Marker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts. {{< admonition type="note" >}} Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files. {{< /admonition >}} Alternatively, you can configure the Compactor to store marker files in object storage instead of local disk, by setting `compactor.deletion-marker-object-store-prefix` (`deletion_marker_object_store_prefix`) to a prefix that ends with `/`. When set, the Compactor writes marker files to the same per-period object store bucket used for chunks, under that prefix. On startup it also copies any existing marker files from local disk to that prefix, then removes the local copies. This lets you run the Compactor without a persistent volume for marker files. Leave this setting empty to keep using local disk. ### Retention Configuration This Compactor configuration example activates retention. ```yaml compactor: working_directory: /data/retention compaction_interval: 10m retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 delete_request_store: gcs schema_config: configs: - from: "2020-07-31" index: period: 24h prefix: index_ object_store: gcs schema: v13 store: tsdb storage_config: tsdb_shipper: active_index_directory: /data/index cache_location: /data/index_cache gcs: bucket_name: loki ``` {{< admonition type="note" >}} Retention is only available if the index period is 24h. Single store TSDB requires a 24h index period. {{< /admonition >}} `retention_enabled` should be set to true. Without this, the Compactor will only compact tables. `delete_request_store` should be set to configure the store for delete requests. This is required when retention is enabled. `working_directory` is the directory where marked chunks and temporary tables will be saved. `compaction_interval` dictates how often compaction is applied. If the Compactor falls behind, compaction occurs as soon as possible. `apply_retention_interval` dictates how often retention is applied, independently of `compaction_interval`. Refer to [Compactor](#compactor) for how the default value is derived from `compaction_interval`. `retention_delete_delay` is the delay after which the Compactor will delete marked chunks. `retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks. A few other options are also relevant when tuning retention: - `retention_table_timeout` bounds how long the Compactor spends applying retention and processing delete requests on any single table. It defaults to `0s`, meaning no timeout. If the work on a table exceeds this timeout, the Compactor logs a warning and stops early rather than failing, and the remaining work is picked up again on a later run. - `retention_backoff_config` controls the retry behavior (minimum and maximum backoff, and number of retries) used by the sweeper when it fails to delete a marked chunk. - `delete_request_store_key_prefix` sets the path prefix used to store delete requests within `delete_request_store` (default `index/`). - `delete_request_store_db_type` sets the type of database used to store delete requests (default `boltdb`). Refer to the [Configuration reference](https://grafana.com/docs/loki//configure/#compactor) for the full list of `compactor` options and their defaults. #### Configuring the retention period Retention period is configured within the [`limits_config`](https://grafana.com/docs/loki//configure/#limits_config) configuration section. There are two ways of setting retention policies: - `retention_period` which is applied globally for all log streams. - `retention_stream` which is only applied to log streams matching the selector. {{< admonition type="note" >}} Each `retention_stream` rule's `period` must be at least 24h. Loki rejects configuration with a shorter value, both in the global `retention_stream` list and in any per-tenant override. This minimum does not apply to the `retention_period` setting, either globally or per tenant. {{< /admonition >}} This example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides): ```yaml ... limits_config: retention_period: 744h retention_stream: - selector: '{namespace="dev"}' priority: 1 period: 24h runtime_config: file: /etc/overrides.yaml ... ``` {{< admonition type="note" >}} You can only use label matchers in the `selector` field of a `retention_stream` definition. Arbitrary LogQL expressions are not supported. {{< /admonition >}} Per tenant retention can be defined by configuring [runtime overrides](https://grafana.com/docs/loki//configure/#runtime-configuration-file). For example: ```yaml overrides: "29": retention_period: 168h retention_stream: - selector: '{namespace="prod"}' priority: 2 period: 336h - selector: '{container="loki"}' priority: 1 period: 72h "30": retention_stream: - selector: '{container="nginx", level="debug"}' priority: 1 period: 24h ``` Retention period for a given stream is decided based on the first match in this list: 1. If multiple per-tenant `retention_stream` selectors match the stream, retention period with the highest priority is picked. 2. If multiple global `retention_stream` selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant `retention_stream` is set. 3. If a per-tenant `retention_period` is specified, it will be applied. 4. The global `retention_period` will be applied if none of the above match. 5. If no global `retention_period` is specified, the default value of `0s` is used, which means logs are kept indefinitely. {{< admonition type="note" >}} The larger the priority value, the higher the priority. If two matching `retention_stream` rules have the same priority, Loki applies the shorter (lowest) of the two periods. {{< /admonition >}} Stream matching uses the same syntax as Prometheus label matching: - `=`: Select labels that are exactly equal to the provided string. - `!=`: Select labels that are not equal to the provided string. - `=~`: Select labels that regex-match the provided string. - `!~`: Select labels that do not regex-match the provided string. The example configurations defined above will result in the following retention periods: - For tenant `29`: - Streams that have the namespace label `prod` will have a retention period of `336h` (2 weeks), even if the container label is `loki`, since the priority of the `prod` rule is higher. - Streams that have the container label `loki` but are not in the namespace `prod` will have a `72h` retention period. - For the rest of the streams in this tenant, per-tenant override `retention_period` value of `168h` is applied. - For tenant `30`: - Streams that have the label `nginx` and level `debug` will have a retention period of `24h`. - For the rest of the streams in this tenant the global retention period of `744h`, since there is no override specified. - All tenants except `29` and `30`: - Streams that have the namespace label `dev` will have a retention period of `24h` hours. - Streams except those with the namespace label `dev` will have the retention period of `744h`. {{< admonition type="note" >}} If you are a Grafana Cloud customer, you can use the [config self-serve API](https://grafana.com/docs/grafana-cloud/send-data/logs/config-self-serve/#configure-retention) to configure your tenant retention. {{< /admonition >}} ## Example Configuration Example configuration with GCS with a 28 day retention: ```yaml schema_config: configs: - from: 2018-04-15 store: tsdb object_store: gcs schema: v13 index: prefix: loki_index_ period: 24h storage_config: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache gcs: bucket_name: GCS_BUCKET_NAME limits_config: max_query_lookback: 672h # 28 days retention_period: 672h # 28 days compactor: working_directory: /data/retention delete_request_store: gcs retention_enabled: true ``` ## Object store lifecycle policies Loki stores chunks and indexes as objects in the configured object store (S3, GCS, Azure Blob Storage, and so on). Chunks are deleted by the [Compactor](#compactor) according to your configured retention, but the Compactor only removes chunk objects: it never deletes the objects that hold cluster state, such as the index or the cluster seed. If you also configure an object store lifecycle policy on the bucket, scope it carefully. A blanket "delete everything older than N days" rule will eventually remove objects that Loki must keep, which corrupts the store and causes query failures. The note at the top of this page applies here as well: any lifecycle expiration must be longer than your retention period. {{< admonition type="warning" >}} Never apply a lifecycle rule with an empty prefix (one that targets the whole bucket). Always scope lifecycle rules to the per-tenant chunk prefixes described below. {{< /admonition >}} ### Bucket object layout The following objects are written to the configured bucket. The exact prefixes depend on your `schema_config` and on whether multi-tenancy is enabled, so confirm them against your own configuration before writing a lifecycle rule. | Object or prefix | Content | Safe to expire with a lifecycle policy? | |---|---|---| | `/` (for example, `fake/` in single-tenant mode, or your org ID per tenant) | Chunk objects (the log data). | Yes. These are the only objects a lifecycle rule should target. | | Index objects (under the index path prefix, `index/` by default, set by `index.path_prefix` in `schema_config`) | TSDB or BoltDB index files, written by ingesters and rewritten by the Compactor. The `index.prefix` value (for example `index_` or `loki_index_`) is the table-name prefix used inside this path, not the path itself, so match a lifecycle rule against the path prefix. | No. Loki manages the index lifecycle internally. Deleting index objects removes references to live chunks and breaks queries. | | `loki_cluster_seed.json` | A small file at the bucket root holding the cluster seed used by usage reporting. It is written once and never rotated. | No. A blanket age-based rule will delete it once it ages past the time to live (TTL). | | Delete-request store objects (a `delete_requests/` table under `delete_request_store_key_prefix`, `index/` by default, in the bucket configured by `delete_request_store`) | The Compactor's database of delete requests submitted through the [Log entry deletion](https://grafana.com/docs/loki//operations/storage/logs-deletion/) API, and their processing status. | No. Removing these makes the Compactor lose track of pending delete requests. | | Deletion marker objects (only present if you set `deletion_marker_object_store_prefix`; otherwise markers are kept on the Compactor's local disk and never appear in the bucket) | The Compactor's record of chunks that are marked for deletion but not yet swept. Written under the configured prefix in the same per-period bucket used for chunks. | No. Removing these makes the Compactor lose track of chunks pending deletion. | | Ruler store objects (if the Ruler is backed by the same bucket) | Recording and alerting rule files. | No. These are configuration, not time-series data. | ### Compactor retention versus object store lifecycle policies Compactor retention and object store lifecycle policies are two independent mechanisms, and they should not be configured in ways that conflict: - The Compactor is index-aware. It only deletes a chunk after it has removed every reference to that chunk from the index, and it waits `compactor.retention-delete-delay` (default `2h`) before the sweeper deletes the marked chunk, so that index gateways pick up the rewritten index first. - An object store lifecycle policy is index-unaware. It deletes objects purely by age and prefix, with no knowledge of whether a chunk is still referenced by the index. For most deployments, enabling Compactor retention is sufficient and an object store lifecycle policy is not required. If you do add a lifecycle policy as a cost safety net, set its expiration **longer** than your retention period plus `compactor.retention-delete-delay`, so the Compactor always deletes chunks first and the lifecycle rule only ever catches objects the Compactor has already orphaned. ### Example: Amazon S3 lifecycle policy This policy expires only chunk objects under a single tenant prefix. Replace `fake/` with your tenant prefix (use one rule per tenant, or your org ID prefix, when multi-tenancy is enabled). The `Days` value must be larger than your retention period plus the delete delay. ```json { "Rules": [ { "ID": "loki-chunk-expiration", "Status": "Enabled", "Filter": { "Prefix": "fake/" }, "Expiration": { "Days": 395 } } ] } ``` Apply it with the AWS CLI: ```bash aws s3api put-bucket-lifecycle-configuration \ --bucket YOUR_LOKI_BUCKET \ --lifecycle-configuration file://lifecycle.json ``` ### Example: Google Cloud Storage lifecycle rule The equivalent GCS rule expires objects by age and prefix. As with S3, scope `matchesPrefix` to the tenant chunk prefix and keep `age` larger than your retention period plus the delete delay. ```json { "rule": [ { "action": {"type": "Delete"}, "condition": { "age": 395, "matchesPrefix": ["fake/"] } } ] } ``` Apply it with the Google Cloud CLI: ```bash gcloud storage buckets update gs://YOUR_LOKI_BUCKET --lifecycle-file=lifecycle.json ``` For Azure Blob Storage, use a management policy with a `prefixMatch` set to the tenant chunk prefix and the same "expiration must exceed retention plus delete delay" rule. # Single Store TSDB (tsdb) Starting with Loki v2.8, TSDB is the recommended Loki index. It is heavily inspired by the Prometheus TSDB [sub-project](https://github.com/prometheus/prometheus/tree/main/tsdb). This new index is more efficient, faster, and more scalable. It also resides in object storage like the [boltdb-shipper](https://grafana.com/docs/loki//operations/storage/boltdb-shipper/) index which preceded it. For the on-disk binary layout of the index files, see [TSDB index format](https://grafana.com/docs/loki//reference/tsdb-index-format/). ## Example Configuration To get started using TSDB, add the following configurations to your `config.yaml`: ```yaml schema_config: configs: # Old boltdb-shipper schema. Included in example for reference but does not need changing. - from: "2023-01-03" # <---- A date in the past index: period: 24h prefix: index_ object_store: gcs schema: v12 store: boltdb-shipper # New TSDB schema below - from: "2023-01-05" # <---- A date in the future index: period: 24h prefix: index_ object_store: gcs schema: v13 store: tsdb storage_config: # Old boltdb-shipper configuration. Included in example for reference but does not need changing. boltdb_shipper: active_index_directory: /data/index build_per_tenant_index: true cache_location: /data/boltdb-cache index_gateway_client: # only applicable if using microservices where index-gateways are independently deployed. # This example is using kubernetes-style naming. server_address: dns:///index-gateway..svc.cluster.local:9095 # New tsdb-shipper configuration tsdb_shipper: active_index_directory: /data/tsdb-index cache_location: /data/tsdb-cache index_gateway_client: # only applicable if using microservices where index-gateways are independently deployed. # This example is using kubernetes-style naming. server_address: dns:///index-gateway..svc.cluster.local:9095 query_scheduler: # the TSDB index dispatches many more, but each individually smaller, requests. # We increase the pending request queue sizes to compensate. max_outstanding_requests_per_tenant: 32768 querier: # Each `querier` component process runs a number of parallel workers to process queries simultaneously. # You may want to adjust this up or down depending on your resource usage # (more available cpu and memory can tolerate higher values and vice versa), # but we find the most success running at around `16` with tsdb max_concurrent: 16 ``` ## Operations ### Limits We've added a user per-tenant limit called `tsdb_max_query_parallelism` in the `limits_config`. This functions the same as the prior `max_query_parallelism` configuration but applies to tsdb queries instead. Since the TSDB index will create many more smaller queries compared to the other index types before it, we've added a separate configuration so they can coexist. This is helpful when transitioning between index types. The default parallelism is `128` which should work well for most cases, but you can extend it globally in the `limits_config` or per-tenant in the `overrides` file as needed. ### Dynamic Query Sharding Previously we would statically shard queries based on the index row shards configured [here](https://grafana.com/docs/loki//configure/#period_config). TSDB does Dynamic Query Sharding based on how much data a query is going to be processing. We additionally store size(KB) and number of lines for each chunk in the TSDB index which is then used by the [Query Frontend](https://grafana.com/docs/loki//get-started/components/#query-frontend) for planning the query. Based on our experience from operating many Loki clusters, we have configured TSDB to aim for processing 300-600 MBs of data per query shard. This means with TSDB we will be running more, smaller queries. The per-tenant `tsdb_max_bytes_per_shard` setting in `limits_config` controls this target size. It defaults to `600MB`. Each time a query would need to process more data than this target, Loki doubles the number of shards for that query. As a result, each shard typically ends up processing between half and all of the target amount. This is why, with the default target of `600MB`, query shards typically process between 300 MB and 600 MB of data each. Loki also supports the following related per-tenant settings in `limits_config`: - `tsdb_sharding_strategy`: The sharding strategy to use during query planning. The default is `power_of_two`, the strategy described above. An alternative, `bounded`, is also available; refer to the [configuration reference](https://grafana.com/docs/loki//configure/#limits_config) before switching, since not all Loki components in a cluster may support it during a rolling upgrade. - `tsdb_precompute_chunks`: When set to `true`, Loki computes chunk references once during query planning instead of resolving them again from the index during query execution. This can reduce the number of index calls a query makes, at the cost of higher memory usage. It defaults to `false`. This setting only takes effect when `tsdb_sharding_strategy` is set to `bounded`, because the `power_of_two` strategy does not support precomputed chunk references. ### Index Caching not required TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of [retention](https://grafana.com/docs/loki//operations/storage/retention/) or your configured `max_query_lookback` under [limits_config](https://grafana.com/docs/loki//configure/#limits_config). After that, we suggest running without an index cache (it isn't used in TSDB). # Write Ahead Log Ingesters temporarily store data in memory. In the event of a crash, there could be data loss. The Write Ahead Log (WAL) helps fill this gap in reliability. The WAL in Grafana Loki records incoming data and stores it on the local file system in order to guarantee persistence of acknowledged data in the event of a process crash. Upon restart, Loki will "replay" all of the data in the log before registering itself as ready for subsequent writes. This allows Loki to maintain the performance and cost benefits of buffering data in memory _and_ durability benefits (it won't lose data once a write has been acknowledged). This section will use Kubernetes as a reference deployment paradigm in the examples. ## Disclaimer and WAL nuances The Write Ahead Log in Loki takes a few particular tradeoffs compared to other WALs you may be familiar with. The WAL aims to add additional durability guarantees, but _not at the expense of availability_. Particularly, there are two scenarios where the WAL sacrifices these guarantees. 1. Corruption/Deletion of the WAL prior to replaying it In the event the WAL is corrupted/partially deleted, Loki will not be able to recover all of its data. In this case, Loki will attempt to recover any data it can, but will not prevent Loki from starting. You can use the Prometheus metric `loki_ingester_wal_corruptions_total` to track and alert when this happens. 1. No space left on disk By default, Loki monitors disk usage on the volume that stores the WAL. If usage reaches 90% of capacity, the ingester rejects new writes instead of accepting data it cannot persist. This check runs every 10 seconds and measures the entire filesystem that holds the WAL directory, not just the size of the WAL files. Because of this, you should not share the WAL volume with other data. Writes resume automatically once disk usage drops back below the threshold. You can change the threshold with the `--ingester.wal-disk-full-threshold` flag, or set it to `0` to disable this protection. This throttling behavior isn't available on Windows. If you disable this threshold, or if a write reaches the disk in the short window before the check runs, the underlying disk can still become completely full. In that case, Loki will not fail the incoming write, but neither will it log the entry to the WAL. In this case, the persistence guarantees across process restarts will not hold for that entry. You can use the Prometheus metric `loki_ingester_wal_disk_full_failures_total` to track and alert when this happens. This counter increases both when disk usage crosses the throttling threshold and when a WAL write fails because the disk is completely full, so a nonzero value means the WAL disk needs attention, but it does not count each individual rejected write. You can also monitor the gauge `loki_ingester_wal_disk_usage_percent` to see current disk usage and how close it is to the threshold. ### Backpressure The WAL also includes a backpressure mechanism to allow a large WAL to be replayed within a smaller memory bound. This is helpful after bad scenarios (i.e. an outage) when a WAL has grown past the point it may be recovered in memory. In this case, the ingester will track the amount of data being replayed and once it's passed 90% of the `ingester.wal-replay-memory-ceiling` threshold, will flush to storage. When this happens, it's likely that the Loki attempt to deduplicate chunks via content addressable storage will suffer. We deemed this efficiency loss an acceptable tradeoff considering how it simplifies operation and that it should not occur during regular operation (rollouts, rescheduling) where the WAL can be replayed without triggering this threshold. Setting `ingester.wal-replay-memory-ceiling` to `0` disables backpressure entirely. ### Metrics The following metrics are available for monitoring the WAL: * `loki_ingester_wal_corruptions_total`: Total number of WAL corruptions encountered * `loki_ingester_wal_disk_full_failures_total`: Increments when disk usage crosses the `--ingester.wal-disk-full-threshold` throttling threshold, and when a WAL write fails because the disk is completely full. It does not count each individual throttled write. * `loki_ingester_wal_disk_usage_percent`: Current disk usage, from 0.0 to 1.0, for the volume that stores the WAL. This is only reported when `--ingester.wal-disk-full-threshold` is greater than `0`. * `loki_ingester_wal_records_logged_total`: Counter for WAL records logged * `loki_ingester_wal_logged_bytes_total`: Total bytes written to WAL * `loki_ingester_wal_replay_active`: Set to 1 while an ingester is replaying its WAL and checkpoint on startup, 0 otherwise * `loki_ingester_wal_replay_duration_seconds`: Time taken to replay the checkpoint and the WAL ## Changes to deployment 1. Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) with fixed volumes. 1. The following flags are relevant when deploying with a persistent volume: * `--ingester.wal-enabled` enables writing to WAL during ingestion. It defaults to `true`, so this typically doesn't need to be set explicitly. * `--ingester.wal-dir` sets the directory where the WAL data should be stored and/or recovered from. Note that this should be on the mounted volume. * `--ingester.checkpoint-duration` to the interval at which checkpoints should be created. * `--ingester.wal-replay-memory-ceiling` (default 4GB) may be set higher/lower depending on your resource settings. It handles memory pressure during WAL replays, allowing a WAL many times larger than available memory to be replayed. This is provided to minimize reconciliation time after very bad situations, i.e. an outage, and will likely not impact regular operations/rollouts _at all_. We suggest setting this to a high percentage (~75%) of available memory. * `--ingester.wal-disk-full-threshold` (default 0.90) sets the fraction of disk usage (0.0 to 1.0) at which the ingester throttles incoming writes to protect WAL durability. Set to `0` to disable this protection. For details, refer to [Disclaimer and WAL nuances](#disclaimer-and-wal-nuances). ## Changes in lifecycle when WAL is enabled Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL. If you need to ensure that data is always flushed to the chunk store when your pod shuts down, you can set the `--ingester.flush-on-shutdown` flag to `true`. ## Disk space requirements Based on tests in real world: * Numbers from an ingester with 5000 series ingesting ~5mb/s. * Checkpoint period was 5mins. * disk utilization on a WAL-only disk was steady at ~10-15GB. You should not target 100% disk utilization. ## Migrating from stateless deployments The ingester _Deployment without WAL_ and _StatefulSet with WAL_ should be scaled down and up respectively in sync without transfer of data between them to ensure that any ingestion after migration is reliable immediately. Let's take an example of 4 ingesters. The migration would look something like this: 1. Bring up one stateful ingester `ingester-0` and wait until it's ready (accepting read and write requests). 1. Scale down the old ingester deployment to 3 and wait until the leaving ingester flushes all the data to chunk store. 1. Once that ingester has disappeared from `kc get pods ...`, add another stateful ingester and wait until it's ready. Now you have `ingester-0` and `ingester-1`. 1. Repeat step 2 to reduce remove another ingester from old deployment. 1. Repeat step 3 to add another stateful ingester. Now you have `ingester-0 ingester-1 ingester-2`. 1. Repeat step 4 and 5, and now you will finally have `ingester-0 ingester-1 ingester-2 ingester-3`. ## How to scale up/down ### Scale up Scaling up is same as what you would do without WAL or StatefulSets. Nothing to change here. ### Scale down When scaling down, we must ensure existing data on the leaving ingesters are flushed to storage instead of just the WAL. This is because we won't be replaying the WAL on an ingester that will no longer exist and we need to make sure the data is not orphaned. Consider you have 4 ingesters `ingester-0 ingester-1 ingester-2 ingester-3` and you want to scale down to 2 ingesters, the ingesters which will be shut down according to StatefulSet rules are `ingester-3` and then `ingester-2`. Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the [`/ingester/shutdown?flush=true`](https://grafana.com/docs/loki//reference/loki-http-api#flush-in-memory-chunks-and-shut-down) endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted. After hitting the endpoint for `ingester-2 ingester-3`, scale down the ingesters to 2. Also you can set the `--ingester.flush-on-shutdown` flag to `true`. This enables chunks to be flushed to long-term storage when the ingester is shut down. ## Additional notes ### Kubernetes hacking StatefulSets are significantly more cumbersome to work with, upgrade, and so on. Much of this stems from immutable fields on the specification. For example, if one wants to start using the WAL with single store Loki and wants separate volume mounts for the WAL and the TSDB, you may see immutability errors when attempting updates the Kubernetes StatefulSets. In this case, try `kubectl -n delete sts ingester --cascade=false`. This will leave the Pods alive but delete the StatefulSet. Then you may recreate the (updated) StatefulSet and one-by-one start deleting the `ingester-0` through `ingester-n` Pods _in that order_, allowing the StatefulSet to spin up new pods to replace them. #### Scaling Down Using `/flush_shutdown` Endpoint and Lifecycle Hook 1. **StatefulSets for Ordered Scaling Down**: The Loki ingesters should be scaled down one by one, which is efficiently handled by Kubernetes StatefulSets. This ensures an ordered and reliable scaling process, as described in the [Deployment and Scaling Guarantees](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#deployment-and-scaling-guarantees) documentation. 1. **Using PreStop Lifecycle Hook**: During the Pod scaling down process, the PreStop [lifecycle hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/) triggers the `/flush_shutdown` endpoint on the ingester. This action flushes the chunks and removes the ingester from the ring, allowing it to register as unready and become eligible for deletion. 1. **Using terminationGracePeriodSeconds**: Provides time for the ingester to flush its data before being deleted, if flushing data takes more than 30 minutes, you may need to increase it. 1. **Cleaning Persistent Volumes**: Persistent volumes are automatically cleaned up by leveraging the [enableStatefulSetAutoDeletePVC](https://kubernetes.io/blog/2021/12/16/kubernetes-1-23-statefulset-pvc-auto-deletion/) feature in Kubernetes. By following the above steps, you can ensure a smooth scaling down process for the Loki ingesters while maintaining data integrity and minimizing potential disruptions. ### Non-Kubernetes or baremetal deployments * When the ingester restarts for any reason (upgrade, crash, etc), it should be able to attach to the same volume in order to recover back the WAL and tokens. * 2 ingesters should not be working with the same volume/directory for the WAL. * A rollout should bring down an ingester completely and then start the new ingester, not the other way around. # Troubleshoot Logs Drilldown [//]: # 'Shared content for troubleshooting logs drilldown' [//]: # 'This content is located in /logs-drilldown/blob/main/docs/sources/shared/troubleshoot-logs-drilldown.md' {{< docs/shared lookup="troubleshoot-logs-drilldown.md" source="explore-logs" version="next" >}} # Troubleshoot log ingestion (WRITE) This guide helps you troubleshoot errors that occur when ingesting logs into Loki and writing logs to storage. When Loki rejects log ingestion requests, it's typically due to exceeding rate limits, violating validation rules, or encountering storage issues. Before you begin, ensure you have the following: - Access to Grafana Loki logs and metrics - Understanding of your log ingestion pipeline (Alloy or custom clients) - Permissions to configure limits and settings if needed ## Monitoring ingestion errors All ingestion errors are tracked using Prometheus metrics: - `loki_discarded_samples_total` - Count of discarded log samples by reason - `loki_discarded_bytes_total` - Volume of discarded log data by reason The `reason` label on these metrics indicates the specific error type. Set up alerts on these metrics to detect ingestion problems. ## Rate limit errors Rate limits protect Loki from being overwhelmed by excessive log volume. When rate limits are exceeded, Loki returns HTTP status code `429 Too Many Requests`. ### Error: `rate_limited` **Error message:** `ingestion rate limit exceeded for user (limit: bytes/sec) while attempting to ingest lines totaling bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased` **Cause:** The tenant has exceeded their configured ingestion rate limit. This is a global per-tenant limit enforced by the distributor. **Default configuration:** - `ingestion_rate_mb`: 4 MB/sec - `ingestion_burst_size_mb`: 6 MB - `ingestion_rate_strategy`: "global" (shared across all distributors) **Resolution:** * **Increase rate limits** (if you have sufficient cluster resources): ```yaml limits_config: ingestion_rate_mb: 8 ingestion_burst_size_mb: 12 ``` * **Reduce log volume** by: - Filtering unnecessary log lines using Alloy processing stages - Collecting logs from fewer targets - Sampling high-volume streams - Using Alloy's rate limiting and filtering capabilities * **Switch to local strategy** (if using single-region deployment): ```yaml limits_config: ingestion_rate_strategy: local ``` {{< admonition type="note" >}} This multiplies the effective limit by the number of distributor replicas. {{< /admonition >}} **Properties:** - Enforced by: Distributor - Retryable: Yes - HTTP status: 429 Too Many Requests - Configurable per tenant: Yes ### Error: `per_stream_rate_limit` **Error message:** `Per stream rate limit exceeded (limit: /sec) while attempting to ingest for stream '' totaling , consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased` **Cause:** A single stream has exceeded its own rate limit. Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to. This limit is enforced by the ingester, not the distributor. **Default configuration:** - `per_stream_rate_limit`: 3 MB/sec - `per_stream_rate_limit_burst`: 15 MB **Resolution:** * **Split the stream** into several smaller streams by adding more distinguishing labels, so the write volume is spread across more streams. * **Reduce the rate of writes to the stream**, for example with an Alloy `stage.limit` block. * **Increase the per-stream limits** (we typically recommend no higher than 5MB for `per_stream_rate_limit` and 20MB for `per_stream_rate_limit_burst`): ```yaml limits_config: per_stream_rate_limit: 5MB per_stream_rate_limit_burst: 20MB ``` **Properties:** - Enforced by: Ingester - Retryable: Yes - HTTP status: 429 Too Many Requests - Configurable per tenant: Yes ### Error: `stream_limit` **Error message:** `maximum active stream limit exceeded when trying to create stream , reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: ` **Cause:** The tenant has reached the maximum number of active streams. Active streams are held in memory on ingesters, and excessive streams can cause out-of-memory errors. {{< admonition type="note" >}} Under normal conditions, this limit only blocks the creation of genuinely new streams. Log lines for streams that are already active continue to be accepted. However, during ingester scaling (autoscaling, rollouts, or ring changes such as a partition-ingester migration), streams can be rebalanced to ingesters that were not previously handling them. The new ingester has no record of that stream in memory, so it treats the next write for that stream as a stream creation and checks it against its local share of the limit. If that share is already used up, for example because the same tenant is also adding new high-cardinality streams during the rebalance, writes to the already-active stream can be rejected with `stream_limit`, even though the tenant has not added new label cardinality for it. This effect is temporary. It is bounded by the number of ingesters added or restarted, and it typically resolves once the ring is stable and stream ownership is recalculated. If you see `stream_limit` errors that line up with ingester scaling or rollouts rather than with new label values, this is the likely cause. {{< /admonition >}} **Default configuration:** - `max_global_streams_per_user`: 5000 (globally) - `max_streams_per_user`: 0 (no local limit per ingester) - Active stream window: `chunk_idle_period` (default: 30 minutes) **Terminology:** - **Active stream**: A stream that has received logs within the `chunk_idle_period` (default: 30 minutes) **Resolution:** * **Reduce stream cardinality** by: - Using fewer unique label combinations by removing unnecessary labels - Avoiding high-cardinality labels (IDs, timestamps, UUIDs) - Using structured metadata instead of labels for high-cardinality data * **Increase stream limits** (if you have sufficient memory): ```yaml limits_config: max_global_streams_per_user: 10000 ``` {{< admonition type="note" >}} Do not increase stream limits to accommodate high cardinality labels, this can result in Loki flushing extremely high numbers of small files which will make for extremely poor query performance. As volume and the size of the infrastructure being monitored increase, it would be expected to increase the stream limit. However even for hundreds of TBs per day of logs you should avoid exceeding 300,000 max global streams per user. {{< /admonition >}} * **Check for a recent ingester rollout or scaling event** before increasing limits. If `stream_limit` errors started when ingesters were added, restarted, or migrated, the tenant may not have a real cardinality problem. Increasing the limit will not prevent this transient effect and can mask genuine cardinality growth. **Properties:** - Enforced by: Ingester - Retryable: Yes - HTTP status: 429 Too Many Requests - Configurable per tenant: Yes ## Validation errors Validation errors occur when log data doesn't meet Loki's requirements. These return HTTP status code `400 Bad Request` and are **not retryable**. ### Error: `line_too_long` **Error message:** `max entry size bytes exceeded for stream while adding an entry with length bytes` **Cause:** A log line exceeds the maximum allowed size. **Default configuration:** - `max_line_size`: 256 KB - `max_line_size_truncate`: false **Resolution:** * **Truncate long lines** instead of discarding them: ```yaml limits_config: max_line_size: 256KB max_line_size_truncate: true ``` * **Increase the line size limit** (not recommended above 256KB): ```yaml limits_config: max_line_size: 256KB ``` {{< admonition type="warning" >}} Loki was built as a large multi-user, multi-tenant database and as such this limit becomes very important to maintain stability and performance with many users query the database simultaneously. We strongly recommend against increasing the max_line_size, doing so will make it very difficult to provide consistent query performance and stability of the system without having to throw extremely large amounts of memory and/or increasing the GRPC message size limits to really high levels, both of which will likely lead to poorer performance and worse experiences. {{< /admonition >}} * **Filter or truncate logs before sending** using Alloy processing stages: ```alloy stage.replace { expression = "^(.{10000}).*$" replace = "$1" } ``` **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `invalid_labels` **Error message:** `error parsing labels with error: ` **Cause:** Label names or values contain invalid characters or don't follow Prometheus naming conventions: - Label names must match regex: `[a-zA-Z_][a-zA-Z0-9_]*` - Label names cannot start with `__` (reserved for Grafana internal use) - Labels must be valid UTF-8 - Properly escape special characters in label values - Ensure label syntax follows PromQL format: `{key="value"}` **Default configuration:** This validation is always enabled and cannot be disabled. **Resolution:** * **Fix label names** in your log shipping configuration: ```yaml # Incorrect labels: "123-app": "value" # Starts with number "app-name": "value" # Contains hyphen "__internal": "value" # Reserved prefix # Correct labels: app_123: "value" app_name: "value" internal_label: "value" ``` * **Use Alloy's relabeling stages** to fix labels: ```alloy stage.label_drop { values = ["invalid_label"] } stage.labels { values = { app_name = "", } } ``` **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: No ### Error: `missing_labels` **Error message:** `error at least one label pair is required per stream` **Cause:** A log stream was submitted without any labels. Loki requires at least one label pair for each log stream. **Default configuration:** This validation is always enabled and cannot be disabled. Every stream must have at least one label. **Resolution:** Ensure all log streams have at least one label. Update your log shipping configuration to include labels: ```alloy loki.write "default" { endpoint { url = "http://loki:3100/loki/api/v1/push" } } loki.source.file "logs" { targets = [ { __path__ = "/var/log/*.log", job = "myapp", environment = "production", }, ] forward_to = [loki.write.default.receiver] } ``` **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: No ### Error: `too_far_behind` These errors occur when log entries arrive in the wrong chronological order. **Error messages:** - `entry too far behind, entry timestamp is: , oldest acceptable timestamp is: ` **Cause:** Logs are being ingested with timestamps that violate Loki's ordering constraints: - Logs older than half of `max_chunk_age` from the newest entry are rejected **Default configuration:** - `unordered_writes`: true (since Loki 2.4). This flag, and `-ingester.unordered-writes`, are deprecated and will be removed in a future major release, after which out-of-order writes will be accepted unconditionally. - `max_chunk_age`: 2 hours - Acceptable timestamp window: 1 hour behind the newest entry (half of `max_chunk_age`) **Resolution:** * **Ensure timestamps are assigned correctly**: - Use a timestamp parsing stage in Alloy if logs have embedded timestamps - Avoid letting Alloy assign timestamps at read time for delayed log delivery ```alloy stage.timestamp { source = "timestamp_field" format = "RFC3339" } ``` * **Check for clock skew** between log sources and Loki. Ensure clocks are synchronized across your infrastructure. **Properties:** - Enforced by: Ingester - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes, for `unordered_writes`. No, for `max_chunk_age` (global only). ### Error: `greater_than_max_sample_age` **Error message:** `entry for stream has timestamp too old: , oldest acceptable timestamp is: ` **Cause:** The log entry's timestamp is older than the configured maximum sample age. This prevents ingestion of very old logs. **Default configuration:** - `reject_old_samples`: true - `reject_old_samples_max_age`: 168 hours (7 days) **Resolution:** * **Increase the maximum sample age**: ```yaml limits_config: reject_old_samples_max_age: 336h # 14 days ``` * **Fix log delivery delays** causing old timestamps. * **For historical log imports**, temporarily disable the check per tenant. * **Check for clock skew** between log sources and Loki. Ensure clocks are synchronized across your infrastructure. * **Disable old sample rejection** (not recommended): ```yaml limits_config: reject_old_samples: false ``` **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `too_far_in_future` **Error message:** `entry for stream has timestamp too new: ` **Cause:** The log entry's timestamp is further in the future than the configured grace period allows. **Default configuration:** - `creation_grace_period`: 10 minutes **Resolution:** * **Check for clock skew** between log sources and Loki. Ensure clocks are synchronized across your infrastructure. * **Verify application timestamps** Validate timestamp generation in your applications. * **Verify timestamp parsing** in your log shipping configuration. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `max_label_names_per_series` **Error message:** `entry for stream has label names; limit ` **Cause:** The stream has more labels than allowed. **Default configuration:** - `max_label_names_per_series`: 15 **Resolution:** * **Reduce the number of labels** by: - Using structured metadata for high-cardinality data - Removing unnecessary labels - Combining related labels {{< admonition type="warning" >}} We strongly recommend against increasing `max_label_names_per_series`, doing so creates a larger index which hurts query performance as well as opens the door for cardinality explosions. You should be able to categorize your logs with 15 labels or typically much less. In all our years of running Loki, out of thousands of requests to increase this value, the number of valid exceptions we have seen can be counted on one hand. {{< /admonition >}} **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `label_name_too_long` **Error message:** `stream has label name too long: ` **Cause:** A label name exceeds the maximum allowed length. **Default configuration:** - `max_label_name_length`: 1024 bytes **Resolution:** * **Shorten label names** in your log shipping configuration to under the configured limit. You can use abbreviations or shorter descriptive names. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `label_value_too_long` **Error message:** `stream has label value too long: ` **Cause:** A label value exceeds the maximum allowed length. **Default configuration:** - `max_label_value_length`: 2048 bytes **Resolution:** * **Shorten label values** in your log shipping configuration to under the configured limit. You can use hash values for very long identifiers. * **Use structured metadata** for long values instead of labels. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: `duplicate_label_names` **Error message:** `stream has duplicate label name: ` **Cause:** The stream has two or more labels with identical names. **Default configuration:** This validation is always enabled and cannot be disabled. Duplicate label names are never allowed. **Resolution:** * **Remove duplicates** Remove duplicate label definitions in your log shipping configuration to ensure unique label names per stream. * **Check clients** Check your ingestion pipeline for label conflicts. * **Verify processing** Verify your label processing and transformation rules. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: No ### Error: `request_body_too_large` **Error message:** `Request body too large: bytes, limit: bytes` **Cause:** The HTTP compressed push request body exceeds the configured limit in your gateway/reverse proxy or service that fronts Loki after decompression. **Default configuration:** - Loki server max request body size: Configured at the HTTP server level - Nginx gateway (Helm): `gateway.nginxConfig.clientMaxBodySize` (default: 4 MB) - Alloy default batch size: 1 MB **Resolution:** * **Split large batches** into smaller, more frequent requests. * **Increase batch limit** Increase the allowed body size on your gateway/reverse proxy. For example, in the Helm chart set `gateway.nginxConfig.clientMaxBodySize`; default is 4M. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 413 Request Entity Too Large - Configurable per tenant: No ## Storage errors Storage errors occur when Loki cannot write data to its storage backend or the Write-Ahead Log (WAL). ### Error: Disk space full **Error message:** `no space left on device` **Cause:** Local storage has run out of space. This can affect ingester WAL, chunk cache, or temporary files. **Default configuration:** Disk space is not limited by Loki configuration; it depends on your infrastructure provisioning. **Resolution:** * **Free up disk space** on Loki instances. * **Configure retention policies** to remove old data. * **Scale storage capacity** by adding more disk space. * **Monitor disk usage** and set up alerts before reaching critical levels. * **Review compactor settings** to ensure old data is being cleaned up. **Properties:** - Enforced by: Operating system - Retryable: Yes (after freeing space) - HTTP status: 500 Internal Server Error - Configurable per tenant: No ### Error: Chunk write failure **Error message:** `failed to store chunks: storage unavailable` **Common causes and errors:** * **S3/Object Storage errors:** - `NoSuchBucket`: Storage bucket doesn't exist - `AccessDenied`: Invalid credentials or permissions - `RequestTimeout`: Network or storage latency issues **Resolution:** * **Verify storage configuration**: ```yaml storage_config: aws: s3: s3://region/bucket-name s3forcepathstyle: true ``` * **Check credentials and permissions**: - Ensure service account has write permissions - Verify IAM roles for S3 - Check API keys for cloud storage * **Monitor storage health**: - Check object storage availability * **Review storage metrics**: - `loki_ingester_chunks_flushed_total` - `loki_ingester_chunks_flush_errors_total` * **Increase retries and timeouts**: ```yaml storage_config: aws: s3: http_config: idle_conn_timeout: 90s response_header_timeout: 0s ``` **Properties:** - Enforced by: Ingester (during chunk flush) - Retryable: Yes (with backoff) - HTTP status: 500 Internal Server Error - Configurable per tenant: No ### Error: WAL disk full **Error logged:** `Error writing to WAL, disk full, no further messages will be logged for this error` **Metric:** `loki_ingester_wal_disk_full_failures_total` **Cause:** By default, the ingester monitors disk usage on the volume that stores the WAL and rejects new writes once usage reaches 90% of capacity (`-ingester.wal-disk-full-threshold`), so that it never accepts data it cannot persist. This error and the associated log message describe a different, rarer case: the disk still filled up completely, either because this throttling is disabled (threshold set to `0`), or because a write reached the disk in the short window before the check could catch it. In that case, the ingester continues accepting writes but doesn't log them to the WAL, losing durability guarantees for those entries. **Default configuration:** - WAL enabled by default - WAL location: configured by `-ingester.wal-dir` - Checkpoint interval: `ingester.checkpoint-duration` (default: 5 minutes) - Disk full throttling threshold: `ingester.wal-disk-full-threshold` (default: 0.9, meaning 90%) **Resolution:** * **Increase disk space** for the WAL directory. * **Monitor disk usage** and set up alerts: ```promql loki_ingester_wal_disk_full_failures_total > 0 ``` This counter increases both when disk usage crosses the throttling threshold and when a WAL write fails because the disk is completely full. It does not count each individual throttled write, so also watch the gauge `loki_ingester_wal_disk_usage_percent` to see how close the disk is to the threshold. * **Reduce log volume** to decrease WAL growth. * **Check WAL checkpoint frequency**: ```yaml ingester: checkpoint_duration: 5m ``` * **Verify WAL cleanup** is working - old segments should be deleted after checkpointing. **Properties:** - Enforced by: Ingester - Retryable: N/A (writes are accepted but not persisted to WAL) - HTTP status: N/A (writes succeed, but durability is compromised) - Configurable per tenant: No {{< admonition type="note" >}} This section describes the disk becoming completely full. In normal operation, the ingester's disk usage throttle (`-ingester.wal-disk-full-threshold`, default 90%) rejects writes to that ingester before the disk actually fills up, so that durability guarantees are preserved. Because Loki replicates each write to multiple ingesters, a client write can still succeed even while one ingester is throttled. Use metric `loki_ingester_wal_disk_usage_percent` to monitor disk usage. {{< /admonition >}} ### Error: WAL corruption **Error message:** `encountered WAL read error, attempting repair` **Metric:** `loki_ingester_wal_corruptions_total` **Cause:** The WAL has become corrupted, possibly due to: - Disk errors - Process crashes during write - Filesystem issues - Partial deletion of WAL files **Default configuration:** - WAL enabled by default - Automatic repair attempted on startup - WAL location: configured by `-ingester.wal-dir` **Resolution:** * **Monitor for corruption**: ```promql increase(loki_ingester_wal_corruptions_total[5m]) > 0 ``` * **Automatic recovery**: Loki attempts to recover readable data and continues starting. * **Investigate root cause**: - Check disk health - Review system logs for I/O errors - Verify filesystem integrity **Properties:** - Enforced by: Ingester (on startup) - Retryable: N/A (Loki attempts automatic recovery) - HTTP status: N/A (occurs during startup, not request handling) - Configurable per tenant: No {{< admonition type="note" >}} Loki prioritizes availability over complete data recovery. The replication factor provides redundancy if one ingester loses WAL data. Recovered data may be incomplete if corruption is severe. {{< /admonition >}} ## Network and connectivity errors These errors occur due to network issues or service unavailability. ### Error: Connection refused **Error message:** `connection refused when connecting to loki:3100` **Cause:** The Loki service is unavailable or not listening on the expected port. **Default configuration:** - Loki HTTP listen port: 3100 (`-server.http-listen-port`) - Loki gRPC listen port: 9095 (`-server.grpc-listen-port`) **Resolution:** * **Verify Loki is running** and healthy. * **Check network connectivity** between client and Loki. * **Confirm the correct hostname and port** configuration. * **Review firewall and security group** settings. * **Check for CPU starvation** as a Loki pod that is overwhelmed could fail to respond to a request. **Properties:** - Enforced by: Network/OS - Retryable: Yes (after service recovery) - HTTP status: N/A (connection-level error) - Configurable per tenant: No ### Error: Context deadline exceeded **Error message:** `context deadline exceeded` **Cause:** Requests are timing out due to slow response times or network issues. **Default configuration:** - Alloy default timeout: 10 seconds - Loki server timeout: `-server.http-server-write-timeout` (default: 30s) **Resolution:** * **Increase timeout values** in your ingestion client. * **Check Loki performance** and resource utilization. * **Review network latency** between client and server. * **Scale Loki resources** if needed. **Properties:** - Enforced by: Client/Server - Retryable: Yes - HTTP status: 504 Gateway Timeout (or client-side timeout) - Configurable per tenant: No ### Error: Client canceled request (Code 499) **Error message:** `level=warn method=/logproto.Querier/GetChunkIDs duration= msg=gRPC err="rpc error: code = Code(499) desc = the request was cancelled by the client"` **Cause:** A downstream client (such as Grafana, a load balancer, or another Loki component) canceled a gRPC request before the server finished processing it. `Code(499)` is a non-standard gRPC status code derived from the HTTP 499 status ("Client Closed Request"), which Loki uses internally when it detects a Go `context.Canceled` error. Because it is not the standard gRPC `Canceled` code (code 1), the gRPC logging middleware logs it at `warn` level instead of `debug` level. The `method` field indicates which gRPC operation was interrupted. Common methods include `/logproto.Querier/GetChunkIDs`, `/logproto.Querier/Query`, and `/logproto.Querier/Series`. Although these are query-path operations, the error appears in ingester logs because ingesters serve gRPC read requests from queriers. Common triggers include: - A user cancels a running query in Grafana - A query timeout is reached in the query frontend, query scheduler, or a reverse proxy - The query frontend cancels redundant split or sharded sub-queries after it has enough results - A load balancer or ingress controller closes the connection due to its own timeout settings **Default configuration:** - Per-tenant query timeout: `query_timeout` in `limits_config` / `-querier.query-timeout` (default: 1m) - Loki server HTTP write timeout: `-server.http-server-write-timeout` (default: 30s) - Load balancer / reverse proxy timeout: depends on your infrastructure **Resolution:** * **Determine whether this is expected behavior.** Occasional Code(499) warnings are normal in production and indicate that a client disconnected before the server responded. These are safe to ignore if they occur infrequently. * **Investigate frequent occurrences.** If you see a high rate of Code(499) errors, check for: - Queries that are too broad or slow, causing clients to time out - Timeout mismatches between layers (for example, a load balancer timeout shorter than the Loki query timeout) - Overloaded ingesters or queriers that cannot respond quickly enough * **Align timeouts across your stack.** Ensure that timeouts increase from outer layers inward so that Loki has time to respond before upstream components give up: ``` Grafana timeout > Load balancer timeout > Query frontend timeout > Querier timeout ``` * **Check query performance.** Use Loki metrics to identify slow queries: ```promql histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (le, route)) ``` * **Reduce log noise.** If the warnings are expected and you want to reduce log volume, increase the log level for the affected component to `error` using `-log.level=error`. Note that this suppresses all warn-level messages, not just Code(499) warnings. **Properties:** - Enforced by: gRPC logging middleware (dskit) - Retryable: N/A (the original client is no longer waiting for a response) - HTTP status: 499 (non-standard, client closed request) - Configurable per tenant: No ### Error: Kafka producer backpressure These errors occur only when the distributor is configured to write to Kafka (`-distributor.kafka-writes-enabled=true`), and indicate that the Kafka producer could not keep up with the incoming write volume. **Error messages:** - `records have timed out before they were able to be produced` - `the maximum amount of records are buffered, cannot buffer more` **Cause:** The distributor's Kafka producer wasn't able to deliver records to the Kafka brokers within `kafka_config.write_timeout`, or the amount of unacknowledged, buffered data reached `kafka_config.producer_max_buffered_bytes`. Both are transient backpressure conditions: the Kafka brokers or the network path to them are slower than the rate at which the distributor is trying to produce records, for example during a broker rollout, a partition rebalance, or a broker-side incident. **Default configuration:** - `kafka_config.write_timeout` (`-kafka.write-timeout`): 10s - `kafka_config.producer_max_buffered_bytes` (`-kafka.producer-max-buffered-bytes`): 1 GiB (0 disables the limit) **Resolution:** * **Implement retry logic** with exponential backoff in your client. * **Check Kafka broker health and load** (under-replicated partitions, broker CPU/disk/network saturation, ongoing rollouts). * **Increase `kafka_config.write_timeout`** if brokers are healthy but consistently slow to acknowledge writes: ```yaml kafka_config: write_timeout: 20s ``` Ensure that this remains less than the distributor's `-distributor.write-timeout` (default: 30s) to avoid distributor-level timeouts. **Properties:** - Enforced by: Distributor (Kafka producer) - Retryable: Yes - HTTP status: 500 Internal Server Error - Configurable per tenant: No ### Error: Service unavailable **Error message:** `service unavailable (503)` **Cause:** Loki is temporarily unable to handle requests due to high load or maintenance. This can occur when ingesters are unhealthy or the ring is not ready. **Default configuration:** - Ingester ring readiness: Requires minimum healthy ingesters based on replication factor - Distributor: Returns 503 when no healthy ingesters available **Resolution:** * **Implement retry logic** with exponential backoff in your client. * **Scale Loki ingester instances** to handle load. * **Check for resource constraints** (CPU, memory, storage). * **Review ingestion patterns** for sudden spikes. **Properties:** - Enforced by: Distributor/Gateway - Retryable: Yes - HTTP status: 503 Service Unavailable - Configurable per tenant: No ## Structured metadata errors These errors occur when using structured metadata incorrectly. ### Error: Structured metadata disabled **Error message:** `stream includes structured metadata, but this feature is disallowed. Please see limits_config.allow_structured_metadata or contact your Loki administrator to enable it` **Cause:** `allow_structured_metadata` defaults to `true`, so this error usually means it has been explicitly disabled, either globally or as a per-tenant override. **Default configuration:** - `allow_structured_metadata`: true (enabled by default in recent versions) **Resolution:** * **Enable structured metadata**: ```yaml limits_config: allow_structured_metadata: true ``` * **Move metadata to regular labels** if structured metadata isn't needed. * **Contact your Loki administrator** to enable the feature. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: Structured metadata too large **Error message:** `stream '{job="app"}' has structured metadata too large: '70000' bytes, limit: '65536' bytes. Please see limits_config.max_structured_metadata_size or contact your Loki administrator to increase it` **Cause:** The structured metadata size exceeds the configured limit. **Default configuration:** - `max_structured_metadata_size`: 64 KB **Resolution:** * **Reduce the size** of structured metadata by removing unnecessary fields. * **Increase the limit**: ```yaml limits_config: max_structured_metadata_size: 128KB ``` * **Use compression or abbreviations** for metadata values. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ### Error: Too many structured metadata entries **Error message:** `stream '{job="app"}' has too many structured metadata labels: '150', limit: '128'. Please see limits_config.max_structured_metadata_entries_count or contact your Loki administrator to increase it` **Cause:** The number of structured metadata entries exceeds the limit. **Default configuration:** - `max_structured_metadata_entries_count`: 128 **Resolution:** * **Reduce the number** of structured metadata entries by consolidating fields. * **Increase the limit**: ```yaml limits_config: max_structured_metadata_entries_count: 256 ``` * **Combine related metadata** into single entries using JSON or other formats. **Properties:** - Enforced by: Distributor - Retryable: No - HTTP status: 400 Bad Request - Configurable per tenant: Yes ## Blocked ingestion errors These errors occur when ingestion is administratively blocked. ### Ingestion blocked **Error message:** `ingestion blocked for user until