# Community # Contributing to Loki For the full contribution guide, see [CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) in the root of the repository. ## Building Clone the repository and use `make` to build: ```bash git clone https://github.com/grafana/loki.git cd loki ``` | Command | Output | |---|---| | `make loki` | `./cmd/loki/loki` | | `make logcli` | `./cmd/logcli/logcli` | | `make loki-canary` | `./cmd/loki-canary/loki-canary` | | `make all` | all of the above | | `make loki-image` | Docker image for Loki | ## Running tests ```bash make test # unit tests make test-integration # integration tests (requires Docker, ~15 min) make lint # run linters ``` ## Contribute to the Helm Chart Refer to [production/helm/loki/CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/production/helm/loki/CONTRIBUTING.md) for chart-specific guidelines. The official Loki Helm chart is maintained in the [grafana/helm-charts](https://github.com/grafana/helm-charts) repository. # Loki community frequently asked questions ### Is Loki open source? What is the license? Grafana Loki is open source, licensed under the GNU Affero General Public License v3 (AGPLv3). The license change from Apache 2.0 to AGPLv3 took effect in 2021. The practical implication of AGPLv3 is that if you modify Loki and make it available over a network, you must make your modifications available under the same license. For most teams running Loki internally for log aggregation, this has no effect. If you are embedding Loki in a commercial product or offering it as a service, review the [license terms](https://github.com/grafana/loki/blob/main/LICENSE) or contact Grafana Labs about a commercial license. Other components in the Grafana observability stack (Grafana, Tempo, Mimir, Alloy) each have their own license terms, detailed in their respective repositories. ### How is the documentation organized, and where do I find docs for a specific product? The Loki documentation lives at [grafana.com/docs/loki/](https://grafana.com/docs/loki/latest/) and is organized into the following top-level sections: - **[Get started](https://grafana.com/docs/loki/latest/get-started/)** — Architecture overview, deployment modes, and label concepts. - **[Set up](https://grafana.com/docs/loki/latest/setup/)** — Installation instructions (Helm, Docker, local binary), migration guides, and upgrade steps. - **[Configure](https://grafana.com/docs/loki/latest/configure/)** — Configuration reference and example configurations for common object storage backends. - **[Send data](https://grafana.com/docs/loki/latest/send-data/)** — Guides for shipping logs to Loki using Grafana Alloy, Promtail, the Docker driver, and the HTTP API. - **[Query](https://grafana.com/docs/loki/latest/query/)** — LogQL language reference, query examples, and the query API. - **[Manage](https://grafana.com/docs/loki/latest/operations/)** — Day-two operations including storage, retention, caching, and multi-tenancy. - **[Alert](https://grafana.com/docs/loki/latest/alert/)** — Recording rules and alerting rules powered by LogQL. Documentation for other Grafana Labs products (Grafana, Mimir, Tempo, Alloy, and others) is available from the [Grafana documentation home page](https://grafana.com/docs/). ### Am I reading the right version of the docs? The Loki documentation is versioned to match each release. By default, links from grafana.com and search results take you to the latest version, but if you arrived via an older link — for example, from a blog post, a forum thread, or a search result — you may be reading documentation for an older release. To check which version you are reading, look at the version selector in the left-hand navigation panel. If it does not say **latest**, you can switch to the current version from that selector, or navigate directly using `/latest/` in the URL — for example, `grafana.com/docs/loki/latest/`. If the version selector shows **next**, you are reading unpublished documentation for the upcoming release. This content may describe features or configuration options that are not yet available in any released version of Loki. The **next** docs are useful for previewing what is coming, but should not be used as a reference for your current deployment. If a configuration option or feature described in the docs does not appear to exist in your Loki deployment, a version mismatch is the most common explanation. Check your installed Loki version (run `loki -version` or check the container image tag) and compare it against the version shown in the docs before assuming the documentation is wrong. ### How can I contribute to or report a problem with the docs? The Loki documentation is open source and lives in the [`docs/sources`](https://github.com/grafana/loki/tree/main/docs/sources) directory of the Loki GitHub repository. Community contributions are welcome. At the bottom of every documentation page you will find three options: - **Suggest an edit** — opens the page source directly in GitHub so you can submit a pull request for a correction or improvement. - **Contribute to docs** — for reporting a minor issue or mistake without making the edit yourself. - **Report a problem** — for reporting technical issues with how the documentation page renders or behaves. For questions that the documentation has not answered, the [Grafana Community Forum](https://community.grafana.com/) is the best place to ask. You can also join the `#loki` channel in the [Grafana Labs Community Slack](https://slack.grafana.com/) to discuss Loki-specific topics with maintainers and other users. ### How can I contribute to or report a problem with the code? Loki is developed in the open on GitHub at [grafana/loki](https://github.com/grafana/loki). There are several ways to contribute: - **Report a bug.** Open an [issue](https://github.com/grafana/loki/issues/new/choose) on GitHub. Include your Loki version, deployment mode, relevant configuration, and steps to reproduce the problem. - **Suggest a feature.** Feature requests are also tracked as GitHub issues. Describe the use case and the behavior you would like to see. - **Submit a pull request.** The full contribution guide is in [CONTRIBUTING.md](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md). It covers building from source, running tests, code style, and the review process. The [Contributing to Loki](https://grafana.com/docs/loki/latest/community/contributing/) docs page provides a quick-start summary. {{< admonition type="note" >}} First-time contributors should look for issues labeled [`good first issue`](https://github.com/grafana/loki/labels/good%20first%20issue) for approachable starting points. {{< /admonition >}} # Contacting the Loki Team For questions regarding Loki: - Ask a question in the `#loki` channel of our [Slack](https://slack.grafana.com/) community - Open source Loki users are welcome to post technical questions on the Grafana Labs Community Forums under the Grafana Loki category at [community.grafana.com](https://community.grafana.com). Please be mindful that this is a community-driven support channel moderated by Grafana Labs staff where Loki maintainers and community members answer questions when bandwidth allows. Be sure to review the [Community Guidelines](https://community.grafana.com/guidelines) before posting. - Users using Loki via [Grafana Cloud](/products/cloud/) can submit support tickets via the [Grafana.com Account Portal](/login). - For questions regarding Enterprise support for Loki, you can get in touch with the Grafana Labs team [here](/contact?pg=docs). Your feedback is always welcome! To submit feedback or a report a potential bug: - [File a GitHub issue](https://github.com/grafana/loki/issues/new) for bugs, issues and feature suggestions. Grafana UI issues for the Loki data source plugin should be posted directly to the [Grafana repository](https://github.com/grafana/grafana/issues/new), not the Loki repository. - Send an email to [lokiproject@googlegroups.com](mailto:lokiproject@googlegroups.com), or visit the [google groups](https://groups.google.com/forum/#!forum/lokiproject) page. - Join us for our monthly [Loki Community Call](https://docs.google.com/document/d/1MNjiHQxwFukm2J4NJRWyRgRIiK7VpokYyATzJ5ce-O8/edit?usp=sharing) on first Thursday of every month alternating between EU (12:00 UTC) and US (17:00 UTC) timezones. Subscribe to the calendar in the doc for being up to date. # Governance This document describes the rules and governance of the project. It is meant to be followed by all the developers of the project and the Loki community. Common terminology used in this governance document are listed below: - **Team members**: Any members of the private [team mailing list][team]. - **Maintainers**: Maintainers lead an individual project or parts thereof ([`MAINTAINERS.md`][maintainers]). - **Projects**: A single repository in the Grafana GitHub organization and listed below is referred to as a project: - loki - puppet-promtail - **The Loki project**: The sum of all activities performed under this governance, concerning one or more repositories or the community. - **The Loki SIG (Special Interest Group) Operator**: The sum of all activities performed under this governance, concerning the repository subfolder `operator` or its specific community. ## Values The Loki developers and community are expected to follow the values defined in the [Code of Conduct][coc]. Furthermore, the Loki community strives for kindness, giving feedback effectively, and building a welcoming environment. The Loki developers generally decide by consensus and only resort to conflict resolution by a majority vote if consensus cannot be reached. ## Projects Each project must have a [`MAINTAINERS.md`][maintainers] file with at least one maintainer. Where a project has a release process, access and documentation should be such that more than one person can perform a release. Releases should be announced on the [announcement][announce] and [users][users] mailing lists. Any new projects should be first proposed on the [team mailing list][team] following the voting procedures listed below. ## Decision making ### Team members Team member status may be given to those who have made ongoing contributions to the Loki project for at least 3 months. This is usually in the form of code improvements and/or notable work on documentation, but organizing events or user support could also be taken into account. New members may be proposed by any existing member by email to the [team mailing list][team]. It is highly desirable to reach consensus about acceptance of a new member. However, the proposal is ultimately voted on by a formal [supermajority vote](#supermajority-vote). If the new member proposal is accepted, the proposed team member should be contacted privately via email to confirm or deny their acceptance of team membership. This email will also be CC'd to the [team mailing list][team] for record-keeping purposes. If they choose to accept, the [onboarding](#onboarding) procedure is followed. Team members may retire at any time by emailing [the team][team]. Team members can be removed by [supermajority vote](#supermajority-vote) on [the team mailing list][team]. For this vote, the member in question is not eligible to vote and does not count towards the quorum. Any removal vote can cover only one single person. Upon death of a member, they leave the team automatically. In case a member leaves, the [offboarding](#offboarding) procedure is applied. The current team members are: - Aditya C S - [adityacs](https://github.com/adityacs) - Ashwanth Goli - [ashwanthgoli](https://github.com/ashwanthgoli) ([Grafana Labs](/)) - Cyril Tovena - [cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](/)) - Danny Kopping - [dannykopping](https://github.com/dannykopping) ([Grafana Labs](/)) - David Kaltschmidt - [davkal](https://github.com/davkal) ([Grafana Labs](/)) - Dylan Guedes - [dylanguedes](https://github.com/dylanguedes) ([Grafana Labs](/)) - Edward Welch - [slim-bean](https://github.com/slim-bean) ([Grafana Labs](/)) - Goutham Veeramachaneni - [gouthamve](https://github.com/gouthamve) ([Grafana Labs](/)) - Joe Elliott - [joe-elliott](https://github.com/joe-elliott) ([Grafana Labs](/)) - Karsten Jeschkies - [jeschkies](https://github.com/jeschkies) ([Grafana Labs](/)) - Kaviraj Kanagaraj - [kavirajk](https://github.com/kavirajk) ([Grafana Labs](/)) - Li Guozhong - [liguozhong](https://github.com/liguozhong) ([Alibaba Cloud](https://alibabacloud.com/)) - Michel Hollands - [michelhollands](https://github.com/michelhollands) ([Grafana Labs](/)) - Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) - Periklis Tsirakidis - [periklis](https://github.com/periklis) ([Red Hat](https://www.redhat.com/)) - Salva Corts - [salvacorts](https://github.com/salvacorts) ([Grafana Labs](/)) - Sandeep Sukhani - [sandeepsukhani](https://github.com/sandeepsukhani) ([Grafana Labs](/)) - Susana Ferreira - [ssncferreira](https://github.com/ssncferreira) - Tom Braack - [sh0rez](https://github.com/sh0rez) ([Grafana Labs](/)) - Tom Wilkie - [tomwilkie](https://github.com/tomwilkie) ([Grafana Labs](/)) The current Loki SIG Operator team members are: - Brett Jones - [blockloop](https://github.com/blockloop/) ([InVision](https://www.invisionapp.com/)) - Cyril Tovena - [cyriltovena](https://github.com/cyriltovena) ([Grafana Labs](/)) - Gerard Vanloo - [Red-GV](https://github.com/Red-GV) ([IBM](https://www.ibm.com)) - Periklis Tsirakidis - [periklis](https://github.com/periklis) ([Red Hat](https://www.redhat.com)) - Sashank Agrawal - [sasagarw](https://github.com/sasagarw/) ([Red Hat](https://www.redhat.com)) ### Maintainers Maintainers lead one or more project(s) or parts thereof and serve as a point of conflict resolution amongst the contributors to this project. Ideally, maintainers are also team members, but exceptions are possible for suitable maintainers that, for whatever reason, are not yet team members. Changes in maintainership have to be announced on the [developers mailing list][devs]. They are decided by [rough consensus](#consensus) and formalized by changing the [`MAINTAINERS.md`][maintainers] file of the respective repository. Maintainers are granted commit rights to all projects covered by this governance. A maintainer or committer may resign by notifying the [team mailing list][team]. A maintainer with no project activity for a year is considered to have resigned. Maintainers that wish to resign are encouraged to propose another team member to take over the project. A project may have multiple maintainers, as long as the responsibilities are clearly agreed upon between them. This includes coordinating who handles which issues and pull requests. ### Technical decisions Technical decisions that only affect a single project are made informally by the maintainer of this project, and [rough consensus](#consensus) is assumed. Technical decisions that span multiple parts of the project should be discussed and made on the [developer mailing list][devs]. Decisions are usually made by [rough consensus](#consensus). If no consensus can be reached, the matter may be resolved by [majority vote](#majority-vote). ### Governance changes Changes to this document are made by Grafana Labs. ### Other matters Any matter that needs a decision may be called to a vote by any member if they deem it necessary. For private or personnel matters, discussion and voting takes place on the [team mailing list][team], otherwise on the [developer mailing list][devs]. ## Voting The Loki project usually runs by informal consensus, however sometimes a formal decision must be made. Depending on the subject matter, as laid out [above](#decision-making), different methods of voting are used. For all votes, voting must be open for at least one week. The end date should be clearly stated in the call to vote. A vote may be called and closed early if enough votes have come in one way so that further votes cannot change the final decision. In all cases, all and only [team members](#team-members) are eligible to vote, with the sole exception of the forced removal of a team member, in which said member is not eligible to vote. Discussion and votes on personnel matters (including but not limited to team membership and maintainership) are held in private on the [team mailing list][team]. All other discussion and votes are held in public on the [developer mailing list][devs]. For public discussions, anyone interested is encouraged to participate. Formal power to object or vote is limited to [team members](#team-members). ### Consensus The default decision making mechanism for the Loki project is [rough][rough] consensus. This means that any decision on technical issues is considered supported by the [team][team] as long as nobody objects or the objection has been considered but not necessarily accommodated. Silence on any consensus decision is implicit agreement and equivalent to explicit agreement. Explicit agreement may be stated at will. Decisions may, but do not need to be called out and put up for decision on the [developers mailing list][devs] at any time and by anyone. Consensus decisions can never override or go against the spirit of an earlier explicit vote. If any [team member](#team-members) raises objections, the team members work together towards a solution that all involved can accept. This solution is again subject to rough consensus. In case no consensus can be found, but a decision one way or the other must be made, any [team member](#team-members) may call a formal [majority vote](#majority-vote). ### Majority vote Majority votes must be called explicitly in a separate thread on the appropriate mailing list. The subject must be prefixed with `[VOTE]`. In the body, the call to vote must state the proposal being voted on. It should reference any discussion leading up to this point. Votes may take the form of a single proposal, with the option to vote yes or no, or the form of multiple alternatives. A vote on a single proposal is considered successful if more vote in favor than against. If there are multiple alternatives, members may vote for one or more alternatives, or vote “no” to object to all alternatives. It is not possible to cast an “abstain” vote. A vote on multiple alternatives is considered decided in favor of one alternative if it has received the most votes in favor, and a vote from more than half of those voting. Should no alternative reach this quorum, another vote on a reduced number of options may be called separately. ### Supermajority vote Supermajority votes must be called explicitly in a separate thread on the appropriate mailing list. The subject must be prefixed with `[VOTE]`. In the body, the call to vote must state the proposal being voted on. It should reference any discussion leading up to this point. Votes may take the form of a single proposal, with the option to vote yes or no, or the form of multiple alternatives. A vote on a single proposal is considered successful if at least two thirds of those eligible to vote vote in favor. If there are multiple alternatives, members may vote for one or more alternatives, or vote “no” to object to all alternatives. A vote on multiple alternatives is considered decided in favor of one alternative if it has received the most votes in favor, and a vote from at least two thirds of those eligible to vote. Should no alternative reach this quorum, another vote on a reduced number of options may be called separately. ## On- / Offboarding ### Onboarding The new member is - added to the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. - announced on the [developers mailing list][devs] by an existing team member. Ideally, the new member replies in this thread, acknowledging team membership. - added to the projects with commit rights. - added to the [team mailing list][team]. ### Offboarding The ex-member is - removed from the list of [team members](#team-members). Ideally by sending a PR of their own, at least approving said PR. In case of forced removal, no approval is needed. - removed from the projects. Optionally, they can retain maintainership of one or more repositories if the [team](#team-members) agrees. - removed from the team mailing list and demoted to a normal member of the other mailing lists. - not allowed to call themselves an active team member any more, nor allowed to imply this to be the case. - added to a list of previous members if they so choose. If needed, we reserve the right to publicly announce removal. [announce]: https://groups.google.com/forum/#!forum/loki-announce [coc]: https://github.com/grafana/loki/blob/main/CODE_OF_CONDUCT.md [devs]: https://groups.google.com/forum/#!forum/loki-developers [maintainers]: https://github.com/grafana/loki/blob/main/MAINTAINERS.md [rough]: https://tools.ietf.org/html/rfc7282 [team]: https://groups.google.com/forum/#!forum/loki-team [users]: https://groups.google.com/forum/#!forum/loki-users # Promtail Push API - Author: Robert Fratto (@rfratto) - Date: Feb 4 2020 - Status: DRAFT Despite being an optional piece of software, Promtail provides half the power of Loki's story: log transformations, service discovery, metrics from logs, and context switching between your existing metrics and logs. Today, Promtail can only be operated to consume logs from very specific sources: files, journal, or syslog. If users wanted to write custom tooling to ship logs, the tooling has to bypass Promtail and push directly to Loki. This can lead users to reimplement functionality Promtail already provides, including its error retries and batching code. This document proposes a Push API for Promtail. The preferred implementation is by copying the existing Loki Push API and implementing it for Promtail. By being compatible with the Loki Push API, the Promtail Push API can allow batches of logs to be processed at once for optimize performance. Matching the Promtail API also allows users to transparently switch their push URLs from their existing tooling. Finally, a series of alternative solutions are detailed. ## Configuration Promtail will have a new target called HTTPTarget, configurable in the `scrape_config` array with the following schema: ```yaml # Defines an HTTP target, which exposes an endpoint against the Promtail # HTTP server to accept log traffic. http: # Defines the base URL for the push path, adding a prefix to the # exposed endpoint. The final endpoint path is # loki/api/v1/push. If omitted, defaults to /. # # Multiple http targets with the same base_url must not exist. base_url: / # Map of labels to add to every log line passed through to the target. labels: {} ``` ### Considerations Users will be able to define multiple `http` scrape configs, but the base URL value must be different for each instance. This allows to cleanly separate pipelines through different push endpoints. Users must also be aware about problems with running Promtail with an HTTP target behind a load balancer: if payloads are load balanced between multiple Promtail instances, ordering of logs in Loki will be disrupted leading to rejected pushes. Users are recommended to do one of the following: 1. Have a dedicated Promtail instance for receiving pushes. This also applies to using the syslog target. 1. Have a separate Kubernetes service that always resolves to the same Promtail pod, bypassing the load balancing issue. ## Implementation As discussed in this document, this feature will be implemented by copying the existing [Loki Push API](/docs/loki//api/#post-lokiapiv1push) and exposing it via Promtail. ## Considered Alternatives Using the existing API was chosen for its simplicity and capabilities of being used for interesting configurations (e.g., chaining Promtails together). These other options were considered but rejected as not the best solution for the problem being solved. Note that Option 3 has value and may be implemented separately from this feature. ### Option 1: JSON / Protobuf Payload A new JSON and Protobuf payload format can be designed instead of the existing Loki push payload. Both formats would have to be exposed to support clients that either can't or won't use protobuf marshalling. The primary benefit of this approach is to allow us to tweak the payload schema independently of Loki's existing schema, but otherwise may not be very useful and is essentially just code duplication. ### Option 2: gRPC Service The [logproto.Pusher](https://github.com/grafana/loki/blob/f7ee1c753c76ef63338d53cfba782188a165144d/pkg/logproto/logproto.proto#L8-L10) service could be exposed through Promtail. This would enable clients stubs to be generated for languages that have gRPC support, and, for HTTP1 support, a [gRPC gateway](https://github.com/grpc-ecosystem/grpc-gateway) would be embedded in Promtail itself. This implementation option is similar to the original proposed solution, but uses the gRPC gateway to handle HTTP/1 traffic instead of the HTTP1 shim that Loki uses. There are some concerns with this approach: 1. The gRPC Gateway reverse proxy will need to play nice with the existing HTTP mux used in Promtail. 1. We couldn't control the HTTP and Protobuf formats separately as Loki can. 1. Log lines will be double-encoded thanks to the reverse proxy. 1. A small overhead of using a reverse proxy in-process will be introduced. 1. This breaks our normal pattern of writing our own shim functions; may add some cognitive overhead of having to deal with the gRPC gateway as an outlier in the code. ### Option 3: Plaintext Payload Prometheus' [Push Gateway API](https://github.com/prometheus/pushgateway#command-line) is cleverly designed and we should consider implementing our API in the same format: users would push to `http://promtail-url/push/label1/value1?timestamp=now` with a plaintext POST body. For example: ``` curl -X POST http://promtail.default/push/foo/bar/fizz/buzz -d “hello, world!” ``` This approach may be slightly faster when compared to a non-plaintext payload as no unmarshaling needs to be performed. This URL path and timestamp still needs to be parsed, but this will generally be faster than the reflection requirements imposed by JSON. However, note that this API limits Promtail to accepting one line at a time and may cause performance issues when trying to handle large volumes of traffic. As an alternative, this API could also be implemented by external tooling and be built on top of any of the other implementation options. An [example implementation](https://github.com/grafana/loki/pull/1270) was created and has received positive support for its simplicity and ease of integration. # Write-Ahead Logs Author: Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) Date: 30/09/2020 ## Impetus Loki already takes numerous steps to ensure the persistence of log data, most notably the use of a configurable replication factor (redundancy) in the ingesters. However, this still leaves much to be desired in persistence guarantees, especially for single binary deployments. This proposal outlines a write ahead log (WAL) in order to complement existing measures by allowing storage/replay of incoming writes via local disk on the ingester components. ## Strategy We suggest a two pass WAL implementation which includes an initial recording of accepted writes (`segments`) and a subsequent checkpointing (`checkpoints`) which coalesces the first pass into more efficient representations to speed up replaying. ### Segments Segments are the first pass and most basic WAL. They store individual records of incoming writes that have been accepted and can be used to reconstruct the in memory state of an ingester without any external input. Each segment is some multiple of 32kB and upon filling one segment, a new segment is created. Initially Loki will try 256kb segment sizes, readjusting as necessary. They are sequentially named on disk and are automatically created when a target size is hit as follows: ``` data └── wal ├── 000000 ├── 000001 └── 000002 ``` ### Truncation In order to prevent unbounded growth and remove operations which have been flushed to storage from the WAL, it is regularly truncated and all but the last segment (which is currently active) are deleted at a configurable interval (`ingester.checkpoint-duration`). This is where checkpoints come into the picture. ### Checkpoints Before truncating the WAL, we advance the WAL segments by one in order to ensure we don't delete the currently writing segment. The directory will look like: ``` data └── wal ├── 000000 ├── 000001 ├── 000002 <- likely not full, no matter └── 000003 <- newly written, empty ``` Each in memory stream is iterated across an interval, calculated by `checkpoint_duration / in_memory_streams` and written to the checkpoint. After the checkpoint completes, it is moved from its temp directory to the `ingester.wal-dir`, taking the name of the last segment before it started (`checkpoint.000002`) and then all applicable segments (`00000`, `00001`, `00002`) and any previous checkpoint are deleted. Afterwards, it will look like: ``` data └── wal ├── checkpoint.000002 <- completed checkpoint └── 000003 <- currently active wal segment ``` #### Queueing Checkpoint operations It’s possible that one checkpoint operation will start at the same time another is running. In this case, the existing checkpoint operation should disregard its internal ticker and flush its series as fast as possible. Afterwards, the next checkpoint operation can begin. This will likely create a localized spike in IOPS before the amortization of the following checkpoint operation takes over and is another important reason to run the WAL on an isolated disk in order to mitigate noisy neighbor problems. After we've written/moved the current checkpoint, we reap the old one. ### WAL Record Types #### Streams A `Stream` record type is written when an ingester receives a push for a series it doesn't yet have in memory. At a high level, this will contain ```golang type SeriesRecord struct { UserID string Labels labels.Labels Fingerprint uint64 // label fingerprint } ``` #### Logs A `Logs` record type is written when an ingester receives a push, containing the fingerprint of the series it refers to and a list of `(timestamp, log_line)` tuples, _after_ a `Stream` record type is written, if applicable. ```golang type LogsRecord struct { UserID string Fingperprint uint64 // label fingerprint for the series these logs refer to Entries []logproto.Entry } ``` ### Restoration Replaying a WAL is done by loading any available checkpoints into memory and then replaying any operations from successively named segments on top (`checkpoint.000003` -> `000004` -> `000005`, etc). It's likely some of these operations will fail because they're already included in the checkpoint (due to delay introduced in our amortizations), but this is ok -- we won't _lose_ any data, only try to write some data twice, which will be ignored. ### Deployment Introduction of the WAL requires that ingesters have persistent disks which are reconnected across restarts (this is a good fit for StatefulSets in Kubernetes). Additionally, it's recommended that the WAL uses an independent disk such that it's isolated from being affected by or causing noisy neighbor problems, especially during any IOPS spike(s). ### Implementation goals - Use underlying prometheus wal pkg when possible for consistency and to mitigate undifferentiated heavy lifting. Interfaces handle page alignment and use []byte. - Ensure this package handles arbitrarily long records (log lines in Loki’s case). - Ensure our in memory representations can be efficiently moved to/from `[]byte` in order to generate conversions for fast/efficient loading from checkpoints. - Ensure chunks which have already been flushed to storage are kept around for `ingester.retain-period`, even after a WAL replay. ### Alternatives #### Use the Cortex WAL Since we're not checkpointing from the WAL records but instead doing a memory dump, this isn't bottlenecked by throughput but rather memory size. Therefore we can start with checkpointing by duration rather than accounting for throughput as well. This makes the proposed solution nearly identical to the Cortex WAL approach. The one caveat is that wal segments will accrue between checkpoint operations and may constitute a large amount of data (log throughput varies). We may eventually consider other routes to handle this if duration based checkpointing proves insufficient. #### Don't build checkpoints from memory, instead write new WAL elements Instead of building checkpoints from memory, this would build the same efficiencies into two distinct WAL Record types: `Blocks` and `FlushedChunks`. The former is a record type which will contain an entire compressed block after it's cut and the latter will contain an entire chunk + the sequence of blocks it holds when it's flushed. This may offer good enough amortization of writes because block cuts are assumed to be evenly distributed and chunk flushes have the same property and use jitter for synchronization. This could be used to drop WAL records which have already elapsed the `ingester.retain-period`, allowing for faster WAL replays and more efficient loading. ```golang type FlushRecord struct { Fingerprint uint64 // labels FlushedAt uint64 // timestamp when it was flushed, can be used with `ingester.retain-period` to either keep or discard records on replay LastEntry logproto.Entry // last entry included in the flushed chunk } ``` It would also allow building checkpoints without relying on an ingester's internal state, but would likely require multiple WALs, partitioned by record type in order to be able to iterate all `FlushedChunks` -> `Blocks` -> `Series` -> `Samples` such that we could no-op the later (lesser priority) types that are superseded by the former types. The benefits do not seem worth the cost here, especially considering the simpler suggested alternative and the extensibility costs if we need to add new record types if/when the ingester changes internally. # Ordering Constraint Removal Author: Owen Diehl - [owen-d](https://github.com/owen-d) ([Grafana Labs](/)) Date: 28/01/2021 ## Problem Loki imposes an ordering constraint on ingested data; that is to say incoming data must have monotonically increasing timestamps, partitioned by stream. This has historical inertia from our parent project, Cortex, but presents unintended consequences specific to log ingestion. In contrast to metric scraping, Loki has reasonable use cases where the ordering constraint poses a problem, including: - Ingesting logs from a cloud function without feeling pressured to add high cardinality labels like invocation_id to avoid out of order errors. - Ingesting logs from other agents/mechanisms that don’t take into account Loki’s ordering constraint. For instance, fluent{d,bit} variants may batch and retry writes independently of other batches, causing unpredictable log loss via out of order errors. Many of these illustrate the adversity between _ordering_ and _cardinality_. In addition to enabling some previously difficult/impossible use cases, removing the ordering constraint lets us avoid potential conflict between these two concepts and helps incentivize good practice in the form of fewer useful labels. ### Requirements - Enable out of order writes - Maintain query interface parity #### Bonuses - Optimize for in order writes ## Alternatives - Implement order-agnostic blocks + increase memory usage by compression_ratio: Deemed unacceptable due to TCO (total cost of ownership). - Implement order-agnostic blocks + scale horizontally (reduce per-ingester streams): Deemed unacceptable due to TCO and increasing ring pressure. - Implement order-agnostic blocks + flush chunks more frequently: Deemed unacceptable due to negatively increasing the index and the number of chunks requiring merge during reads. ## Design ### Background I suggest allowing a stream's head block to accept unordered writes and later re-order cut blocks similar to merge-sort before flushing them to storage. Currently, writes are accepted in monotonically increasing timestamp order to a _headBlock_, which is occasionally "cut" into a compressed, immutable _block_. In turn, these _blocks_ are combined into a _chunk_ and persisted to storage. ``` Figure 1 Data while being buffered in Ingester | Chunk in storage | Blocks Head | --------------------------------------------------------------------- | | ts0 ts1 ts2 ts3 ts4 ts5 ts6 ts7 ts8 ts9 | -------------- ---------------- | | --------- --------- --------- --------- --------- | | blocks |-- | head block | | | |block 0| |block 1| |block 2| |block 4| |block 5| | |(compressed)| | |(uncompressed)| | | | | | | | | | | | | | | | | ------> | | | | --------- --------- --------- --------- --------- | | | | | | | | | -------------- | ---------------- | --------------------------------------------------------------------- | | | -------------- | ``` Historically because of Loki's ordering constraint, these blocks maintain a monotonically increasing timestamp (abbreviated `ts`) order where ``` Figure 2 start end ts0 ts1 ts2 ts3 -------------- -------------- | | | | | | --------> | | | | | | | | | | -------------- -------------- ``` This allows us two optimizations: 1) We can store much more data in memory because each block is compressed after being cut from a head block. 2) We can query the block's metadata, such as `ts0` and `ts1` and skip querying it in the case of i.e. the timestamps are outside a request's bounds. ### Unordered Head Blocks The head block's internal structure will be replaced with a tree structure, enabling logarithmic inserts/lookups and `n log(n)` scans. _Cutting_ a block from the head block will iterate through this tree, creating a sorted block identical to the ones currently in use. However, because we'll be accepting arbitrarily-ordered writes, there will no longer be any guaranteed inter-block order. In contrast to figure 2, blocks may have overlapping data: ``` Figure 3 start end ts1 ts3 ts0 ts2 -------------- -------------- | | | | | | --------> | | | | | | | | | | -------------- -------------- ``` Thus _all_ blocks must have their metadata checked against a query. In this example, a query for the bounds `[ts1,ts2]` would need to decompress and scan the `[ts1, ts2]` range across both of them, but a query against `[ts3, ts4]` would only decompress and scan _one_ block. ``` Figure 4 chunk1 ------------------- chunk2 --------------------- query range requiring both ---------- query range requiring chunk2 only ----------- ts0 ts1 ts2 ts3 ts4 (not in any block) ------------------------------ | | | | | | | | | | | | | | | | ------------------------------ ``` The performance losses against the current approach includes: 1) Appending a log line is now performed in logarithmic time instead of amortized (due to array resizing) constant time. 2) Blocks may contain overlapping data (although ordering is still guaranteed within each block). 3) Head block scans are now `O(n log(n))` instead of `O(n)` ### Flushing and Chunk Creation Loki regularly combines multiple blocks into a chunk and "flushes" it to storage. In order to ensure that reads over flushed chunks remain as performant as possible, we will re-order a possibly-overlapping set of blocks into a set of blocks that maintain monotonically increasing order between them. From the perspective of the rest of Loki’s components (queriers/rulers fetching chunks from storage), nothing has changed. {{< admonition type="note" >}} **In the case that data for a stream is ingested in order, this is effectively a no-op, making it well optimized for in-order writes (which is both the requirement and default in Loki currently). Thus, this should have little performance impact on ordered data while enabling Loki to ingest unordered data.** {{< /admonition >}} #### Chunk Durations When `--validation.reject-old-samples` is enabled, Loki accepts incoming timestamps within the range ``` [now() - `--validation.reject-old-samples.max-age`, now() + `--validation.create-grace-period`] ``` For most of our clusters, this would mean the range of acceptable data is one week long. In contrast, our max chunk age is `2h`. Allowing unordered writes would mean that ingesters would willingly receive data for 168h, or up to 84 distinct chunk lengths. This presents a problem: a malicious user could be writing to many (84 in this case) distinct chunks simultaneously, flooding Loki with underutilized chunks which bloat the index. In order to mitigate this, there are a few options (not mutually exclusive): 1) Lower the valid acceptance range 2) Create an _active_ validity window, such as `[most_recent_sample-max_chunk_age, now() + creation_grace_period]`. The first option is simple, already available, and likely somewhat reasonable. The second is simple to implement and an effective way to ensure Loki can ingest unordered logs but maintain a sliding validity window. I expect this to cover nearly all reasonable use cases and effectively mitigate bad actors. #### Chunk Synchronization We also cut chunks according to the `sync_period`. The first timestamp ingested past this bound will trigger a cut. This process aids in increasing chunk determinism and therefore our deduplication ratio in object storage because chunks are [content addressed](https://en.wikipedia.org/wiki/Content-addressable_storage). With the removal of our ordering constraint, it's possible that in some cases the synchronization method will not be as effective, such as during concurrent writes to the same stream across this bound. {{< admonition type="note" >}} **It's important to mention that this is possible today with the current ordering constraint, but we'll be increasing the likelihood by removing it.** {{< /admonition >}} ``` Figure 5 Concurrent Writes over threshold ^ ^ | | | | -----------------|----------------- | v Sync Marker ``` To mitigate this problem and preserve the benefits of chunk deduplication, we'll need to make chunk synchronization less susceptible to non-determinism during concurrent writes. To do this, we can move the synchronization trigger from the `Append` code path to the asynchronous `FlushLoop`. Note, the semantics for _when_ a chunk is cut will not change: that is, on the first timestamp crossing the synchronization bound. However, _cutting_ the chunks for synchronization on the flush path mitigates the likelihood of _different_ chunks being cut. In order to cut multiple chunks with different hashes, appends would then need to cross this boundary at the same time the flush loop checks the stream, which should be very unlikely. ### Future Opportunities This ends the initial design portion of this document. Below, I'll describe some possible changes we can address in the future, should they become warranted. #### Variance Budget The intended approach of a "sliding validity" window for each stream is simple and effective at preventing misuse and bad actors from writing across the entire acceptable range for incoming timestamps. However, we may in the future wish to take a more sophisticated approach, introducing per tenant "variance" budgets, likely derived from the stream limit. This ingester limit could, for example use an incremental (online) standard deviation/variance algorithm such as [Welford's](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance), which would allow writing to larger ranges than option (2) in the _Chunk Durations_ section. #### LSM Tree Much of the proposed approach mirrors an [LSM-Tree](http://www.benstopford.com/2015/02/14/log-structured-merge-trees/) (Log Structured Merge Tree), albeit in memory instead of using disk. What a weird choice -- LSM Trees are designed to effectively use disk, so why not go that route? We currently have no wish to add extra disk dependencies to Loki where we can avoid it, but below I will outline what an LSM-Tree approach would look like. Ultimately, using disk would enable buffering more data in the ingester before flushing, **Allowing us to** - Flush more efficiently utilized chunks (in some cases) - Keep open a wider validity window for incoming logs **At the cost of** - being susceptible to disk-related complexity and problems ##### MemTable (head block) Writes in an LSM-Tree are first accepted to an in-memory structure called a _memtable_ (generally a balancing tree such as red-black) until the memtable hits a preconfigured size. In Loki, this corresponds to the stream’s head block, which is uncompressed. ##### SSTables (blocks) Once a Memtable (head block) in an LSM-Tree hits a predefined size, it is flushed to disk as an immutable sorted structure called an SSTable (sorted strings table). In Loki, we can use either the pre-existing MemChunk format, which is ordered, compact, and contains a block index within it, or the pre-existing block format directly. These are stored on disk to lessen memory pressure and loaded for queries when necessary. ##### Block Index Incoming reads in an LSM-Tree may need access to the SSTable entries in addition to the currently active memtable (head block). In order to improve this, we may cache the metadata including block offsets, start and end timestamps within an SSTable (block || MemChunk) in memory to mitigate lookups, seeking, and loading unnecessary data from disk. ##### Compaction (flushing) Compaction in an LSM-Tree combines and reorders multiple SSTables (blocks || MemChunks). This is mainly covered in the _Flushing_ section of the in-memory approach, but _compaction_ is equivalent to _flushing_ for our case. That is, merge multiple SSTables on disk together in an algorithm reminiscent of merge sort and flush them to storage in our ordered chunk format. # Labels Author: Ed Welch Date: February 2019 This is the official version of this doc as of 2019/04/03, the original discussion was had via a [Google doc](https://docs.google.com/document/d/16y_XFux4h2oQkJdfQgMjqu3PUxMBAq71FoKC_SkHzvk/edit?usp=sharing), which is being kept for posterity but will not be updated moving forward. ## Problem Statement We should be able to filter logs by labels extracted from log content. Keeping in mind: Loki is not a log search tool and we need to discourage the use of log labels as an attempt to recreate log search functionality. Having a label on “order number” would be bad, however, having a label on “orderType=plant” and then filtering the results on a time window with an order number would be fine. (think: grep “plant” | grep “12324134” ) Loki as a grep replacement, log tailing or log scrolling tool is highly desirable, log labels will be useful in reducing query results and improving query performance, combined with logQL to narrow down results. ## Use Cases As defined for prometheus “Use labels to differentiate the characteristics of the thing that is being measured” There are common cases where someone would want to search for all logs which had a level of “Error” or for a certain HTTP path (possibly too high cardinality), or of a certain order or event type. Examples: - Log levels. - HTTP Status codes. - Event type. ## Challenges - Logs are often unstructured data, it can be very difficult to extract reliable data from some unstructured formats, often requiring the use of complicated regular expressions. - Easy to abuse. Easy to create a Label with high cardinality, even possibly by accident with a rogue regular expression. - Where do we extract metrics and labels at the client (Promtail or other?) or Loki? Extraction at the server (Loki) side has some pros/cons. Can we do both? At least with labels we could define a set of expected labels and if Loki doesn’t receive them they could be extracted. - Server side extraction would improve interoperability at the expense of increase server workload and cost. - Are there discoverability questions/concerns with metrics exposed via Loki vs the agent? Maybe this is better/easier to manage? - Potentially more difficult to manage configuration with the server side having to match configs to incoming log streams ## Existing Solutions There already exist solutions for extracting processing and extracting metrics from unstructured log data, however, they will not quite work for extracting labels without some work and neither support easy inclusion as a library. It’s worth noting and understanding how they work to try to get the best features in our solution. mtail https://github.com/google/mtail 1721 github stars, huge number of commits, releases and contributors, google project All go, uses go RE2 regular expressions which is going to be more performant than grok_exporter below which uses a full regex implementation allowing backtracking and lookahead required to be compliant with Grok but which are also slower. grok_exporter https://github.com/fstab/grok_exporter 278 github stars, mature/active project If you are familiar with Grok this would be more comfortable, many people use ELK stacks and would likely be familiar with or already have Grok strings for their logs, making it easy to use grok_exporter to extract metrics. One caveat is the dependency on the oniguruma C library which parses the regular expressions. ## Implementation ### Details As mentioned previously in the challenges for working with unstructured data, there isn’t a good one size fits all solution for extracting structured data. The Docker log format is an example where multiple levels of processing may be required, where the docker log is json, however, it also contains the log message field which itself could be embedded json, or a log message which needs regex parsing. A pipelined approach should allow for handling these more challenging scenarios There are 2 interfaces within Promtail already that should support constructing a pipeline: ```go type EntryMiddleware interface { Wrap(next EntryHandler) EntryHandler } ``` ```go type EntryHandler interface { Handle(labels model.LabelSet, time time.Time, entry string) error } ```` Essentially every entry in the pipeline will Wrap the log line with another EntryHandler which can add to the LabelSet, set the timestamp, and mutate(or not) the log line) before it gets handed to the next stage in the pipeline. ### Example ```json { "log": "level=info msg=\”some log message\”\n", "stream": "stderr", "time": "2012-11-01T22:08:41+00:00" } ``` This is a docker format log file which is JSON but also contains a log message which has some key-value pairs. Our pipelined config might look like this: ```yaml scrape_configs: - job_name: system pipeline_stages: - json: timestamp: source: time format: RFC3339 labels: stream: source: json_key_name.json_sub_key_name output: log - regex: expr: '.*level=(?P[a-zA-Z]+).*' labels: level: - regex: expr: '.*msg=(?P[a-zA-Z]+).*' output: message ``` Looking at this a little closer: ```yaml - json: timestamp: source: time format: TODO ① labels: stream: source: json_key_name.json_sub_key_name ② output: log ③ ``` ① The format key will likely be a format string for Go’s time.Parse or a format string for strptime, this still needs to be decided, but the idea would be to specify a format string used to extract the timestamp data, for the regex parser there would also need to be a expr key used to extract the timestamp. ② One of the json elements was “stream” so we extract that as a label, if the json value matches the desired label name it should only be required to specify the label name as a key, if some mapping is required you can optionally provide a “source” key to specify where to find the label in the document. (Note the use of `json_key_name.json_sub_key_name` is just an example here and doesn't match our example log) ③ Tell the pipeline which element from the json to send to the next stage. ```yaml - regex: expr: '.*level=(?P[a-zA-Z]+).*' ① labels: level: ② ``` ① Define the Go RE2 regex, making sure to use a named capture group. ② Extract labels using the named capture group names. Notice there was not an output section defined here, omitting the output key should instruct the parser to return the incoming log message to the next stage with no changes. ```yaml - regex: expr: '.*msg=(?P[a-zA-Z]+).*' output: message ① ``` ① Send the log message as the output to the last stage in the pipeline, this will be what you want Loki to store as the log message. There is an alternative configuration that could be used here to accomplish the same result: ```yaml scrape_configs: - job_name: system pipeline_stages: - json: timestamp: source: time format: FIXME labels: stream: output: log - regex: expr: '.*level=(?P[a-zA-Z]+).*msg=(?P[a-zA-Z]+).*' labels: level: ① log: source: message ② output: message ``` ① Similar to the json parser, if your log label matches the regex named group, you need only specify the label name as a yaml key ② If you had a use case for specifying a different label name from the regex group name you can optionally provide the `source` key with the value matching the named capture group. You can define a more complicated regular expression with multiple capture groups to extract many labels and/or the output log message in one entry parser. This has the advantage of being more performant, however, the regular expression will also get much more complicated. Note the regex for `message` is incomplete and would do a terrible job of matching any standard log message which might contain spaces or non alpha characters. ### Concerns - Debugging, especially if a pipeline stage is mutating the log entry. - Clashing labels and how to handle this (two stages try to set the same label) - Performance vs ease of writing/use, if every label is extracted one at a time and there are a lot of labels and a long line, it would force reading the line many times, however contrast this to a really long complicated regex which only has to read the line once but is difficult to write and/or change and maintain ### Further improvements There are some basic building blocks for our pipeline which will use the EntryMiddleware interface, the two most commonly used will likely be: - Regex Parser - JSON Parser However we don’t want to ask people to copy and paste basic configs over and over for very common use cases, so it would make sense to add some additional parsers which would really be supersets of the base parsers above. For example, the config above might be simplified to: ```yaml scrape_configs: - job_name: system pipeline_stages: - docker: ``` or ```yaml scrape_configs: - job_name: system pipeline_stages: - cri: ``` Which could still easily be extended to extract additional labels: ```yaml scrape_configs: - job_name: system pipeline_stages: - docker: - regex: expr: '.*level=(?P[a-zA-Z]+).*' labels: level: ``` ### Auto Detection? An even further simplification would be to attempt to autodetect the log format, a PR for this work has been submitted, then the config could be as simple as: ```yaml scrape_configs: - job_name: system pipeline_stages: - auto: ``` This certainly has some advantages for people first adopting and testing Loki, allowing them to point it at their logs and at least get the timestamp and log message extracted properly for the common formats like Docker and CRI. There are also some challenges with auto detection and edge cases, though most people are going to want to augment the basic config with additional labels, so maybe it makes sense to default to auto but suggest when people start writing configs they chose the correct parser? ## Other Thoughts and Considerations - We should have a standalone client in some fashion which allows for testing of log parsing at the command line, allowing users to validate regular expressions or configurations to see what information is extracted. - Other input formats to the pipeline which are not reading from log files, such as containerd grpc api, or from stdin or unix pipes, etc. - It would be nice if at some point we could support loading code into the pipeline stages for even more advanced/powerful parsing capabilities. # Releasing Loki Build Image The [`loki-build-image`](https://github.com/grafana/loki/blob/main/loki-build-image) is the Docker image used to run tests and build Grafana Loki binaries in CI. The build and publish process of the image is triggered upon a merge to `main` if any changes were made in the folder `./loki-build-image/`. **To build and use the `loki-build-image`:** 1. Create a branch with the desired changes to the `./loki-build-image/Dockerfile`. 1. Update the `BUILD_IMAGE_VERSION` variable in the `Makefile`. 1. Commit your changes. 1. Run `make build-image-push` to build and publish the new version of the build image. 1. Run `make release-workflows` to update the Github workflows. 1. Commit your changes. 1. Push your changes to the remote branch. 1. Open a PR against the `main` branch. # Backport commits Any PRs or commits not on the release branch that you want to include in the release must be backported to the release branch. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 2. If the release branch already has all the code changes on it, skip this step. ## Steps 1. Pick a PR that you want to backport to `release-VERSION_PREFIX` branch. 1. Add two labels to the PR. First, one of the `product-approved`, `type/doc` or `type/bug` appropriately. This is to make sure the PRs that are backported are done with right intention. Second `backport release-VERSION_PREFIX` label. Now CI should automatically create backport PR to the correct release branch. Example [PR](https://github.com/grafana/loki/pull/10333) {{< admonition type="note" >}} CI automation can fail sometimes if there are some merge conflicts in cherry picking the commits. In those cases, the original PR where you added the label should have additional comment explaining how to backport it manually. {{< /admonition >}} {{< admonition type="note" >}} The CI job that helps with backporting PR is `.github/workflows/backport.yml`. Useful for debugging purposes. {{< /admonition >}} 1. Repeat the above steps for any PRs that need to be backported. ## Backporting Release PRs If backporting a release PR, make sure you remove any `autorelease: pending` or `autorelease: tagged` labels before merging the backport PR. By default our backport action brings over all labels, but these labels are reserved for the release workflow and will cause future pipelines to fail if left of backport PRs. # Create Release Branch A single release branch is created for every `major` or `minor` release (not for patch releases). That release branch is then used for all the Stable Releases, and all Patch Releases for that `major` and `minor` versions of the Grafana Loki. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 1. Announce about the upcoming release in `#loki-releases` internal slack channel. 1. Skip this announcement for a patch release. Create an issue to communicate beginning of the release process with the community. Example issue [here](https://github.com/grafana/loki/issues/10468). ## Steps 1. Determine which commit should be used as a base for the release branch. Usually this is a `kxx` weekly release branch. 1. Create and push the release branch from the selected base commit: The name of the release branch should be `release-VERSION_PREFIX`, such as `release-2.9.x`. {{< admonition type="note" >}} Branches are only made for VERSION_PREFIX; do not create branches for the full VERSION such as `release-2.9.1`. {{< /admonition >}} {{< admonition type="note" >}} Don't create any other branches that are prefixed with `release` when creating PRs or those branches will collide with our automated release build publish rules. {{< /admonition >}} 1. Create a label to make backporting PRs to this branch easy. The name of the label should be `backport release-VERSION_PREFIX`, such as `backport release-2.9.x`. {{< admonition type="note" >}} Note there is space in the label name. The label name must follow this naming convention to trigger CI related jobs. {{< /admonition >}} # Document metrics and configurations changes Any metrics and configurations that are removed or modified need to be documented in the upgrade guide. Configurations whose default values changed should also be documented in the upgrade guide. ## Before you begin All the steps are performed on `release-VERSION_PREFIX` branch. ## Steps 1. Check which configs changed, including whose default values changed. ``` $ OLD_VERSION=X.Y.Z ./tools/diff-config.sh ``` 1. Record configurations that are modified (either renamed or had its default value changed) in the [upgrade guide](../prepare-upgrade-guide/). 1. Check if any metrics have changed. ``` $ OLD_VERSION=X.Y.Z ./tools/diff-metrics.sh ``` 1. Record metrics whose names have been modified in the [upgrade guide](../prepare-upgrade-guide/). # Prepare Major Release A major release follows the same process as [minor and patch releases](../prepare-release/), but requires a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don't do major releases very often, so it is not economical to keep those workflows running all the time. To create a major release workflow, follow the steps below. 1. Edit `./github/release-workflows.jsonnet` 1. Add a new workflow for the major release. For example, the 3.0 release looked like the following: ```jsonnet 'major-release-pr.yml': std.manifestYamlDoc( lokiRelease.releasePRWorkflow( branches=['release-*.0.0'], buildImage=buildImage, checkTemplate=checkTemplate, distRunsOn='ubuntu-x64', distOptionalTargets=['dist/loki-linux-riscv64'], golangCiLintVersion=golangCiLintVersion, imageBuildTimeoutMin=imageBuildTimeoutMin, imageJobs=imageJobs, imagePrefix=imagePrefix, releaseLibRef=releaseLibRef, releaseRepo='grafana/loki', skipArm=false, skipValidation=false, // Instead of hard-coding the release version, push an empty to the target branch `release-X.0.0` // git commit --allow-empty -m "chore: Release X.0.0-rcY" -m "Release-As: X.0.0-rcY" // https://github.com/googleapis/release-please#how-do-i-change-the-version-number // releaseAs='x.0.0', ) + { name: 'Prepare Major Release PR', }, false, false ), ``` 1. Make sure the `branches` field is set to the release branch you want to release from. 1. Make sure the `releaseAs` field is set to the version you want to release. 1. Run `make release-workflows` to generate the new workflow. Merge this change to both the main and release branch. You may also want to disable the patch release workflow as part of this PR, see below. ## Race Condition with Patch Release Once this new workflow is created there is a bit of a race condition with the patch workflow. This can be resolved in 1 of 2 ways: 1. Disable the patch release workflow until after the major release is complete. 1. Watch all actions for the release branch (ie. ) and cancel and patch release runs. # Merge Release PR To release Loki, merge the release PR. This PR will have the title `chore(): release `. Here's what the 3.0 release PR looked like. ![3.0 release PR](.3.0-release-pr.png) 3.0 was the first major release using this new process, and enforcing conventional commits was relatively new. Going forward the release notes in the release PR will be much more thorough, as we're now enforcing that every PR has a conventional commit message. To test the artifacts before releasing, you can download the artifacts from the link in the footer of the PR description. Merging the PR will: 1. Fetch the built artifacts. 1. Create a draft GitHub release with the release notes in the PR description. 1. Upload fetched binaries to the draft release. 1. Publish fetched images to Docker Hub as multi-arch images. 1. Publish the draft release and create the GitHub tag. 1. (Optionally) Mark the release as the latest if it represents the newest version of Loki. ## Troubleshooting / Retrying Release If something goes wrong with the release automation, you may need to rerun the job. This may require updating the release code in the `grafana/loki` repo via a PR, or it may include updating the code in `grafana/loki-release` that the release pipeline fetches. In either case, if you need to re-release a merge release PR, you'll need to remove the `autorelease: tagged` label from that PR and add the `autorelease: pending` label. The automation relies on these labels to determine which merged PRs have and have not yet been released. After fixing the labels, re-run the release workflow, and it should correctly get passed the `should-release` step. The release process is idempotent, so if a draft release has already been created via a previously failed release, the process will continue, re-upload the binaries, and re-publish the images. Once the release is finished, you may need to manually flip the label back to `autorelease: tagged`. # Patch Go version Update vulnerable Go version to non-vulnerable Go version to build Grafana Loki binaries. ## Steps 1. Find Go version to which you need to update. Example `1.20.5` to `1.20.6` 1. Update Go version (`GO_VERSION`) in the `Makefile`. 1. Run `make release-workflows` to update the version in generated release workflows. 1. Run `make update-go-version` to update all relevant files. 1. Open a pull request against the target branch (`main` or `release-MAJOR.MINOR.x`). # Patch vulnerabilities This step patches vulnerabilities in Grafana Loki binaries and Docker images. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). Vulnerabilities can be from two main sources. 1. Grafana Loki source code. 1. Grafana Loki dependencies. Grafana Loki dependencies can be 1. Go dependencies in `go.mod` 1. The Go compiler itself 1. Grafana Loki Docker dependencies, for example, the base images Before start patching vulnerabilities, know what are you patching. It can be one or more from sources mentioned above. Use `#security-go`, `#security` slack channels to clarify. ## Steps 1. Patch Grafana Loki source code. Means, there are vulnerabilities in Grafana Loki source code itself. 1. Patch it on `main` branch 1. [Backport](../backport-commits/) to `release-$VERSION_PREFIX` branch. 1. Patch Go dependencies. 1. Pick all the Go dependencies that need to be patched. 1. Check if [dependabot already patched the dependency](https://github.com/grafana/loki/pulls?q=is%3Apr+label%3Adependencies+is%3Aclosed) or [have a PR opened to patch](https://github.com/grafana/loki/pulls?q=is%3Apr+is%3Aopen+label%3Adependencies) . If not, manually upgrade the package on the `main` branch as follows. ```shell go get -u -v @ go mod tidy go mod vendor ``` 1. [Backport](../backport-commits/) it to `release-$VERSION_PREFIX` branch. 1. Repeat for each Go dependency 1. [Patch Go compiler](../patch-go-version/). 1. Patch Grafana Loki Docker dependencies, for example: Alpine Linux base images). 1. Update Docker image version. [Example PR](https://github.com/grafana/loki/pull/10573). 1. [Backport](../backport-commits/) to `release-$VERSION_PREFIX` branch # Prepare Release Releasing Grafana Loki consists of merging a long-running release PR. Two workflows keep these PRs up to date, one for patch releases (which runs on commits to branches matching `release-[0-9]+.[0-9]+.x`) and one for minor releases (which runs on commits to branches matching `k[0-9]+`). These pipelines use release please to do the following: 1. Run tests and linting 1. Build binaries and images for the proposed release version 1. Generate release notes based on conventional commits since the last release 1. Create or update the long-running release PR for that release, indicating which commit will be released if the PR is merged with a link to the built artifacts that will be published. ## Major releases Major releases follow the same process as minor and patch releases, but require a custom workflow to be created to run on the branch we want to release from. The reason for this is that we don't do major releases very often, so it is not economical to keep those workflows running all the time.To create a major release workflow, follow the steps in the [major release workflow](../major-release/) document. # Prepare Upgrade guide The upgrade guide records changes that require user attention or interaction to upgrade to specific Loki version from previous versions. ## Before you begin The upgrade guide in Grafana Loki works as follows. We have a `setup/upgrade/_index.md` file that records upgrade information for all Loki releases. ## Steps 1. Make sure the upgrade guide is up to date on the `release-VERSION_PREFIX` branch under the `Main/Unreleased` section. 1. On the `release-VERSION_PREFIX` branch promote `Main/Unreleased` to `VERSION`. Example [PR](https://github.com/grafana/loki/pull/10470). 1. On the `main` branch remove entries from `Main/Unreleased` that are already part of `VERSION (YYY-MM-DD)`. # Update version numbers Upgrade the Loki version to the new release version in documents, examples, jsonnets, etc. ## Before you begin 1. Determine the [VERSION_PREFIX](../concepts/version/). 2. Skip this step if you are doing a patch release on old release branch. Example: Latest release branch is `release-2.9.x` but you are releasing patch version in `release-2.8.x` (or older) ## Steps 1. Upgrade the versions in the `release-VERSION_PREFIX` branch. Example commands: ``` LOKI_NEW_VERSION=$VERSION ./tools/release_update_tags.sh ``` {{< admonition type="note" >}} Do not upgrade the version numbers in the `operator/` directory as @periklis and team have a different process to upgrade the Operator version. {{< /admonition >}} # Version Grafana Loki uses Semantic Versioning. The next version can be determined by looking at the current version and incrementing it. When the release is a new `major` version (for example, after 2.9.1 we decide to go 3.0.0 instead of 2.9.2) this should not be done. You need to set two environmental values `VERSION` and `VERSION_PREFIX` to do the release process. ## Version To determine the `VERSION` for a Stable Release or Patch Release, use the Semantic Version `A.B.C`. - Examples - For example, `2.9.0` is the Stable Release `VERSION` for the v2.9.0 release. - For example, `2.9.1` is the first Patch Release `VERSION` for the v2.9.0 release. ## Version Prefix To determine the `VERSION PREFIX`, use only the major and minor version `A.B.x`. - Examples - `2.9.x`